Building a router [Spring 2018]
One of the problems I had after building my cluster was to do with address allocation. The router at the house where I was living at the time had been locked down by the landlords, so I was unable to set static IP addresses on it, or even look at the router settings for that matter. It is possible to configure a static address on each machine, and that is exactly what I did when setting everything up. Occasionally though, the router would allocate one of my static IPs to another device on the network, leaving me unable to connect to some of my machines or otherwise messing things up.
The protocol responsible for handing out addresses on an IP network is called DHCP (dynamic host configuration protocol). When a device connects to a network, one of the first things it does is broadcast a DHCP request. A a listening DHCP server then responds with an IP address for the device to use, ensuring that it doesn't conflict with any already allocated addresses. It stands to reason then that there can only be one DHCP server on a network at any given time, or there will be trouble. So simply running my own DHCP server on the local network wasn't going to cut it. It was through this conundrum that I got my first real taste of networking. I ended up going down something of a rabbit hole to solve this one and building my own little subnet and router/firewall server, and ended up learning a lot along the way.
THE HARDWARE:
If you are looking to build your own router, hardware wise all you really need is a system with two or more network interfaces. It really is that simple. If you want to connect it directly to a broadband connection, you might need a modem card or similar device. In my case though, my router was going to host a subnet inside an existing gigabit Ethernet network, so all I needed was a machine with two or more ordinary Ethernet ports. The machine was built using spare parts bolted to a plank of wood. For the operating system, I broke out my trusty Ubuntu server 16.04 install USB. The resulting machine was quiet, low power and suitable for 24/7 operation. There really isn't much more to tell.
NETWORK ADDRESS TRANSLATION:
The first piece of the puzzle was NAT. Broadly speaking, what NAT, aka network address translation, does is allow a single machine to expose services hosted on many separate machines via a single network interface. Effectively allowing many machines to share one IP address. This technology is extremely widely deployed, thanks in part to the global shortage of ipv4 addresses. It has become the standard solution for building local networks and connecting many machines to the internet via a single IP. It's used by businesses, home routers, and sometimes even by ISPs, with the (somewhat controversial) carrier-grade NAT. It also acts as a sort of firewall that prevents arbitrary traffic from the internet from making its way to machines on local networks. Although that isn't really it's purpose, and it's generally not considered a security feature.
As mentioned previously, to set up a NAT/firewall server, you first need a machine with two network interfaces, one to face the external network or WAN (wide area network), and another to host your subnet. The internal interface is called the "gateway" and is where machines within the subnet looking to connect to external addresses will send their packets. The NAT server then dutifully forwards those packets on to their intended destinations. Afterwards, any returning packets are directed back to the machine that initiated the connection. For incoming connections, things work a bit differently. The NAT server uses the port number to decide to which device on the subnet the connection should go to in a process known as port forwarding. If you've ever set up a service on your home internet connection, you will be familiar with this process.
The first thing to do is to configure the network interfaces. In most graphical desktop operating systems, networking is handled automagically by a network manager widget that usually resides somewhere on the desktop. In the case of Ubuntu server however, the raising of network interfaces is handled at boot time based on the contents of a configuration file (/etc/network/interfaces). Mine ended up looking like this:
The first interface "lo" is the loop-back interface. This is a virtual interface used for TCP/IP communication between processes running locally, and is not really relevant here. The second interface (enp3s0 in my case, though it varies) is the internal network interface, the one that faces the soon to be intranet. I configured it to use the IP address 10.10.10.1, and will serve as the gateway for local machines. Now you might be thinking, what prevents someone from choosing an IP that conflicts with something on the wider network? And the short answer is ostensibly nothing. There are however a few IP address ranges which are set aside for private networks, of which 10.*.*.* is one. The third interface (enp2s0) is the external interface that will connect to the home router, and then subsequently to the internet. This one has been configured to automatically get an address from the parent network, just as any normal device would.
The next step is to enable kernel IP forwarding. What this means is that if a packet comes into a network interface that matches the subnet of another interface, it will be re-transmitted from that interface. Naturally, this is something that we want. By default though, this behaviour is disabled. Enabling it can be done done using the following command:
This change doesn't persist through reboots however. To make it so that it does, the appropriate line (net.ipv4.ip_forward=1) has to be un-commented in "/etc/sysctl.conf".
The next step was to actually configure NAT. For this I used iptables. Iptables works by matching incoming (or outgoing packets) to lists of user-defined rules, each of which represents a routing decision. Usually, whether or not to block the packet in question. Rules are organised into tables and chains which are selected depending on the stage a packet is at in the routing process. To set up NAT, I used the following commands:
The first appends a rule to the forward chain which accepts packets coming in to the subnet (from enp2s0 to enp3s0), but only if they are part of an established connection. The second command appends another rule to the forward chain, allowing any packets leaving the subnet (from enp3s0 to enp2s0) to go to the wider internet. The third and final rule "caps off" the forward chain by blocking any packets that don't match either of the previous two rules. The final command actually does the business, and sets a rule on the postrouting chain in the NAT table to take any forwarded packets and transform them using the "masquerade" algorithm, a form of NAT.
Lastly, you probably want your iptables configuration to persist across reboots. Which it doesn't do by default. There is a package under Ubuntu called iptables-persistent which will automatically save your iptables rules and restore them on reboot. The following commands save iptables rules to the appropriate file and install the package:
DHCP:
With a subnet and router set up, the final task was to configure a DHCP server to hand out addresses. Thankfully, package managers come to the rescue again, and the DHCP server can be installed on Ubuntu with a single command:
The first thing that needs to be done is to specify which interface the DHCP server should listen for requests on. This option is found in the file "/etc/default/isc-dhcp-server". In my case, the name of the internal interface from which I am hosting the subnet is enp3s0. Once that is done, the next step is to configure the DHCP server itself. DHCP configuration is found in "/etc/dhcp/dhcpd.conf". The default file has a rich set of example configuration options, from which it is pretty easy to infer your own configuration, but here is a basic rundown of my setup.
First I set the addresses to which the dhcp server should forward DNS requests. 8.8.8.8 and 8.8.4.4 are DNS servers hosted by google:
Then I added a section declaring a subnet. I set the default gateway to match the static IP address assigned to the internal ethernet nic chosen earlier:
Finally, for each of the "important" machines on the network, like the cluster nodes and my desktop machine (which serves as the head node), I added a section specifying a static IP address based on that devices mac address:
CLOSING THOUGHTS:
As a final step, I connected the cluster switch to the internal interface server, then reconfigured the cluster nodes and my desktop (which serves as the head node for the cluster) to use DHCP for IP address configuration. The DHCP server diligently handed out the static IPs I assigned, and I was off to the races. No more address allocation silliness.
8 or so months on, I have more or less forgotten about the router machine. It sits in the corner of the room quietly doing it's job. Better still, when I moved about 6 months ago, I was able to just plug everything back together as was, and it handled everything without any reconfiguration. It would probably have been more sensible to use a dedicated firewall/router operating system like pfsense, rather than roll my own. But I say if it aint' broke, don't fix it.
If you are interested in building something like this yourself though, the full dhcpd.conf that I ended up using in my server can be viewed here. I have also provided a template iptables configuration file, which you can find here. It is in a format such that once appropriately modified, it should be readable with the iptables-restore command.
The protocol responsible for handing out addresses on an IP network is called DHCP (dynamic host configuration protocol). When a device connects to a network, one of the first things it does is broadcast a DHCP request. A a listening DHCP server then responds with an IP address for the device to use, ensuring that it doesn't conflict with any already allocated addresses. It stands to reason then that there can only be one DHCP server on a network at any given time, or there will be trouble. So simply running my own DHCP server on the local network wasn't going to cut it. It was through this conundrum that I got my first real taste of networking. I ended up going down something of a rabbit hole to solve this one and building my own little subnet and router/firewall server, and ended up learning a lot along the way.
THE HARDWARE:
If you are looking to build your own router, hardware wise all you really need is a system with two or more network interfaces. It really is that simple. If you want to connect it directly to a broadband connection, you might need a modem card or similar device. In my case though, my router was going to host a subnet inside an existing gigabit Ethernet network, so all I needed was a machine with two or more ordinary Ethernet ports. The machine was built using spare parts bolted to a plank of wood. For the operating system, I broke out my trusty Ubuntu server 16.04 install USB. The resulting machine was quiet, low power and suitable for 24/7 operation. There really isn't much more to tell.
 |
| 1) My makeshift DHCP/NAT server |
NETWORK ADDRESS TRANSLATION:
The first piece of the puzzle was NAT. Broadly speaking, what NAT, aka network address translation, does is allow a single machine to expose services hosted on many separate machines via a single network interface. Effectively allowing many machines to share one IP address. This technology is extremely widely deployed, thanks in part to the global shortage of ipv4 addresses. It has become the standard solution for building local networks and connecting many machines to the internet via a single IP. It's used by businesses, home routers, and sometimes even by ISPs, with the (somewhat controversial) carrier-grade NAT. It also acts as a sort of firewall that prevents arbitrary traffic from the internet from making its way to machines on local networks. Although that isn't really it's purpose, and it's generally not considered a security feature.
As mentioned previously, to set up a NAT/firewall server, you first need a machine with two network interfaces, one to face the external network or WAN (wide area network), and another to host your subnet. The internal interface is called the "gateway" and is where machines within the subnet looking to connect to external addresses will send their packets. The NAT server then dutifully forwards those packets on to their intended destinations. Afterwards, any returning packets are directed back to the machine that initiated the connection. For incoming connections, things work a bit differently. The NAT server uses the port number to decide to which device on the subnet the connection should go to in a process known as port forwarding. If you've ever set up a service on your home internet connection, you will be familiar with this process.
The first thing to do is to configure the network interfaces. In most graphical desktop operating systems, networking is handled automagically by a network manager widget that usually resides somewhere on the desktop. In the case of Ubuntu server however, the raising of network interfaces is handled at boot time based on the contents of a configuration file (/etc/network/interfaces). Mine ended up looking like this:
The first interface "lo" is the loop-back interface. This is a virtual interface used for TCP/IP communication between processes running locally, and is not really relevant here. The second interface (enp3s0 in my case, though it varies) is the internal network interface, the one that faces the soon to be intranet. I configured it to use the IP address 10.10.10.1, and will serve as the gateway for local machines. Now you might be thinking, what prevents someone from choosing an IP that conflicts with something on the wider network? And the short answer is ostensibly nothing. There are however a few IP address ranges which are set aside for private networks, of which 10.*.*.* is one. The third interface (enp2s0) is the external interface that will connect to the home router, and then subsequently to the internet. This one has been configured to automatically get an address from the parent network, just as any normal device would.
The next step is to enable kernel IP forwarding. What this means is that if a packet comes into a network interface that matches the subnet of another interface, it will be re-transmitted from that interface. Naturally, this is something that we want. By default though, this behaviour is disabled. Enabling it can be done done using the following command:
This change doesn't persist through reboots however. To make it so that it does, the appropriate line (net.ipv4.ip_forward=1) has to be un-commented in "/etc/sysctl.conf".
The next step was to actually configure NAT. For this I used iptables. Iptables works by matching incoming (or outgoing packets) to lists of user-defined rules, each of which represents a routing decision. Usually, whether or not to block the packet in question. Rules are organised into tables and chains which are selected depending on the stage a packet is at in the routing process. To set up NAT, I used the following commands:
The first appends a rule to the forward chain which accepts packets coming in to the subnet (from enp2s0 to enp3s0), but only if they are part of an established connection. The second command appends another rule to the forward chain, allowing any packets leaving the subnet (from enp3s0 to enp2s0) to go to the wider internet. The third and final rule "caps off" the forward chain by blocking any packets that don't match either of the previous two rules. The final command actually does the business, and sets a rule on the postrouting chain in the NAT table to take any forwarded packets and transform them using the "masquerade" algorithm, a form of NAT.
Lastly, you probably want your iptables configuration to persist across reboots. Which it doesn't do by default. There is a package under Ubuntu called iptables-persistent which will automatically save your iptables rules and restore them on reboot. The following commands save iptables rules to the appropriate file and install the package:
DHCP:
With a subnet and router set up, the final task was to configure a DHCP server to hand out addresses. Thankfully, package managers come to the rescue again, and the DHCP server can be installed on Ubuntu with a single command:
The first thing that needs to be done is to specify which interface the DHCP server should listen for requests on. This option is found in the file "/etc/default/isc-dhcp-server". In my case, the name of the internal interface from which I am hosting the subnet is enp3s0. Once that is done, the next step is to configure the DHCP server itself. DHCP configuration is found in "/etc/dhcp/dhcpd.conf". The default file has a rich set of example configuration options, from which it is pretty easy to infer your own configuration, but here is a basic rundown of my setup.
First I set the addresses to which the dhcp server should forward DNS requests. 8.8.8.8 and 8.8.4.4 are DNS servers hosted by google:
Then I added a section declaring a subnet. I set the default gateway to match the static IP address assigned to the internal ethernet nic chosen earlier:
Finally, for each of the "important" machines on the network, like the cluster nodes and my desktop machine (which serves as the head node), I added a section specifying a static IP address based on that devices mac address:
CLOSING THOUGHTS:
As a final step, I connected the cluster switch to the internal interface server, then reconfigured the cluster nodes and my desktop (which serves as the head node for the cluster) to use DHCP for IP address configuration. The DHCP server diligently handed out the static IPs I assigned, and I was off to the races. No more address allocation silliness.
8 or so months on, I have more or less forgotten about the router machine. It sits in the corner of the room quietly doing it's job. Better still, when I moved about 6 months ago, I was able to just plug everything back together as was, and it handled everything without any reconfiguration. It would probably have been more sensible to use a dedicated firewall/router operating system like pfsense, rather than roll my own. But I say if it aint' broke, don't fix it.
If you are interested in building something like this yourself though, the full dhcpd.conf that I ended up using in my server can be viewed here. I have also provided a template iptables configuration file, which you can find here. It is in a format such that once appropriately modified, it should be readable with the iptables-restore command.
Comments
Post a Comment