Linux Tutorial: A Machine with Two IPs in the Same Subnet on Gentoo
This is the first in a hopefully long series exposing some internal configuration challenges I have come up against over time. Hopefully it will help other people with similar issues.
While building out a number of Linux based server farms, I ran into several interesting Linux networking configuration based architecture issues. In many of our setups, we have each machine connected to two separate switches. If one switch / Ethernet cable / Ethernet port goes down, the other one takes over. The thought was to initially do this with OSPF but while the idea worked, it didn't play out well in the real world. So we found other ways of migrating traffic from Ethernet port to Ethernet port.
The simplest way to do this was to move IP aliases from device to device but we ran into issues if the primary address on both Ethernet interfaces were within the same IP subnet. The kernel would always choose one of the interfaces to source traffic regardless of which IP the traffic was "from". As it turns out, the kernel consults the routing table to find the device to use to send a packet and the destination address was all that was being used to pick the device. Because the default route is on exactly one device, that device was being used to send all traffic, regardless if that device had the IP we were sending from or not. I needed a way to take the source into consideration before selecting which interface to use to send a packet.
Read on to see how this is done.
My machines have two Ethernet interfaces, eth0 and eth1, which are plugged into two switches. VLAN 10 (realnet) and 11 (fakenet) exist on both switches and all Ethernet ports are in trunk mode so they can participate in both VLANs. I set this up this way so if I were to loose one of the Ethernets, I would still have access to the public and private networks through VLANs on the other Ethernet. Because the IP addresses on each interface are within the same /24 network, special routing rules need to be set up to allow packets sourced from each IP to flow from its respective interface.
Take, for example, a box with this /etc/conf.d/net setup:
(box:~) # cat /etc/conf.d/net
vlans_eth0="10 11"
vlans_eth1="10 11"
config_eth0=( "null" )
config_eth0_10=( "1.2.3.16/24" )
config_eth0_11=( "10.1.1.16/24" )
config_eth1=( "null" )
config_eth1_10=( "1.2.3.17/24" )
config_eth1_11=( "10.1.1.17/24" )
routes_eth0_10=( "default via 1.2.3.1" )
routes_eth0_11=( "10.0.0.0/8 via 10.1.1.1" )
The kernel will always choose eth0 to send data even if the source address is 1.2.3.17 which is bound to eth1. To fix this, we create a set of 4 route tables, one for every possible VLAN / Ethernet combination.
(box:~) # cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
1 e0v10
2 e0v11
3 e1v10
4 e1v11
Then we have routes for each of the 4 route tables directing each to the appropriate interface. (these are set up though /etc/conf.d/net in Gentoo)
(box:~) # ip route show table e0v10
1.2.3.0/24 dev eth0.10 scope link
default via 1.2.3.2 dev eth0.10
(box:~) # ip route show table e1v10
1.2.3.0/24 dev eth1.10 scope link
default via 1.2.3.3 dev eth1.10
(box:~) # ip route show table e0v11
10.1.1.0/24 dev eth0.11 scope link
10.0.0.0/8 via 10.1.1.1 dev eth0.11
(box:~) # ip route show table e1v11
10.1.1.0/24 dev eth1.11 scope link
10.0.0.0/8 via 10.1.1.1 dev eth1.11
Lastly we have to create some rules to tell the kernel to use the appropriate route table for each source IP it comes across. (NOTE: we need to create rules for aliased addresses when we add them as well)
/sbin/ip rule add from 1.2.3.16 table e0v10
/sbin/ip rule add from 10.1.1.16 table e0v11
/sbin/ip rule add from 1.2.3.17 table e1v10
/sbin/ip rule add from 10.1.1.17 table e1v11
Because Gentoo networking scripts don’t yet support this, we have to put these commands into /etc/conf.d/local.start instead.
Now, for example, packets from 1.2.3.17 follow table e1v10 and get routed out of eth1.10. Without this, packets from 1.2.3.17 would flow out of eth0.10 instead because the kernel notices eth0.10 has the 1.2.3.0/24 network and won't look any further for a closer match. The other way to do this would be to scope the interface down to 1.2.3.17/32 but then all packets to other machines in the /24 would be sent to the gateway instead of going direct.
When aliasing a production IP address, you can set it to run through any Ethernet device or VLAN that you want but you SHOULD keep it on the Ethernet device and VLAN that the alias is closest to. The Linux kernel is smart enough to route through any device you want though, but you won’t have much luck with a real address on the fakenet VLAN.
Special Considerations:
Packets that the kernel is asked to make that don't specify a from address won't follow one of the above route rules so we have to take care of this case. If the from address is not specified, the IP address on the interface with the best match for reaching the destination (usually the device used by the default route) will be used. So we must also have our default route in the main route table. This is done in /etc/conf.t/net but if we were to do it on the command line it would look like this:
(box:~) # ip route add default via 1.2.3.1
Here's the main route table with a default route for packets that don't specify a from address:
(box:~) # ip route show
10.1.1.0/24 dev eth0.11 proto kernel scope link src 10.1.1.16
10.1.1.0/24 dev eth1.11 proto kernel scope link src 10.1.1.17
1.2.3.0/24 dev eth0.10 proto kernel scope link src 1.2.3.16
1.2.3.0/24 dev eth1.10 proto kernel scope link src 1.2.3.17
127.0.0.0/8 dev lo scope link
default via 1.2.3.1 dev eth0.10
The gateway we use in our default route is actually an aliased (VRRP) address (.1) that the switches manage. The actual addresses of the switches are .3 and .2. If the switch aliasing .1 goes down, the other switch starts aliasing this address. It therefore makes sense to set the default gateway for each Ethernet interface to the real address of the switch it is connected to so there are no unnecessary switch to switch dependencies. In the event of a single switch outage, the aliased address on the local machine that a service is using could be migrated to the other Ethernet connected to the good switch. If the application doesn't specify a from address, we rely on the working switch to alias .1 so our default route works as well.
Although possible, it usually doesn't make sense to re-point the real IP addresses on the bad Ethernet, so using the VRRP address in anything but the main route table is not worth it.
Tags
Linux Tutorial Two IPs in the Same Subnet GentooTrackbacks
To send a trackback, use the URL of this story appending ?page=tb at the end.Comments (0)
Leave a Comment
To create links in comments:
[link:http://www.anders.com/] becomes http://www.anders.com/
[link:http://www.anders.com/|Anders.com] becomes Anders.com
Notice there is no rel="nofollow" in these hrefs. Links in comments will carry page rank from this site so only link to things worthy of people's attention.
subscribe