Suppose that we have two servers: S1 and S2. Server S1 has one network interface, server S2 has 3 network interfaces. All network interfaces are in the same IPv4 network. Servers are communicating with each other using all the IP addresses. However, without additional tuning, you’ll see that S2 can receive packets on one interface and answer from another.
For example, suppose that you have a setup as described in the schema above. If you issue a ping from S1 to 10.1.0.11, S2 server can reply from eth0 or eth2. Also, the ARP table on S1 will show incorrect entries for S2 addresses.
So, how can we overcome the above issues and configure Linux to provide a desirable behavior?
Tune ARP protocol
Linux has a few parameters for ARP protocol tuning. The most important for our task are arp_filter and arp_ignore.
arp_filter – BOOLEAN
- 0 – (default) The kernel can respond to ARP requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces. This behavior can cause problems, but only for more complex setups, like those that implement load-balancing capabilities.
- 1 – Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP’d IP out that interface (therefore you must use source-based routing for this to work). In other words, it allows control of which cards (usually 1) will respond to an ARP request.
The arp_filter is primarily used when multiple interfaces reside in the same subnet and is used to allow/disallow which interfaces respond to ARP requests.
arp_ignore – INTEGER
Define different modes for sending replies in response to received ARP requests that resolve local target IP addresses:
- 0 – (default) Reply to any local target IP address, configured on any interface.
- 1 – Reply only if the target IP address is a local address configured on the incoming interface.
- 2 – Reply only if the target IP address is a local address configured on the incoming interface and both with the sender’s IP address are part of the same subnet on this interface.
- 3 – Do not reply to local addresses configured with scope host, only resolutions for global and link addresses are replied to.
- 4-7 – Reserved
- 8 – Do not reply to all local addresses.
The default arp_ignore parameter allows the device to reply to an ARP request for any IP address on any interface. While this matches the expectation that an IP address belongs to the device, not an interface, it can cause some unexpected and undesirable behavior on a router.
For example, with the arp_ignore parameter set to 0, if an ARP request is received on one interface for the IP address residing on a different interface, the switch will respond with an ARP reply even if the interface of the target address is down. This can cause a loss of traffic due to incorrect understanding about the reachability of next-hops, and also makes troubleshooting extremely challenging in some failure conditions.
So we need to set:
arp_filter = 1
arp_ignore = 1
It can be done with commands on S2:
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
After setting these options, S2 will answer ARP requests from a valid interface and S1 will have a valid ARP table.
Reply from the same interface
After tuning ARP we are still left with one problem. If you ping any S2 address from S1 you can note that sometimes you get ICMP reply from another interface. Let’s fix it.
We need to create an ip rule with IP addresses table and configure it for all interfaces:
echo 200 eth0t >> /etc/iproute2/rt_tables
echo 201 eth1t >> /etc/iproute2/rt_tables
echo 202 eth2t >> /etc/iproute2/rt_tables
ip rule add from 10.1.0.10 table eth0t
ip route add 10.1.0.0/24 dev eth0 table eth0t
ip rule add from 10.1.0.11 table eth1t
ip route add 10.1.0.0/24 dev eth1 table eth1t
ip rule add from 10.1.0.12 table eth2t
ip route add 10.1.0.0/24 dev eth2 table eth2t
You can check what interface is selected with commands:
ip route get 10.1.0.1 from 10.1.0.10
ip route get 10.1.0.1 from 10.1.0.11
ip route get 10.1.0.1 from 10.1.0.12
In all cases, you should see the correct interface for the corresponding IP address.
