Debugging an ARP issue (no resp to ARP requests)
Hi All, Please let me know if this is not the correct ML for such a question (or if there's a more appropriate list for it). I'm currently debugging an issue where Linux is not responding to ARP requests (testing with custom network interface h/w). I have 2 network interfaces which are basically interfaces on a custom network device sitting on the PCI bus. My setup looks like follows: xeth0 - 192.168.2.1 xeth2 - 192.168.2.2 xeth0/2 are the interfaces created for the custom device (via a lkm) Both these interfaces are on the local machine and are connected back-to-back on the custom device, so packets sent from one arrive on the other. I'm testing this setup by sending a single ARP pack, forced out of one interface, for the other interface's IP. $ sudo arping -c 1 -i xeth0 192.168.2.2 ARPING 192.168.2.2 --- 192.168.2.2 statistics --- 1 packets transmitted, 0 packets received, 100% unanswered (0 extra) If I start tcpdump on the receive interface (xeth2), I see the ARP packet arrive, but I don't see any response go out (I've attached the pcap file for anyone interested in looking at the received packet). The ARP packet's contents seem to be proper under wireshark. Is there any statistics I can look at to see if the packet was dropped by linux during ARP processing? I've put debug stmts in the LKM, but I don't see the kernel IP layer calling the xmit function for an ARP reply (however the xmit function for the ARP request, from xeth0, is being called). I don't see any drops in "netstat -s" output as well. Any hints appreciated. Thanks, -mandeep
Hi, On Mon, Jan 19, 2015 at 03:42:58PM -0800, Mandeep Sandhu wrote:
Hi All,
Please let me know if this is not the correct ML for such a question (or if there's a more appropriate list for it).
I'm currently debugging an issue where Linux is not responding to ARP requests (testing with custom network interface h/w).
I have 2 network interfaces which are basically interfaces on a custom network device sitting on the PCI bus.
My setup looks like follows:
xeth0 - 192.168.2.1 xeth2 - 192.168.2.2
xeth0/2 are the interfaces created for the custom device (via a lkm)
Both these interfaces are on the local machine and are connected back-to-back on the custom device, so packets sent from one arrive on the other.
I'm testing this setup by sending a single ARP pack, forced out of one interface, for the other interface's IP.
$ sudo arping -c 1 -i xeth0 192.168.2.2 ARPING 192.168.2.2
--- 192.168.2.2 statistics --- 1 packets transmitted, 0 packets received, 100% unanswered (0 extra)
If I start tcpdump on the receive interface (xeth2), I see the ARP packet arrive, but I don't see any response go out (I've attached the pcap file for anyone interested in looking at the received packet). The ARP packet's contents seem to be proper under wireshark.
Is there any statistics I can look at to see if the packet was dropped by linux during ARP processing?
I've put debug stmts in the LKM, but I don't see the kernel IP layer calling the xmit function for an ARP reply (however the xmit function for the ARP request, from xeth0, is being called).
Have you tried to put debug prints in arp_process function?
I don't see any drops in "netstat -s" output as well.
Any hints appreciated.
Just a random guess, but try to disable reverse path filtering (rp_filter).
Thanks, -mandeep
????w??T?g<<???????~e?~e???? _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Check arp_* fields in https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt <https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt> -daveti
On Jan 19, 2015, at 7:11 PM, Mike Krinkin <krinkin.m.u@gmail.com> wrote:
Hi,
On Mon, Jan 19, 2015 at 03:42:58PM -0800, Mandeep Sandhu wrote:
Hi All,
Please let me know if this is not the correct ML for such a question (or if there's a more appropriate list for it).
I'm currently debugging an issue where Linux is not responding to ARP requests (testing with custom network interface h/w).
I have 2 network interfaces which are basically interfaces on a custom network device sitting on the PCI bus.
My setup looks like follows:
xeth0 - 192.168.2.1 xeth2 - 192.168.2.2
xeth0/2 are the interfaces created for the custom device (via a lkm)
Both these interfaces are on the local machine and are connected back-to-back on the custom device, so packets sent from one arrive on the other.
I'm testing this setup by sending a single ARP pack, forced out of one interface, for the other interface's IP.
$ sudo arping -c 1 -i xeth0 192.168.2.2 ARPING 192.168.2.2
--- 192.168.2.2 statistics --- 1 packets transmitted, 0 packets received, 100% unanswered (0 extra)
If I start tcpdump on the receive interface (xeth2), I see the ARP packet arrive, but I don't see any response go out (I've attached the pcap file for anyone interested in looking at the received packet). The ARP packet's contents seem to be proper under wireshark.
Is there any statistics I can look at to see if the packet was dropped by linux during ARP processing?
I've put debug stmts in the LKM, but I don't see the kernel IP layer calling the xmit function for an ARP reply (however the xmit function for the ARP request, from xeth0, is being called).
Have you tried to put debug prints in arp_process function?
I don't see any drops in "netstat -s" output as well.
Any hints appreciated.
Just a random guess, but try to disable reverse path filtering (rp_filter).
Thanks, -mandeep
????w??T?g<<???????~e?~e???? _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org <mailto:Kernelnewbies@kernelnewbies.org> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies <http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies>
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org <mailto:Kernelnewbies@kernelnewbies.org> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies <http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies>
On Mon, Jan 19, 2015 at 4:21 PM, Dave Tian <dave.jing.tian@gmail.com> wrote:
Check arp_* fields in https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
Thanks! This looks helpful.
-daveti
On Jan 19, 2015, at 7:11 PM, Mike Krinkin <krinkin.m.u@gmail.com> wrote:
Hi,
On Mon, Jan 19, 2015 at 03:42:58PM -0800, Mandeep Sandhu wrote:
Hi All,
Please let me know if this is not the correct ML for such a question (or if there's a more appropriate list for it).
I'm currently debugging an issue where Linux is not responding to ARP requests (testing with custom network interface h/w).
I have 2 network interfaces which are basically interfaces on a custom network device sitting on the PCI bus.
My setup looks like follows:
xeth0 - 192.168.2.1 xeth2 - 192.168.2.2
xeth0/2 are the interfaces created for the custom device (via a lkm)
Both these interfaces are on the local machine and are connected back-to-back on the custom device, so packets sent from one arrive on the other.
I'm testing this setup by sending a single ARP pack, forced out of one interface, for the other interface's IP.
$ sudo arping -c 1 -i xeth0 192.168.2.2 ARPING 192.168.2.2
--- 192.168.2.2 statistics --- 1 packets transmitted, 0 packets received, 100% unanswered (0 extra)
If I start tcpdump on the receive interface (xeth2), I see the ARP packet arrive, but I don't see any response go out (I've attached the pcap file for anyone interested in looking at the received packet). The ARP packet's contents seem to be proper under wireshark.
Is there any statistics I can look at to see if the packet was dropped by linux during ARP processing?
I've put debug stmts in the LKM, but I don't see the kernel IP layer calling the xmit function for an ARP reply (however the xmit function for the ARP request, from xeth0, is being called).
Have you tried to put debug prints in arp_process function?
I don't see any drops in "netstat -s" output as well.
Any hints appreciated.
Just a random guess, but try to disable reverse path filtering (rp_filter).
Thanks, -mandeep
???? w??T?g <<?????? ?~e ?~e?? ?? _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Have you tried to put debug prints in arp_process function?
Not yet. I'm using the Ubuntu 14.10 kernel. I'm lazy and don't want to rebuild their kernel (an config), so trying to figure out if I can narrow down the problem without it! :)
I don't see any drops in "netstat -s" output as well.
Any hints appreciated.
Just a random guess, but try to disable reverse path filtering (rp_filter).
I just did actually. Unfortunately that didn't work either. Interestingly, I tried the same experiment with 2 "regular" ethernet interfaces on my machine and in that too the ARPs never resolve. This indicates that this looks like a config issue and IP/routing issue?
Thanks, -mandeep
???? w??T?g < < ?????? ?~e ?~e?? ?? _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-----Original Message----- From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies-bounces@kernelnewbies.org] On Behalf Of Mandeep Sandhu Sent: Monday, January 19, 2015 4:55 PM To: Mike Krinkin Cc: kernelnewbies Subject: Re: Debugging an ARP issue (no resp to ARP requests)
Have you tried to put debug prints in arp_process function?
Not yet. I'm using the Ubuntu 14.10 kernel. I'm lazy and don't want to rebuild their kernel (an config), so trying to figure out if I can narrow down the problem without it! :)
I don't see any drops in "netstat -s" output as well.
Any hints appreciated.
Just a random guess, but try to disable reverse path filtering (rp_filter).
I just did actually. Unfortunately that didn't work either.
Interestingly, I tried the same experiment with 2 "regular" ethernet interfaces on my machine and in that too the ARPs never resolve. This indicates that this looks like a config issue and IP/routing issue?
You'd mentioned earlier that your setup was this: xeth0 - 192.168.2.1 xeth2 - 192.168.2.2 That looks to me like two network interfaces on the same subnet, though that's my guess since you don't show the prefix lengths. I'm guessing the subnet on both is 192.168.2/24. Unless things have changed since the last time I looked into this, for IPv4 Linux implements what's referred to as the "weak" address binding model where IP addresses are considered to belong to the host not the interface. That means your host may be transmitting an ARP response, but not out the interface you expect, particularly if in fact you have the same subnet assigned to more than one interface. You might want to tcpdump on all interfaces when you do this. Also, you might want to include the output of the following in future posts: ip addr show ip route show ip neigh show Jeff Haran
You'd mentioned earlier that your setup was this:
xeth0 - 192.168.2.1 xeth2 - 192.168.2.2
That looks to me like two network interfaces on the same subnet, though that's my guess since you don't show the prefix lengths. I'm guessing the subnet on both is 192.168.2/24.
Correct. Netmask is /24.
Unless things have changed since the last time I looked into this, for IPv4 Linux implements what's referred to as the "weak" address binding model where IP addresses are considered to belong to the host not the interface. That means your host may be transmitting an ARP response, but not out the interface you expect, particularly if in fact you have the same subnet assigned to more than one interface.
You might want to tcpdump on all interfaces when you do this.
I actually did that (after some googling) but there's no ARP response being transmitted out of either of the interfaces. As I mentioned before, this problem happens even with regular ethernet interfaces and not specifically with my custom hardware related ones, so looks like a routing (mis)configuration issue.
Also, you might want to include the output of the following in future posts:
ip addr show ip route show ip neigh show
I'll now test with 2 "regular" ethernet interfaces on my test machine (eth0,eth1) and send the o/p of these commands. Thanks!
Jeff Haran
Here's how my current setup looks like: $ ifconfig eth4 eth4 Link encap:Ethernet HWaddr 74:fe:48:04:e7:eb inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 ... $ ifconfig eth5 eth5 Link encap:Ethernet HWaddr 74:fe:48:04:e7:e6 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 ... $ ip route show default via 192.168.1.2 dev eth4 192.168.1.0/24 dev eth4 proto kernel scope link src 192.168.1.1 192.168.1.0/24 dev eth5 proto kernel scope link src 192.168.1.2 I've deliberately added a default gateway to 192.168.1.2 (so that the ARP response is routed back through it). $ip neigh show (there's no neighbor entries) cat /proc/sys/net/ipv4/conf/all/rp_filter 0 cat /proc/sys/net/ipv4/conf/all/arp_filter 1 cat /proc/sys/net/ipv4/conf/eth4/rp_filter 0 cat /proc/sys/net/ipv4/conf/eth4/arp_filter 1 cat /proc/sys/net/ipv4/conf/eth5/rp_filter 0 cat /proc/sys/net/ipv4/conf/eth5/arp_filter 1 Even with these settings, ARP for 192.168.1.2, via eth4 (192.168.1.1) interface fails. On Mon, Jan 19, 2015 at 5:31 PM, Mandeep Sandhu <mandeepsandhu.chd@gmail.com> wrote:
You'd mentioned earlier that your setup was this:
xeth0 - 192.168.2.1 xeth2 - 192.168.2.2
That looks to me like two network interfaces on the same subnet, though that's my guess since you don't show the prefix lengths. I'm guessing the subnet on both is 192.168.2/24.
Correct. Netmask is /24.
Unless things have changed since the last time I looked into this, for IPv4 Linux implements what's referred to as the "weak" address binding model where IP addresses are considered to belong to the host not the interface. That means your host may be transmitting an ARP response, but not out the interface you expect, particularly if in fact you have the same subnet assigned to more than one interface.
You might want to tcpdump on all interfaces when you do this.
I actually did that (after some googling) but there's no ARP response being transmitted out of either of the interfaces. As I mentioned before, this problem happens even with regular ethernet interfaces and not specifically with my custom hardware related ones, so looks like a routing (mis)configuration issue.
Also, you might want to include the output of the following in future posts:
ip addr show ip route show ip neigh show
I'll now test with 2 "regular" ethernet interfaces on my test machine (eth0,eth1) and send the o/p of these commands.
Thanks!
Jeff Haran
What about arp_filter=0 && arp_ignore=0? -daveti
On Jan 19, 2015, at 9:14 PM, Mandeep Sandhu <mandeepsandhu.chd@gmail.com> wrote:
Here's how my current setup looks like:
$ ifconfig eth4 eth4 Link encap:Ethernet HWaddr 74:fe:48:04:e7:eb inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 ... $ ifconfig eth5 eth5 Link encap:Ethernet HWaddr 74:fe:48:04:e7:e6 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 ...
$ ip route show default via 192.168.1.2 dev eth4 192.168.1.0/24 dev eth4 proto kernel scope link src 192.168.1.1 192.168.1.0/24 dev eth5 proto kernel scope link src 192.168.1.2
I've deliberately added a default gateway to 192.168.1.2 (so that the ARP response is routed back through it).
$ip neigh show
(there's no neighbor entries)
cat /proc/sys/net/ipv4/conf/all/rp_filter 0 cat /proc/sys/net/ipv4/conf/all/arp_filter 1 cat /proc/sys/net/ipv4/conf/eth4/rp_filter 0 cat /proc/sys/net/ipv4/conf/eth4/arp_filter 1 cat /proc/sys/net/ipv4/conf/eth5/rp_filter 0 cat /proc/sys/net/ipv4/conf/eth5/arp_filter 1
Even with these settings, ARP for 192.168.1.2, via eth4 (192.168.1.1) interface fails.
On Mon, Jan 19, 2015 at 5:31 PM, Mandeep Sandhu <mandeepsandhu.chd@gmail.com> wrote:
You'd mentioned earlier that your setup was this:
xeth0 - 192.168.2.1 xeth2 - 192.168.2.2
That looks to me like two network interfaces on the same subnet, though that's my guess since you don't show the prefix lengths. I'm guessing the subnet on both is 192.168.2/24.
Correct. Netmask is /24.
Unless things have changed since the last time I looked into this, for IPv4 Linux implements what's referred to as the "weak" address binding model where IP addresses are considered to belong to the host not the interface. That means your host may be transmitting an ARP response, but not out the interface you expect, particularly if in fact you have the same subnet assigned to more than one interface.
You might want to tcpdump on all interfaces when you do this.
I actually did that (after some googling) but there's no ARP response being transmitted out of either of the interfaces. As I mentioned before, this problem happens even with regular ethernet interfaces and not specifically with my custom hardware related ones, so looks like a routing (mis)configuration issue.
Also, you might want to include the output of the following in future posts:
ip addr show ip route show ip neigh show
I'll now test with 2 "regular" ethernet interfaces on my test machine (eth0,eth1) and send the o/p of these commands.
Thanks!
Jeff Haran
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
arp_filter=0 is the default, so I'm sure I've tested it with this setting. I think I've tested with arp_ignore=0 too, although with trying so many different combinations, I might've missed it out. I'll keep this in mind when testing next. Thanks. On Mon, Jan 19, 2015 at 10:14 PM, Dave Tian <dave.jing.tian@gmail.com> wrote:
What about arp_filter=0 && arp_ignore=0?
-daveti
On Jan 19, 2015, at 9:14 PM, Mandeep Sandhu <mandeepsandhu.chd@gmail.com> wrote:
Here's how my current setup looks like:
$ ifconfig eth4 eth4 Link encap:Ethernet HWaddr 74:fe:48:04:e7:eb inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 ... $ ifconfig eth5 eth5 Link encap:Ethernet HWaddr 74:fe:48:04:e7:e6 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 ...
$ ip route show default via 192.168.1.2 dev eth4 192.168.1.0/24 dev eth4 proto kernel scope link src 192.168.1.1 192.168.1.0/24 dev eth5 proto kernel scope link src 192.168.1.2
I've deliberately added a default gateway to 192.168.1.2 (so that the ARP response is routed back through it).
$ip neigh show
(there's no neighbor entries)
cat /proc/sys/net/ipv4/conf/all/rp_filter 0 cat /proc/sys/net/ipv4/conf/all/arp_filter 1 cat /proc/sys/net/ipv4/conf/eth4/rp_filter 0 cat /proc/sys/net/ipv4/conf/eth4/arp_filter 1 cat /proc/sys/net/ipv4/conf/eth5/rp_filter 0 cat /proc/sys/net/ipv4/conf/eth5/arp_filter 1
Even with these settings, ARP for 192.168.1.2, via eth4 (192.168.1.1) interface fails.
On Mon, Jan 19, 2015 at 5:31 PM, Mandeep Sandhu <mandeepsandhu.chd@gmail.com> wrote:
You'd mentioned earlier that your setup was this:
xeth0 - 192.168.2.1 xeth2 - 192.168.2.2
That looks to me like two network interfaces on the same subnet, though that's my guess since you don't show the prefix lengths. I'm guessing the subnet on both is 192.168.2/24.
Correct. Netmask is /24.
Unless things have changed since the last time I looked into this, for IPv4 Linux implements what's referred to as the "weak" address binding model where IP addresses are considered to belong to the host not the interface. That means your host may be transmitting an ARP response, but not out the interface you expect, particularly if in fact you have the same subnet assigned to more than one interface.
You might want to tcpdump on all interfaces when you do this.
I actually did that (after some googling) but there's no ARP response being transmitted out of either of the interfaces. As I mentioned before, this problem happens even with regular ethernet interfaces and not specifically with my custom hardware related ones, so looks like a routing (mis)configuration issue.
Also, you might want to include the output of the following in future posts:
ip addr show ip route show ip neigh show
I'll now test with 2 "regular" ethernet interfaces on my test machine (eth0,eth1) and send the o/p of these commands.
Thanks!
Jeff Haran
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Mandeep Sandhu <mandeepsandhu.chd@gmail.com> writes:
Hi All,
Please let me know if this is not the correct ML for such a question (or if there's a more appropriate list for it).
I'm currently debugging an issue where Linux is not responding to ARP requests (testing with custom network interface h/w).
I have 2 network interfaces which are basically interfaces on a custom network device sitting on the PCI bus.
My setup looks like follows:
xeth0 - 192.168.2.1 xeth2 - 192.168.2.2
xeth0/2 are the interfaces created for the custom device (via a lkm)
Both these interfaces are on the local machine and are connected back-to-back on the custom device, so packets sent from one arrive on the other.
This won't work. Linux will consider both addresses as local to the host and will never respond to arp requests from any of those addresses. See http://en.wikipedia.org/wiki/Host_model You might be able to change this by tuning arp_announce/arp_ignore - But I don't know if that changes the behaviour wrt responses to local addresses... See https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt for the meaning of these knobs. If your main interest is to test how Linux (or your driver/hardware) responds to ARPs, then it's definitely easiest/best to use two Linux hosts. This might of course be two virtual hosts running on the same hardware if you have hardware restrictions. Bjørn
This won't work. Linux will consider both addresses as local to the host and will never respond to arp requests from any of those addresses.
According to this Linux defaults to the 'weak host' model (I don't know if Ubuntu changes this default). Although I don't know why this will matter because I'm sending packets destined for the IP of that specific interface (which should work in strong and weak models). I think I'll sift through the ARP processing code to see if there are points where it can drop a packet.
You might be able to change this by tuning arp_announce/arp_ignore - But I don't know if that changes the behaviour wrt responses to local addresses... See https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt for the meaning of these knobs.
I'm playing around with different combinations of anything even remotely connected to ARP and routing. Nothing's changed till now. Thanks for your time. -mandeep
participants (5)
-
Bjørn Mork -
Dave Tian -
Jeff Haran -
Mandeep Sandhu -
Mike Krinkin