Path MTU discovery and bridged ADSL PPPoE
Hi, This is perhaps more of a networking question than Linux, but all the devices involved are running Linux.. I have what is probably a common home setup -- an ADSL modem operating in "pure bridge" mode, connected by ethernet to a Linux server running pppd with the PPPoE plugin. Same linux server is providing DNS, DHCP and NAT for other clients connected via wireless and ethernet. This setup was running fine for many years with a Billion 7300RA ADSL modem. I just switched it over to a newer Billion 7800NL modem in the hope it'd provide better sync speeds on my hopeless phone line. (It does, with some SNR tweaking) However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492. I'm pretty confused by this -- pppd was setting itself up with a 1492 mtu even on the old modem, and that managed to propagate out just fine. The only thing that changed was the modem, but that shouldn't affect this issue. I'm bamboozled. Could anyone advise me on what could possibly be done to figure out what has happened? Or better, a way to fix it? Investigative notes: 1) New router was switched back to standalone mode (where it does pppoe, dhcp, nat etc) and in this mode, all clients worked fine. So the ADSL link and ethernet ports seem OK. 2) Old router was booted up to verify settings and that stuff works OK with it. Settings for ADSL line were similar, and stuff does work with it. (I say settings were "similar" because the two modems have different interfaces, and the 7800NL has some extra options not present on the 7300RA, but I don't think they're relevant to this issue) Thanks in advance, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
On 26/09/13 10:48, Toby Corkindale wrote:
Hi, This is perhaps more of a networking question than Linux, but all the devices involved are running Linux..
I have what is probably a common home setup -- an ADSL modem operating in "pure bridge" mode, connected by ethernet to a Linux server running pppd with the PPPoE plugin. Same linux server is providing DNS, DHCP and NAT for other clients connected via wireless and ethernet.
This setup was running fine for many years with a Billion 7300RA ADSL modem. I just switched it over to a newer Billion 7800NL modem in the hope it'd provide better sync speeds on my hopeless phone line. (It does, with some SNR tweaking)
However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492.
I'm pretty confused by this -- pppd was setting itself up with a 1492 mtu even on the old modem, and that managed to propagate out just fine. The only thing that changed was the modem, but that shouldn't affect this issue.
I'm bamboozled. Could anyone advise me on what could possibly be done to figure out what has happened? Or better, a way to fix it?
Investigative notes: 1) New router was switched back to standalone mode (where it does pppoe, dhcp, nat etc) and in this mode, all clients worked fine. So the ADSL link and ethernet ports seem OK. 2) Old router was booted up to verify settings and that stuff works OK with it. Settings for ADSL line were similar, and stuff does work with it. (I say settings were "similar" because the two modems have different interfaces, and the 7800NL has some extra options not present on the 7300RA, but I don't think they're relevant to this issue)
Thanks in advance, Toby
Wild guess - TCP/UDP offload is generating bigger segments because the new modem accepts offload. On the interface that connects the server to the modem, issue ethtool -K <interface> tso off ethtool -K <interface> uso off ethtool -K <interface> gso off
On 26 September 2013 10:54, Keith Owens <kaos@ocs.com.au> wrote:
On 26/09/13 10:48, Toby Corkindale wrote:
However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492.
Wild guess - TCP/UDP offload is generating bigger segments because the new modem accepts offload. On the interface that connects the server to the modem, issue
ethtool -K <interface> tso off ethtool -K <interface> uso off ethtool -K <interface> gso off
Interesting theory. Tested, no change though.
In case it's of assistance, here's a tcpdump on a client machine, with it attempting to grab a deb via http from an ubuntu repo: Curiously, I see that there is a line where it receives the need-to-fragment message - it then stalls a bit later having received a couple of packets. 01:03:53.507791 IP 10.23.1.250.42126 > 255.255.255.255.7437: UDP, length 173 01:03:56.346189 IP penfold.dryft.net.17500 > 10.23.1.255.17500: UDP, length 165 01:03:56.346381 IP precise.35028 > penfold.dryft.net.domain: 32155+ PTR? 255.1.23.10.in-addr.arpa. (42) 01:03:56.346459 IP penfold.dryft.net.domain > precise.35028: 32155 NXDomain* 0/0/0 (42) 01:03:56.543791 IP 10.23.1.250.42126 > 255.255.255.255.7437: UDP, length 173 01:03:59.579831 IP 10.23.1.250.42126 > 255.255.255.255.7437: UDP, length 173 01:04:02.615831 IP 10.23.1.250.42126 > 255.255.255.255.7437: UDP, length 173 01:04:05.022421 IP precise.38845 > penfold.dryft.net.domain: 3937+ A? archive.ubuntu.com. (36) 01:04:05.022505 IP penfold.dryft.net.domain > precise.38845: 3937 9/0/0 A 91.189.91.13, A 91.189.92.202, A 91.189.92.201, A 91.189.92.200, A 91.189.92.177, A 91.189.92.176, A 91.189.92.156, A 91.189.91.15, A 91.189.91.14 (180) 01:04:05.022927 IP precise.52759 > penfold.dryft.net.domain: 714+ A? archive.ubuntu.com. (36) 01:04:05.022979 IP penfold.dryft.net.domain > precise.52759: 714 9/0/0 A 91.189.91.14, A 91.189.91.13, A 91.189.92.202, A 91.189.92.201, A 91.189.92.200, A 91.189.92.177, A 91.189.92.176, A 91.189.92.156, A 91.189.91.15 (180) 01:04:05.023466 IP precise.58912 > orobas.canonical.com.http: Flags [S], seq 3617406778, win 14600, options [mss 1460,sackOK,TS val 10552514 ecr 0,nop,wscale 7], length 0 01:04:05.023602 IP precise.42947 > penfold.dryft.net.domain: 15440+ PTR? 14.91.189.91.in-addr.arpa. (43) 01:04:05.215907 IP penfold.dryft.net.domain > precise.42947: 15440 1/0/0 PTR orobas.canonical.com. (77) 01:04:05.294031 IP orobas.canonical.com.http > precise.58912: Flags [S.], seq 2366925927, ack 3617406779, win 14480, options [mss 1460,sackOK,TS val 680841089 ecr 10552514,nop,wscale 8], length 0 01:04:05.294068 IP precise.58912 > orobas.canonical.com.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 10552581 ecr 680841089], length 0 01:04:05.294481 IP precise.58912 > orobas.canonical.com.http: Flags [.], seq 1:1449, ack 1, win 115, options [nop,nop,TS val 10552581 ecr 680841089], length 1448 01:04:05.294515 IP penfold.dryft.net > precise: ICMP orobas.canonical.com unreachable - need to frag (mtu 1492), length 556 01:04:05.294626 IP precise.58912 > orobas.canonical.com.http: Flags [P.], seq 1449:2014, ack 1, win 115, options [nop,nop,TS val 10552581 ecr 680841089], length 565 01:04:05.294656 IP precise.58912 > orobas.canonical.com.http: Flags [.], seq 1:1441, ack 1, win 115, options [nop,nop,TS val 10552581 ecr 680841089], length 1440 01:04:05.294661 IP precise.58912 > orobas.canonical.com.http: Flags [.], seq 1441:1449, ack 1, win 115, options [nop,nop,TS val 10552581 ecr 680841089], length 8 01:04:05.570168 IP orobas.canonical.com.http > precise.58912: Flags [.], ack 1, win 57, options [nop,nop,TS val 680841158 ecr 10552581,nop,nop,sack 1 {1449:2014}], length 0 01:04:05.592570 IP orobas.canonical.com.http > precise.58912: Flags [.], ack 1441, win 68, options [nop,nop,TS val 680841163 ecr 10552581,nop,nop,sack 1 {1449:2014}], length 0 01:04:05.616707 IP orobas.canonical.com.http > precise.58912: Flags [.], ack 2014, win 68, options [nop,nop,TS val 680841163 ecr 10552581], length 0 01:04:05.651831 IP 10.23.1.250.42126 > 255.255.255.255.7437: UDP, length 173 01:04:08.687872 IP 10.23.1.250.42126 > 255.255.255.255.7437: UDP, length 173
Well, putting this iptables rule at the head of my FORWARD chain solves the problem: iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu I still have absolutely no idea why this worked fine on one router, and fails on another though! On 26 September 2013 10:48, Toby Corkindale <toby@dryft.net> wrote:
Hi, This is perhaps more of a networking question than Linux, but all the devices involved are running Linux..
I have what is probably a common home setup -- an ADSL modem operating in "pure bridge" mode, connected by ethernet to a Linux server running pppd with the PPPoE plugin. Same linux server is providing DNS, DHCP and NAT for other clients connected via wireless and ethernet.
This setup was running fine for many years with a Billion 7300RA ADSL modem. I just switched it over to a newer Billion 7800NL modem in the hope it'd provide better sync speeds on my hopeless phone line. (It does, with some SNR tweaking)
However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492.
I'm pretty confused by this -- pppd was setting itself up with a 1492 mtu even on the old modem, and that managed to propagate out just fine. The only thing that changed was the modem, but that shouldn't affect this issue.
I'm bamboozled. Could anyone advise me on what could possibly be done to figure out what has happened? Or better, a way to fix it?
Investigative notes: 1) New router was switched back to standalone mode (where it does pppoe, dhcp, nat etc) and in this mode, all clients worked fine. So the ADSL link and ethernet ports seem OK. 2) Old router was booted up to verify settings and that stuff works OK with it. Settings for ADSL line were similar, and stuff does work with it. (I say settings were "similar" because the two modems have different interfaces, and the 7800NL has some extra options not present on the 7300RA, but I don't think they're relevant to this issue)
Thanks in advance, Toby
-- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
-- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
On 26/09/13 11:19, Toby Corkindale wrote:
Well, putting this iptables rule at the head of my FORWARD chain solves the problem: iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
I still have absolutely no idea why this worked fine on one router, and fails on another though!
That only works for TCP, you will probably still have problems with large UDP transfers.
On 26 September 2013 11:21, Keith Owens <kaos@ocs.com.au> wrote:
On 26/09/13 11:19, Toby Corkindale wrote:
Well, putting this iptables rule at the head of my FORWARD chain solves the problem: iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
I still have absolutely no idea why this worked fine on one router, and fails on another though!
That only works for TCP, you will probably still have problems with large UDP transfers.
Indeed. I'd rather fix the root cause if it can be found.
On 26/09/13 11:27, Toby Corkindale wrote:
On 26 September 2013 11:21, Keith Owens <kaos@ocs.com.au> wrote:
On 26/09/13 11:19, Toby Corkindale wrote:
Well, putting this iptables rule at the head of my FORWARD chain solves the problem: iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
I still have absolutely no idea why this worked fine on one router, and fails on another though!
That only works for TCP, you will probably still have problems with large UDP transfers. Indeed. I'd rather fix the root cause if it can be found.
Have you checked for large packets going between the server and the modem? That is, any packets > MTU. That is a sure sign of offload getting the way.
On 26 September 2013 11:30, Keith Owens <kaos@ocs.com.au> wrote:
On 26/09/13 11:27, Toby Corkindale wrote:
On 26 September 2013 11:21, Keith Owens <kaos@ocs.com.au> wrote:
On 26/09/13 11:19, Toby Corkindale wrote:
Well, putting this iptables rule at the head of my FORWARD chain solves the problem: iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
I still have absolutely no idea why this worked fine on one router, and fails on another though!
That only works for TCP, you will probably still have problems with large UDP transfers. Indeed. I'd rather fix the root cause if it can be found.
Have you checked for large packets going between the server and the modem? That is, any packets > MTU. That is a sure sign of offload getting the way.
It's PPPoE, so the pppd runs on the server and sends ethernet (not IP) packets to the modem. This all used to work fine and nothing has changed on the server side.
However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492.
Is it outbound or inbound MTU path discovery that isn't working? Or both? I've had to set MRU explicitly before to get things working properly, although I don't see why the different modem would make a different, unless it was using baby jumbo frames or something. What does 'ip link show dev ppp0' say? Is MTU definitely 1492 in there? What happens when a client computer tries to send a larger packet? Does your Linux router send a 'frag required' response? James
On 26 September 2013 14:13, James Harper <james.harper@bendigoit.com.au> wrote:
However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492.
Is it outbound or inbound MTU path discovery that isn't working? Or both?
Hmm, possibly inbound, now you mention it.. I was testing by attempting to curl web pages or images, and didn't try sending anything outbound at the time.
I've had to set MRU explicitly before to get things working properly, although I don't see why the different modem would make a different, unless it was using baby jumbo frames or something.
Both models are 10/100 ethernet only.
What does 'ip link show dev ppp0' say? Is MTU definitely 1492 in there?
Yes, definitely 1492 mtu listed there.
What happens when a client computer tries to send a larger packet? Does your Linux router send a 'frag required' response?
If I'm reading the tcpdump correctly (see reply to myself with it this morning) then yes, there is a frag-required response. T
On 26 September 2013 14:13, James Harper <james.harper@bendigoit.com.au> wrote:
However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492.
Is it outbound or inbound MTU path discovery that isn't working? Or both?
Hmm, possibly inbound, now you mention it.. I was testing by attempting to curl web pages or images, and didn't try sending anything outbound at the time.
I've had to set MRU explicitly before to get things working properly, although I don't see why the different modem would make a different, unless it was using baby jumbo frames or something.
Both models are 10/100 ethernet only.
What does 'ip link show dev ppp0' say? Is MTU definitely 1492 in there?
Yes, definitely 1492 mtu listed there.
What happens when a client computer tries to send a larger packet? Does your Linux router send a 'frag required' response?
If I'm reading the tcpdump correctly (see reply to myself with it this morning) then yes, there is a frag-required response.
Right, so your ISP router isn't sending frag required packets properly by the sounds of it, possibly because it thinks that you can accept such packets. Are the modem settings exactly the same, including the VC-MUX / LLC settings? This can throw things out if it makes assumptions based on those settings. Can you try setting the mru explicitly to 1492 in your ppp config? This will tell the other end that you can't accept packets bigger than 1492 bytes. James
On 26 September 2013 14:36, James Harper <james.harper@bendigoit.com.au> wrote:
On 26 September 2013 14:13, James Harper <james.harper@bendigoit.com.au> wrote:
However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492.
Is it outbound or inbound MTU path discovery that isn't working? Or both?
Hmm, possibly inbound, now you mention it.. I was testing by attempting to curl web pages or images, and didn't try sending anything outbound at the time.
I've had to set MRU explicitly before to get things working properly, although I don't see why the different modem would make a different, unless it was using baby jumbo frames or something.
Both models are 10/100 ethernet only.
What does 'ip link show dev ppp0' say? Is MTU definitely 1492 in there?
Yes, definitely 1492 mtu listed there.
What happens when a client computer tries to send a larger packet? Does your Linux router send a 'frag required' response?
If I'm reading the tcpdump correctly (see reply to myself with it this morning) then yes, there is a frag-required response.
Right, so your ISP router isn't sending frag required packets properly by the sounds of it, possibly because it thinks that you can accept such packets.
Are the modem settings exactly the same, including the VC-MUX / LLC settings? This can throw things out if it makes assumptions based on those settings.
As close as they could be, given differences in modem UI and options. However both old and new are set to LLC, yes.
Can you try setting the mru explicitly to 1492 in your ppp config? This will tell the other end that you can't accept packets bigger than 1492 bytes.
OK. I've just tried that now -- does NOT fix the problem. T
Toby Corkindale <toby@dryft.net> writes:
However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492.
No one has mentioned it yet, so I will: PMTUD needs ICMP to make it through your firewalls, though I forget which types/code(s). Check the firewalls on all relevant hosts.
On 26 September 2013 16:13, Trent W. Buck <trentbuck@gmail.com> wrote:
Toby Corkindale <toby@dryft.net> writes:
However for some reason path MTU discovery has completely broken when using the newer modem. The primary Linux box is fine, but all clients (whether wired, wireless *or even virtual*) fail to work unless I manually set their interfaces to an mtu of 1492.
No one has mentioned it yet, so I will:
PMTUD needs ICMP to make it through your firewalls, though I forget which types/code(s). Check the firewalls on all relevant hosts.
Indeed. However the firewall configuration hasn't changed at all. I really just unplugged one modem and plugged in a new one. I didn't even reboot the Linux server or restart the pppd process! (I did a bit later once it became clear everything was borked) I don't understand how using a new bridged adsl modem could affect this. I'm really confused :( Could an ADSL modem really be spotting ICMP packets encapsulated in a PPPOE ethernet frame and throwing them out? Seems unlikely. Or could Exetel have decided to start throwing out ICMP packets at their end suddenly after I reconnected with a new model?? None of these are likely cases, but I'm stumped as to what else could be the cause. T
participants (4)
-
James Harper -
Keith Owens -
Toby Corkindale -
trentbuck＠gmail.com