Working around broken traceroute on FiOS

2021-01-12 17:42 - Tech

After a long and slow process, I've decided to "cut the cord". I haven't been using my cable subscription much lately, and I've done very well filling in the gap with over-the-air and online services. Today I took the first big step in making that happen. I had a TV and internet bundle from Spectrum cable. I got today a faster internet-only plan from Verizon FiOS, for about half the cost. It's mostly fine. But one of the first things I discovered is that mtr was misbehaving. Every host appeared two hops away, and only a few milliseconds away. I know this is not true.

$ mtr -4 --report -n
Start: 2021-01-12T16:57:18-0500
HOST: laptop                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|--                0.0%    10    0.8   1.3   0.8   2.7   0.6
  2.|--              0.0%    10    4.0   9.5   3.5  23.8   7.5
$ mtr -4 --report -n
Start: 2021-01-12T16:58:15-0500
HOST: laptop                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|--                0.0%    10    2.5   1.8   0.8   3.1   0.9
  2.|--              0.0%    10    4.7   4.9   1.9   9.8   2.5

Traceroute usually/sometimes works by sending ICMP echoes with a low TTL. Too low, so the connection fails, and the hop at that distance replies that it fails. Up the TTL by one until a successful reply, and we've found the target. Verizon is forging successful answers when only one TTL (my router, plus one) remains. I discovered this by searching around for an explanation of what was going wrong exactly, and found a forum post (on a thread from 2018, so not a very temporary issue) with an acceptable workaround. Have the router never send a packet with TTL=1, bump it up to 2 instead. This doubles the next hop, but that's the best we can get if Verizon is going to lie. Problem is, the solution was given in a proprietary (?) configuration format.

I use OpenWRT, which is "just" standard Linux. So it uses iptables. I came up with:

# iptables -t mangle -I POSTROUTING -o eth1.2 -m ttl --ttl-eq 1 -j TTL --ttl-inc 1

Phew. That's: -t mangle use the "mangle" table (manpage: "This table is used for specialized packet alteration."), -I POSTROUTING use the "POSTROUTING" chain ("for altering packets as they are about to go out"), -o eth1.2 for packets going out on the eth1.2 interface (my WAN interface), -m ttl use the "ttl" match extension module, --ttl_eq 1 the TTL value equals one, -j TTL jump to the (custom) "TTL" chain, and -ttl-inc 1 increase that TTL value by one (from one, to two).

Now Verizon never sees a TTL=1 ICMP packet coming from me, and never forges the answer, and mtr just works (except that the first hop past the router is a duplicated copy of the next hop):

$ mtr -4 --report -n
Start: 2021-01-12T17:32:33-0500
HOST: laptop                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|--                0.0%    10    1.2   1.0   0.8   1.2   0.1
  2.|--             0.0%    10    5.0   7.4   5.0   9.2   1.4
  3.|--             0.0%    10    5.7   8.5   4.8  15.5   3.0
  4.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|--             0.0%    10    7.7   7.1   4.8   9.3   1.4
  6.|--               0.0%    10    7.3   7.4   5.4   9.4   1.1
  7.|--              0.0%    10    8.9   8.7   7.0  10.0   0.9
  8.|--              0.0%    10    9.3   7.4   5.2   9.3   1.3
  9.|--              0.0%    10    4.9   6.9   4.8  13.2   2.6

All I need to do is put this in OpenWRT's /etc/firewall.user file. Google is actually only a few milliseconds away, but they're also 9 hops away, not two.


nftables equivalent
2022-04-23 14:37 - jtkarlma

Here's an equivalent I think will work for moving up to nftables/fw4 based setup on OpenWrt 22.03 or later. Put it into /etc/nftables.d/01-mangle-ttl-fios.nft

chain mangle_postrouting_fios {
  type filter hook postrouting priority 100; policy accept;
  oifname $wan_devices counter ip ttl 1 ip ttl set 2 comment "!fw4: bump TTL for FIOS"

Post a comment:

  If you do not have an account to log in to yet, register your own account. You will not enter any personal info and need not supply an email address.

You may use Markdown syntax in the comment, but no HTML. Hints:

If you are attempting to contact me, ask me a question, etc, please send me a message through the contact form rather than posting a comment here. Thank you. (If you post a comment anyway when it should be a message to me, I'll probably just delete your comment. I don't like clutter.)