arantius.com

Working around broken traceroute on FiOS

2021-01-12 17:42 - Tech

After a long and slow process, I've decided to "cut the cord". I haven't been using my cable subscription much lately, and I've done very well filling in the gap with over-the-air and online services. Today I took the first big step in making that happen. I had a TV and internet bundle from Spectrum cable. I got today a faster internet-only plan from Verizon FiOS, for about half the cost. It's mostly fine. But one of the first things I discovered is that mtr was misbehaving. Every host appeared two hops away, and only a few milliseconds away. I know this is not true.

$ mtr -4 --report -n www.google.com
Start: 2021-01-12T16:57:18-0500
HOST: laptop                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.1.1                0.0%    10    0.8   1.3   0.8   2.7   0.6
  2.|-- 172.217.3.100              0.0%    10    4.0   9.5   3.5  23.8   7.5
$ mtr -4 --report -n www.ebay.com
Start: 2021-01-12T16:58:15-0500
HOST: laptop                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.1.1                0.0%    10    2.5   1.8   0.8   3.1   0.9
  2.|-- 23.34.248.103              0.0%    10    4.7   4.9   1.9   9.8   2.5

Traceroute usually/sometimes works by sending ICMP echoes with a low TTL. Too low, so the connection fails, and the hop at that distance replies that it fails. Up the TTL by one until a successful reply, and we've found the target. Verizon is forging successful answers when only one TTL (my router, plus one) remains. I discovered this by searching around for an explanation of what was going wrong exactly, and found a forum post (on a thread from 2018, so not a very temporary issue) with an acceptable workaround. Have the router never send a packet with TTL=1, bump it up to 2 instead. This doubles the next hop, but that's the best we can get if Verizon is going to lie. Problem is, the solution was given in a proprietary (?) configuration format.

I use OpenWRT, which is "just" standard Linux. So it uses iptables. I came up with:

# iptables -t mangle -I POSTROUTING -o eth1.2 -m ttl --ttl-eq 1 -j TTL --ttl-inc 1

Phew. That's: -t mangle use the "mangle" table (manpage: "This table is used for specialized packet alteration."), -I POSTROUTING use the "POSTROUTING" chain ("for altering packets as they are about to go out"), -o eth1.2 for packets going out on the eth1.2 interface (my WAN interface), -m ttl use the "ttl" match extension module, --ttl_eq 1 the TTL value equals one, -j TTL jump to the (custom) "TTL" chain, and -ttl-inc 1 increase that TTL value by one (from one, to two).

Now Verizon never sees a TTL=1 ICMP packet coming from me, and never forges the answer, and mtr just works (except that the first hop past the router is a duplicated copy of the next hop):

$ mtr -4 --report -n www.google.com
Start: 2021-01-12T17:32:33-0500
HOST: laptop                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.1.1                0.0%    10    1.2   1.0   0.8   1.2   0.1
  2.|-- 100.41.219.201             0.0%    10    5.0   7.4   5.0   9.2   1.4
  3.|-- 100.41.219.201             0.0%    10    5.7   8.5   4.8  15.5   3.0
  4.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- 140.222.227.35             0.0%    10    7.7   7.1   4.8   9.3   1.4
  6.|-- 72.14.214.36               0.0%    10    7.3   7.4   5.4   9.4   1.1
  7.|-- 108.170.225.4              0.0%    10    8.9   8.7   7.0  10.0   0.9
  8.|-- 209.85.244.65              0.0%    10    9.3   7.4   5.2   9.3   1.3
  9.|-- 172.217.3.100              0.0%    10    4.9   6.9   4.8  13.2   2.6

All I need to do is put this in OpenWRT's /etc/firewall.user file. Google is actually only a few milliseconds away, but they're also 9 hops away, not two.