nixos-config/docs/LINUX-INTERNET-ISSUES.md

Linux internet issues on pureintent

Notes from debugging an outage on pureintent, a NixOS machine with Ethernet as the primary interface and Wi-Fi as fallback.

Expected setup

pureintent has both interfaces on the same LAN:

Interface	Role	Example IP	Expected metric
enp1s0	Ethernet, primary	192.168.2.43	100
wlp2s0	Wi-Fi, fallback	192.168.2.152	600
tailscale0	Tailscale tunnel	100.122.32.106	table 52

The default route should prefer Ethernet:

ip route get 1.1.1.1

Expected result:

1.1.1.1 via 192.168.2.1 dev enp1s0 src 192.168.2.43

Symptoms seen

The machine was reachable over SSH and Tailscale, but internet access from pureintent itself was broken.

Observed behavior:

• ping 1.1.1.1 failed.

• curl http://1.1.1.1 timed out before connecting.

• DNS lookups timed out because /etc/resolv.conf pointed at Tailscale DNS:

  nameserver 100.100.100.100
  

• tailscale status showed Tailscale was running.

• tailscale debug prefs showed no exit node was active:

  "RouteAll": false
  "ExitNodeID": ""
  "CorpDNS": true
  

• LAN reachability still worked for some peers, e.g. 192.168.2.129.

This means the issue was not simply "Tailscale stole the route". Normal internet traffic was still supposed to leave via the LAN gateway.

Useful triage commands

Run these from another machine:

ssh pureintent 'ip -br addr; ip route; cat /etc/resolv.conf'

Check raw IP connectivity first, before DNS:

ssh pureintent 'ping -c 2 -W 2 1.1.1.1'
ssh pureintent 'curl -4 -sS --connect-timeout 4 --max-time 8 -o /dev/null -w "exit=%{exitcode} http=%{http_code} remote=%{remote_ip}\n" http://1.1.1.1'

Check which interface the kernel would use:

ssh pureintent 'ip route get 1.1.1.1'

Check gateway reachability per interface:

ssh pureintent 'ping -I enp1s0 -c 2 -W 2 192.168.2.1'
ssh pureintent 'ping -I wlp2s0 -c 2 -W 2 192.168.2.1'

Check Tailscale state:

ssh pureintent 'tailscale status'
ssh pureintent 'tailscale debug prefs'

What fixed it

Ethernet was the preferred default route, but the Ethernet path appeared stale: the link was up and had a DHCP address, but traffic over that path was not returning.

Disconnecting Ethernet temporarily proved Wi-Fi was healthy:

nmcli dev disconnect enp1s0

With only Wi-Fi active:

• ping -I wlp2s0 192.168.2.1 succeeded.
• ping -I wlp2s0 1.1.1.1 succeeded.
• curl --interface wlp2s0 http://1.1.1.1 succeeded.

Reconnecting Ethernet restored the primary route:

nmcli con up "Wired connection 1"

After reconnecting:

default via 192.168.2.1 dev enp1s0 metric 100
default via 192.168.2.1 dev wlp2s0 metric 600

And:

ping -c 1 -W 2 1.1.1.1
curl -4 https://github.com

both worked.

NetworkManager settings applied

The wired profile had a bad autoconnect priority:

Wired connection 1  autoconnect-priority -999
drapeau             autoconnect-priority 0

This was corrected imperatively:

nmcli connection modify "Wired connection 1" \
  connection.autoconnect yes \
  connection.autoconnect-priority 100 \
  ipv4.route-metric 100 \
  ipv6.route-metric 100 \
  ipv4.never-default no \
  ipv6.never-default no

nmcli connection modify "drapeau" \
  connection.autoconnect yes \
  connection.autoconnect-priority 0 \
  ipv4.route-metric 600 \
  ipv6.route-metric 600 \
  ipv4.never-default no \
  ipv6.never-default no

Verify:

nmcli -f NAME,TYPE,AUTOCONNECT,AUTOCONNECT-PRIORITY connection show
nmcli -f ipv4.route-metric,ipv6.route-metric connection show "Wired connection 1"
nmcli -f ipv4.route-metric,ipv6.route-metric connection show "drapeau"

NixOS durable fix

Because this is NixOS, the long-term fix should be declarative. At the time of debugging, /etc/nixos/configuration.nix only had:

networking.networkmanager.enable = true;

It did not declare the NetworkManager profiles.

Add the profile priorities and route metrics declaratively, either in /etc/nixos/configuration.nix or the equivalent module in this repo:

networking.networkmanager.ensureProfiles.profiles = {
  "Wired connection 1" = {
    connection = {
      id = "Wired connection 1";
      type = "ethernet";
      interface-name = "enp1s0";
      autoconnect = true;
      autoconnect-priority = 100;
    };
    ipv4 = {
      method = "auto";
      route-metric = 100;
    };
    ipv6 = {
      method = "auto";
      route-metric = 100;
    };
  };

  drapeau = {
    connection = {
      id = "drapeau";
      type = "wifi";
      interface-name = "wlp2s0";
      autoconnect = true;
      autoconnect-priority = 0;
    };
    ipv4 = {
      method = "auto";
      route-metric = 600;
    };
    ipv6 = {
      method = "auto";
      route-metric = 600;
    };
  };
};

If Wi-Fi secrets are managed elsewhere, do not duplicate the Wi-Fi profile. Put the fallback metric and priority in the existing drapeau profile instead.

Recovery shortcut

If Ethernet is up but internet is broken while Wi-Fi works:

nmcli dev disconnect enp1s0 && nmcli con up "Wired connection 1"

Then verify:

ip route get 1.1.1.1
ping -c 2 -W 2 1.1.1.1
curl -4 -sS --connect-timeout 4 --max-time 8 https://github.com >/dev/null

Variant: MagicDNS with no upstream resolver

Different outage, similar-looking symptom: /etc/resolv.conf again pointed only at 100.100.100.100, but this time IP-layer routing was healthy and the cause was purely DNS.

Symptoms:

• ping 1.1.1.1 works.
• ping google.com fails with Name or service not known.
• getent hosts github.com returns nothing.
• *.ts.net lookups still work (split-DNS route is independent).

Smoking gun in journalctl -u tailscaled:

dns: resolver: forward: no upstream resolvers set, returning SERVFAIL

And tailscale dns status shows:

Resolvers (in preference order):
  (no resolvers configured, system default will be used: see 'System DNS configuration' below)
...
  (failed to read system DNS configuration: Access denied: dns-osconfig dump access denied)

What's happening: Tailscale is managing /etc/resolv.conf (accept-dns=true) and MagicDNS handles *.ts.net via the split-DNS route, but for every other query it needs an upstream resolver. If the tailnet admin console has no Global nameservers configured, Tailscale tries to fall back to the device's system DNS — which on NixOS/tailscale 1.98 fails with the dns-osconfig access-denied error above. Result: SERVFAIL for everything non-tailnet.

Fix in the admin console at https://login.tailscale.com/admin/dns:

1. Under Global nameservers, add an upstream (e.g. Cloudflare 1.1.1.1).
2. Turn Override DNS servers ON. With it off, the global nameserver is only used when the device's own OS DNS is readable — which on this host it isn't.

Netmap propagation is near-instant; no daemon restart needed. Verify:

ssh pureintent 'tailscale dns status | sed -n "/Resolvers/,/Split/p"'
ssh pureintent 'getent hosts github.com && ping -c2 google.com'

The "Resolvers (in preference order)" list should now contain the upstream IPs instead of (no resolvers configured ...).

Main lesson

When both Ethernet and Wi-Fi are on the same subnet, a stale primary interface can look superficially healthy: it has carrier, DHCP, ARP entries, and LAN reachability, but public traffic still times out. Test raw public IPs before DNS, force tests through each interface, and only then look at Tailscale DNS or exit node settings.