5 ms·

Understanding modern Linux routing (and wg-quick)

179 points by EntICOnc 4y ago

rubatuga 4y ago
It’s kindly explained on the actual WireGuard page as well: https://www.wireguard.com/netns/ https://www.wireguard.com/netns/
- endre 4y ago
  and it is mentioned on the blogpost as well. :)
- mathieuruellan 4y ago
  I prefer flagging packets from cgroup. But since cgroupv2 it can only be done with systemd/slices, it's not easy and stable. (The slice must exist before iptables rule, and if it stops, i experiment kernel crashes etc.)
VTimofeenko 4y ago
I prefer the namespace approach with wg, as it becomes explicit and easier to use with services through JoinsNamespace. The routing inside the namespace is just using the main table. This also allows to isolate the DNS queries by bind-mounting nsswitch.conf and resolv.conf as well as manage the isolated firewall for the namespace.
anonymousiam 4y ago
"Fun fact: wg-quick uses the same numbers for the table and the fwmark: 0xca6c is just 51820 in hexadecimal." An additional "fun fact" is that 51820 is also the default port number used by WireGuard servers.
jandrese 4y ago
> Older VPN scripts used to override your default route in the main table when connecting to the VPN and restore it when disconnecting. Sometimes this wouldn’t work, and after disconnecting from the VPN you would be left without any default route at all. wg-quick doesn’t have this problem, as it never messes with your main routing table. All it has to do when disconnecting is delete its two rules, and your default route is active again, Or you could even do that yourself with ip rule del. Isn't this the same problem the old VPN problems had though? If it fails to delete its two rules then your routing is still boned. For comparison OpenVPN doesn't use the ip rule infrastructure, instead it installs two rules directly on your routing table, but not for 0.0.0.0/0. Instead it installs rules for 0.0.0.0/1 and 128.0.0.0/1, creating a rule that has slightly higher priority than the default route so it can leave the default in place and not get burned with NetworkManager decides to delete and re-add the default route for whatever reason. Not having the routes cleaned up after the VPN exits is usually not the problem, as they are tied to the TUN interface and get cleaned up by the kernel when that interface is destroyed anyway. The bigger issue is clients that forget to install the host route to the VPN endpoint. I see that error a lot and it always drives people crazy because their VPN client negotiates and connects properly, and then everything breaks for no obvious reason.
- throwaway787544 4y ago
  Yeah, other VPNs do the same thing as wg-quick. Although they also may make a ton of more specific routes which take precedence over ones from wg-quick. It's fairly rare that routes don't get cleaned up, but a reconnect/disconnect or a reboot fixes it. The host route to the VPN endpoint is sometimes there, but sometimes it's to a p-2-p interface, sometimes just the local device IP. Depends on the OS, tunnel interface, routes, etc.
- bogomipz 4y ago
  >"For comparison OpenVPN doesn't use the ip rule infrastructure, instead it installs two rules directly on your routing table, but not for 0.0.0.0/0. Instead it installs rules for 0.0.0.0/1 and 128.0.0.0/1, creating a rule that has slightly higher priority than the default route ...." I'm curious is the priority you referring to just longest prefix match by having two /1s, or are you referring to something else like setting a metric on the interface?
  - Hikikomori 4y ago
    Yes, they are more specific routes than 0/0. In Linux can also have a metric, so if two routes are equal one can be preferred over the other. On networking devices you usually have administrative distance (different routing protocols) and metric (within one routing protocol).
    - bogomipz 4y ago
      Yes I understand how routing works and I referenced both of these things in my question. I was asking specifically which one the OP was referring to.
      - Hikikomori 4y ago
        Wasn't it obvious what OP meant then?
- allarm 4y ago
  > Instead it installs rules for 0.0.0.0/1 and 128.0.0.0/1, creating a rule that has slightly higher priority More specific prefix isn’t slightly better, prefix length is one of the most significant rules of routing. This trick is quite common in networking - you point more specific routes to the main link and less specific to a backup. It the main link go down the specifics are getting withdrawn (some mechanism like dynamic routing should be in place for that) and the less specific route takes place, redirecting the traffic over the backup link.
gxt 4y ago
Anybody working on an enterprise bundle and support package for wireguard? All "enterprise" vpn solutions I've had to touch in the past 10years are flaky at best.
- wswope 4y ago
  Does tailscale match what you had in mind?
rustqt6 4y ago
Has anyone been able to roll out a homemade speedify on Linux?
teleforce 4y ago
Someone need to come up with a new reference book on Linux routing. The last reliable book on the topic is about 20 years ago during Linux kernel 2.X, the time when the Linux kernel has a dedicate routing table as mentioned in the article. Now it is even more important due to the popularity of Linux networking in the era of cloud, edge and container. [1] Book Review: Linux Routers - A Primer for Network Administrators, 2nd Ed https://www.linuxjournal.com/article/6314 https://www.linuxjournal.com/article/6314
- CameronNemo 4y ago
  Vincent Bernat has one of my favorite blogs about networking. https://vincent.bernat.ch/en/blog#tag-linux https://vincent.bernat.ch/en/blog#tag-linux
- znpy 4y ago
  I came to say pretty much this. It's high time for somebody to write a guide to linux networking. The closest thing to a comprehensive guide is the Linux Advanced Routing & Traffic Control HOWTO (https://lartc.org https://lartc.org) but the last update (as far as I can tell from the website) is from 2012.
- the_biot 4y ago
  I'm not so sure people buy books like that anymore, not at the volume they did 20 years ago. When was the last time you bought an O'Reilly? I used to buy several of these a year, but can't remember the last time I even saw one. You're right that a new, easy to read guide is needed, but I think it would have to be just a website. The tools you need for Linux networking, mostly ip and tc, are as crude-but-powerful as they were 20 years ago, but now have much more functionality and complexity. Things are actively getting worse: a tool like brctl (bridge control) was obsoleted for the same functionality in iproute2 -- but brctl was much more intuitive to use. Some nicer frontends than iproute2 would really help here.
3np 4y ago
One thing worth adding to this is that what wg-quick does here - adding routes for each entry in AllowedIPs and a corresponding routing policy - is not part of wireguard itselfbut a wg-quick specific thing. This can become important if you want to have redundant gateways or something like that - you can have overlapping AllowedIPs, it just won't work out of the box with wg-quick. For example, systemd-networkd started also adding routes by default in v250 but (rightly so IMO[0]) reverted to not doing it in 250.3. [0]: This comment captures it quite well: https://github.com/systemd/systemd/issues/14176#issuecomment-562475555 https://github.com/systemd/systemd/issues/14176#issuecomment...
vlindhol 4y ago
Great article! Does anyone know of a similar resource for MacOS? I’ve been trying to use Tailscale together with my company’s enterprisey CloudFlare VPN and Tailscale rules never seem to evaluate, even with split tunneling etc enabled. Need to dig into those routing tables one of these days…
unionpivo 4y ago
Is there any current, Linux Administration book, or online resource, that would anyone here would recommend? Most I found still use old and outdated commands, or don't really explain thing in depth.
- hellohowareu 4y ago
vbezhenar 4y ago
Unrelated question. But may be someone knows enough. Is it possible to create high-available wireguard server? Like I put load balancer on UDP :51820 before servers and then those servers somehow share a session pool, so when one goes down, another can continue serving clients without interruptions. Naive approach does not work: wireguard establishes connection and if backend server changes, this connection stuck for a few minutes until wireguard decided to establish new connection. Persistent Keep Alive does not help for some reason. Eventually it'll re-establish new session and will work from now on, but this delay is unacceptable.
- goodburb 4y ago
  Not a LB, look into Engarde.
- paol 4y ago
  Take a look a these (not tested these configurations personally): - High Availability WireGuard on AWS [https://www.procustodibus.com/blog/2021/02/ha-wireguard-on-aws/ https://www.procustodibus.com/blog/2021/02/ha-wireguard-on-a...] - High Availability WireGuard on Azure [https://www.procustodibus.com/blog/2021/06/ha-wireguard-on-azure/ https://www.procustodibus.com/blog/2021/06/ha-wireguard-on-a...] - High Availability WireGuard Site to Site [https://www.procustodibus.com/blog/2021/10/ha-wireguard-site-to-site/ https://www.procustodibus.com/blog/2021/10/ha-wireguard-site...]
  - vbezhenar 4y ago
    Thanks, it seems that they just OK with user being offline for minute or two, they use ordinary configuration. I'm using the same approach, but I'd really love instant switchover.

179 points by EntICOnc 4y ago

It’s kindly explained on the actual WireGuard page as well: https://www.wireguard.com/netns/ https://www.wireguard.com/netns/

endre 4y ago
and it is mentioned on the blogpost as well. :)
mathieuruellan 4y ago
I prefer flagging packets from cgroup. But since cgroupv2 it can only be done with systemd/slices, it's not easy and stable. (The slice must exist before iptables rule, and if it stops, i experiment kernel crashes etc.)

VTimofeenko 4y ago

I prefer the namespace approach with wg, as it becomes explicit and easier to use with services through JoinsNamespace. The routing inside the namespace is just using the main table. This also allows to isolate the DNS queries by bind-mounting nsswitch.conf and resolv.conf as well as manage the isolated firewall for the namespace.

anonymousiam 4y ago

"Fun fact: wg-quick uses the same numbers for the table and the fwmark: 0xca6c is just 51820 in hexadecimal." An additional "fun fact" is that 51820 is also the default port number used by WireGuard servers.

jandrese 4y ago

> Older VPN scripts used to override your default route in the main table when connecting to the VPN and restore it when disconnecting. Sometimes this wouldn’t work, and after disconnecting from the VPN you would be left without any default route at all. wg-quick doesn’t have this problem, as it never messes with your main routing table. All it has to do when disconnecting is delete its two rules, and your default route is active again, Or you could even do that yourself with ip rule del. Isn't this the same problem the old VPN problems had though? If it fails to delete its two rules then your routing is still boned. For comparison OpenVPN doesn't use the ip rule infrastructure, instead it installs two rules directly on your routing table, but not for 0.0.0.0/0. Instead it installs rules for 0.0.0.0/1 and 128.0.0.0/1, creating a rule that has slightly higher priority than the default route so it can leave the default in place and not get burned with NetworkManager decides to delete and re-add the default route for whatever reason. Not having the routes cleaned up after the VPN exits is usually not the problem, as they are tied to the TUN interface and get cleaned up by the kernel when that interface is destroyed anyway. The bigger issue is clients that forget to install the host route to the VPN endpoint. I see that error a lot and it always drives people crazy because their VPN client negotiates and connects properly, and then everything breaks for no obvious reason.

throwaway787544 4y ago
Yeah, other VPNs do the same thing as wg-quick. Although they also may make a ton of more specific routes which take precedence over ones from wg-quick. It's fairly rare that routes don't get cleaned up, but a reconnect/disconnect or a reboot fixes it. The host route to the VPN endpoint is sometimes there, but sometimes it's to a p-2-p interface, sometimes just the local device IP. Depends on the OS, tunnel interface, routes, etc.
bogomipz 4y ago
>"For comparison OpenVPN doesn't use the ip rule infrastructure, instead it installs two rules directly on your routing table, but not for 0.0.0.0/0. Instead it installs rules for 0.0.0.0/1 and 128.0.0.0/1, creating a rule that has slightly higher priority than the default route ...." I'm curious is the priority you referring to just longest prefix match by having two /1s, or are you referring to something else like setting a metric on the interface?
- Hikikomori 4y ago
  Yes, they are more specific routes than 0/0. In Linux can also have a metric, so if two routes are equal one can be preferred over the other. On networking devices you usually have administrative distance (different routing protocols) and metric (within one routing protocol).
  - bogomipz 4y ago
    Yes I understand how routing works and I referenced both of these things in my question. I was asking specifically which one the OP was referring to.
    - Hikikomori 4y ago
      Wasn't it obvious what OP meant then?
allarm 4y ago
> Instead it installs rules for 0.0.0.0/1 and 128.0.0.0/1, creating a rule that has slightly higher priority More specific prefix isn’t slightly better, prefix length is one of the most significant rules of routing. This trick is quite common in networking - you point more specific routes to the main link and less specific to a backup. It the main link go down the specifics are getting withdrawn (some mechanism like dynamic routing should be in place for that) and the less specific route takes place, redirecting the traffic over the backup link.

gxt 4y ago

Anybody working on an enterprise bundle and support package for wireguard? All "enterprise" vpn solutions I've had to touch in the past 10years are flaky at best.

wswope 4y ago
Does tailscale match what you had in mind?

rustqt6 4y ago

Has anyone been able to roll out a homemade speedify on Linux?

teleforce 4y ago

Someone need to come up with a new reference book on Linux routing. The last reliable book on the topic is about 20 years ago during Linux kernel 2.X, the time when the Linux kernel has a dedicate routing table as mentioned in the article. Now it is even more important due to the popularity of Linux networking in the era of cloud, edge and container. [1] Book Review: Linux Routers - A Primer for Network Administrators, 2nd Ed https://www.linuxjournal.com/article/6314 https://www.linuxjournal.com/article/6314

CameronNemo 4y ago
Vincent Bernat has one of my favorite blogs about networking. https://vincent.bernat.ch/en/blog#tag-linux https://vincent.bernat.ch/en/blog#tag-linux
znpy 4y ago
I came to say pretty much this. It's high time for somebody to write a guide to linux networking. The closest thing to a comprehensive guide is the Linux Advanced Routing & Traffic Control HOWTO (https://lartc.org https://lartc.org) but the last update (as far as I can tell from the website) is from 2012.
the_biot 4y ago
I'm not so sure people buy books like that anymore, not at the volume they did 20 years ago. When was the last time you bought an O'Reilly? I used to buy several of these a year, but can't remember the last time I even saw one. You're right that a new, easy to read guide is needed, but I think it would have to be just a website. The tools you need for Linux networking, mostly ip and tc, are as crude-but-powerful as they were 20 years ago, but now have much more functionality and complexity. Things are actively getting worse: a tool like brctl (bridge control) was obsoleted for the same functionality in iproute2 -- but brctl was much more intuitive to use. Some nicer frontends than iproute2 would really help here.

3np 4y ago

One thing worth adding to this is that what wg-quick does here - adding routes for each entry in AllowedIPs and a corresponding routing policy - is not part of wireguard itselfbut a wg-quick specific thing. This can become important if you want to have redundant gateways or something like that - you can have overlapping AllowedIPs, it just won't work out of the box with wg-quick. For example, systemd-networkd started also adding routes by default in v250 but (rightly so IMO[0]) reverted to not doing it in 250.3. [0]: This comment captures it quite well: https://github.com/systemd/systemd/issues/14176#issuecomment-562475555 https://github.com/systemd/systemd/issues/14176#issuecomment...

vlindhol 4y ago

Great article! Does anyone know of a similar resource for MacOS? I’ve been trying to use Tailscale together with my company’s enterprisey CloudFlare VPN and Tailscale rules never seem to evaluate, even with split tunneling etc enabled. Need to dig into those routing tables one of these days…

unionpivo 4y ago

Is there any current, Linux Administration book, or online resource, that would anyone here would recommend? Most I found still use old and outdated commands, or don't really explain thing in depth.

hellohowareu 4y ago

vbezhenar 4y ago

Unrelated question. But may be someone knows enough. Is it possible to create high-available wireguard server? Like I put load balancer on UDP :51820 before servers and then those servers somehow share a session pool, so when one goes down, another can continue serving clients without interruptions. Naive approach does not work: wireguard establishes connection and if backend server changes, this connection stuck for a few minutes until wireguard decided to establish new connection. Persistent Keep Alive does not help for some reason. Eventually it'll re-establish new session and will work from now on, but this delay is unacceptable.

goodburb 4y ago
Not a LB, look into Engarde.
paol 4y ago
Take a look a these (not tested these configurations personally): - High Availability WireGuard on AWS [https://www.procustodibus.com/blog/2021/02/ha-wireguard-on-aws/ https://www.procustodibus.com/blog/2021/02/ha-wireguard-on-a...] - High Availability WireGuard on Azure [https://www.procustodibus.com/blog/2021/06/ha-wireguard-on-azure/ https://www.procustodibus.com/blog/2021/06/ha-wireguard-on-a...] - High Availability WireGuard Site to Site [https://www.procustodibus.com/blog/2021/10/ha-wireguard-site-to-site/ https://www.procustodibus.com/blog/2021/10/ha-wireguard-site...]
- vbezhenar 4y ago
  Thanks, it seems that they just OK with user being offline for minute or two, they use ordinary configuration. I'm using the same approach, but I'd really love instant switchover.