9 ms·

Virtualize Your Network on FreeBSD with VNET

85 points by vermaden 6y ago

kitotik 6y ago
I’ve dabbled with OpenBSD and FreeBSD for personal projects for a long time, but have never had the opportunity to use them for anything “serious”. I am trying hard to figure out a path to change this. ZFS, Jails, VNET, and the stellar security track record are very enticing. In-kernel wireguard support was the last missing piece for me. The elegance and simplicity of the BSDs are becoming increasingly apparent in contrast to the Linux world. As an example, compare the jails UX to say, LXC/LXD…
- rufius 6y ago
  I used to chase bleeding edge Linux distros but these days, I nearly always reach for either an LTS release or a BSD for my personal projects. Sadly, I got all my network appliances setup and configured with Ubuntu 20.04 LTS a month or so before OpenBSD merged their WireGuard kernel module. I'm too lazy to touch those servers given they're relatively low traffic, especially during Pandemic times as I'm always on my local network.
- Aaronstotle 6y ago
  I need to try OpenBSD again, I always got an issue where the screen would hang after X amount of time and couldn't figure it out.
  - hellcow 6y ago
    On the off-chance you were using amdgpu, they merged a huge driver update in 6.8 that improved reliability for me from "unusable" to "easy daily driver."
- remexre 6y ago
  Why is in-kernel Wireguard a priority? I'm happily using https://github.com/WireGuard/wireguard-go https://github.com/WireGuard/wireguard-go with no issues to speak of, on FreeBSD.
  - kitotik 6y ago
    In my experience using the userspace daemon on both BSD and macOS there was increased cpu overhead and slightly higher network latency compared to Linux with a module or built in.
    - remexre 6y ago
      Ah, perhaps. I haven't measured it; I only use FreeBSD on "applicance-ish devices," i.e. my router and NAS, both of which are pretty overspecced for CPU.
- vermaden 6y ago
  I got a chance to use FreeBSD in such 'serious' case: https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise-1-pb-storage/ https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise... ... I recently added Jails there with Minio inside in each of those jails to serve S3 storage. Works like a charm.
  - dygd 6y ago
    Very comprehensive write-up. Thank you.
    - vermaden 6y ago
      Thanks.
infofarmer 6y ago
I'm always exasperated how an iPhone or any other smartphone cannot properly check if a WiFi has functional internet connection before switching over from cellular data. Half a century of relying on networking abstractions — and we're basically unable to say, hey, let this particular process work on this network connection while the rest of the system is relying on the other adapter. VIMAGE gracefully and painlessly allows that.
- emj 6y ago
  That seems semitrivial on Android since at least a decade back, you just need to implement it and find all the edge cases, so I guess there is a reason this is not done already. e.g . one routing table per interface add rules in global routing table for each new interface that is started: ip rule add from/to $IP_OF_IFACE/32\ table $UNIQUE_RT_FOR_IFACE ip route add default via IP_OF_IFACE_GW/32\ dev $IFACE_NAME table $UNIQUE_RT_FOR_IFACE Then when something changes on the iface that is not defaul; 1. remove old configs for that iface, 2. open a socket on the new ip on that iface to test internet connection. 3. If ok: change default in main routing table to the GW that iface uses.
- swiley 6y ago
  On sane OSes you could manually modify the routing tables. On iOS you can't even do basic things like turn of discoverable mode for bluetooth so this shouldn't be surprising.
znpy 6y ago
It always seemed to me that FreeBSD in particular could have all the pieces needed to implement the container runtime interface (and thus be able to run containers) but no one ever bothered. That's kind of a pity, because FreeBSD could be an ideal OS to run containers: a stable system with unified APIs, without 1.8e308 different distros, most of them subtly incompatible with each other.
- yjftsjthsd-h 6y ago
  This is close to one of my pet peeves about FreeBSD: They're happy to say that they've had containers for longer than Linux and that Jails are better than Linux containers, and I don't really disagree on technical merit, but they seem to completely overlook that UX and images are Docker's killer features and FreeBSD doesn't have an answer to that[0]. I personally think that implementing a jail-backed OCI runtime and getting docker running on top of it (all of which should be fairly simple to do) would be a huge boost to getting more users - bonus points if it can run FreeBSD-native and Linux images by using the existing Linux-compat layer. I've even looked into actually implementing this myself, but the runc-containerd-docker codebases were too much for me to make sense of so far, unfortunately. [0] Actually, partial caveat: cbsd and bhyve are at least meant to fill this space, I think, but they're neither a perfect match nor as easy to use IMO.
  - throwaway201103 6y ago
    Maybe the FreeBSD project doesn't care about getting more users? It's certainly not a big motivating factor for OpenBSD, but I don't know how FreeBSD feels about it.
  - kevans91 6y ago
    I can't speak for anyone else, but from my perspective (and I suspect others are similar): I like the idea of a jail-backed OCI runtime, but it's not interesting enough to my personal use to overtake any of the other OS-adjacent projects I'd otherwise work on in my spare time. I could be convinced (well, paid) to make it a higher priority, but ultimately it's currently at the bottom of a barrel of much more enticing fish since I'm not really in the demographic that that kind of stuff targets.
cperciva 6y ago
FWIW, this is how my imds-filterd (https://github.com/cperciva/imds-filterd https://github.com/cperciva/imds-filterd) works: Traffic for 169.254.169.254 is routed into a VNET jail. That way I can have a web server which proxies IMDS requests while staying out of the way of any networking being done on the host system (e.g. anything else which might want to bind to port 80).

85 points by vermaden 6y ago

I’ve dabbled with OpenBSD and FreeBSD for personal projects for a long time, but have never had the opportunity to use them for anything “serious”. I am trying hard to figure out a path to change this. ZFS, Jails, VNET, and the stellar security track record are very enticing. In-kernel wireguard support was the last missing piece for me. The elegance and simplicity of the BSDs are becoming increasingly apparent in contrast to the Linux world. As an example, compare the jails UX to say, LXC/LXD…

rufius 6y ago
I used to chase bleeding edge Linux distros but these days, I nearly always reach for either an LTS release or a BSD for my personal projects. Sadly, I got all my network appliances setup and configured with Ubuntu 20.04 LTS a month or so before OpenBSD merged their WireGuard kernel module. I'm too lazy to touch those servers given they're relatively low traffic, especially during Pandemic times as I'm always on my local network.
Aaronstotle 6y ago
I need to try OpenBSD again, I always got an issue where the screen would hang after X amount of time and couldn't figure it out.
- hellcow 6y ago
  On the off-chance you were using amdgpu, they merged a huge driver update in 6.8 that improved reliability for me from "unusable" to "easy daily driver."
remexre 6y ago
Why is in-kernel Wireguard a priority? I'm happily using https://github.com/WireGuard/wireguard-go https://github.com/WireGuard/wireguard-go with no issues to speak of, on FreeBSD.
- kitotik 6y ago
  In my experience using the userspace daemon on both BSD and macOS there was increased cpu overhead and slightly higher network latency compared to Linux with a module or built in.
  - remexre 6y ago
    Ah, perhaps. I haven't measured it; I only use FreeBSD on "applicance-ish devices," i.e. my router and NAS, both of which are pretty overspecced for CPU.
vermaden 6y ago
I got a chance to use FreeBSD in such 'serious' case: https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise-1-pb-storage/ https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise... ... I recently added Jails there with Minio inside in each of those jails to serve S3 storage. Works like a charm.
- dygd 6y ago
  Very comprehensive write-up. Thank you.
  - vermaden 6y ago
    Thanks.

infofarmer 6y ago

I'm always exasperated how an iPhone or any other smartphone cannot properly check if a WiFi has functional internet connection before switching over from cellular data. Half a century of relying on networking abstractions — and we're basically unable to say, hey, let this particular process work on this network connection while the rest of the system is relying on the other adapter. VIMAGE gracefully and painlessly allows that.

emj 6y ago
That seems semitrivial on Android since at least a decade back, you just need to implement it and find all the edge cases, so I guess there is a reason this is not done already. e.g . one routing table per interface add rules in global routing table for each new interface that is started: ip rule add from/to $IP_OF_IFACE/32\ table $UNIQUE_RT_FOR_IFACE ip route add default via IP_OF_IFACE_GW/32\ dev $IFACE_NAME table $UNIQUE_RT_FOR_IFACE Then when something changes on the iface that is not defaul; 1. remove old configs for that iface, 2. open a socket on the new ip on that iface to test internet connection. 3. If ok: change default in main routing table to the GW that iface uses.
swiley 6y ago
On sane OSes you could manually modify the routing tables. On iOS you can't even do basic things like turn of discoverable mode for bluetooth so this shouldn't be surprising.

znpy 6y ago

It always seemed to me that FreeBSD in particular could have all the pieces needed to implement the container runtime interface (and thus be able to run containers) but no one ever bothered. That's kind of a pity, because FreeBSD could be an ideal OS to run containers: a stable system with unified APIs, without 1.8e308 different distros, most of them subtly incompatible with each other.

yjftsjthsd-h 6y ago
This is close to one of my pet peeves about FreeBSD: They're happy to say that they've had containers for longer than Linux and that Jails are better than Linux containers, and I don't really disagree on technical merit, but they seem to completely overlook that UX and images are Docker's killer features and FreeBSD doesn't have an answer to that[0]. I personally think that implementing a jail-backed OCI runtime and getting docker running on top of it (all of which should be fairly simple to do) would be a huge boost to getting more users - bonus points if it can run FreeBSD-native and Linux images by using the existing Linux-compat layer. I've even looked into actually implementing this myself, but the runc-containerd-docker codebases were too much for me to make sense of so far, unfortunately. [0] Actually, partial caveat: cbsd and bhyve are at least meant to fill this space, I think, but they're neither a perfect match nor as easy to use IMO.
- throwaway201103 6y ago
  Maybe the FreeBSD project doesn't care about getting more users? It's certainly not a big motivating factor for OpenBSD, but I don't know how FreeBSD feels about it.
- kevans91 6y ago
  I can't speak for anyone else, but from my perspective (and I suspect others are similar): I like the idea of a jail-backed OCI runtime, but it's not interesting enough to my personal use to overtake any of the other OS-adjacent projects I'd otherwise work on in my spare time. I could be convinced (well, paid) to make it a higher priority, but ultimately it's currently at the bottom of a barrel of much more enticing fish since I'm not really in the demographic that that kind of stuff targets.

cperciva 6y ago

FWIW, this is how my imds-filterd (https://github.com/cperciva/imds-filterd https://github.com/cperciva/imds-filterd) works: Traffic for 169.254.169.254 is routed into a VNET jail. That way I can have a web server which proxies IMDS requests while staying out of the way of any networking being done on the host system (e.g. anything else which might want to bind to port 80).