16 ms·
Building a WireGuard jail with FreeBSD's standard tools
- Mic92 6y agoI wonder if FreeBSD can saturate 1Gbit/s with the TUN wireguard driver. Linux's native driver is likely faster.
- sigwinch28 6y agoLet's find out with a sample size of 1: Server is an HP Microserver with an Intel Xeon E3-1265L V2 @ 2.50GHz running FreeBSD 12.1. Client is a custom build with an Intel Core i7-4790K @ 4.00GHz running NixOS 20.03. $ ip route show default via 192.168.0.1 dev eno1 proto dhcp src 192.168.0.4 metric 203 192.168.0.0/24 dev eno1 proto dhcp scope link src 192.168.0.4 metric 203 192.168.1.0/24 dev wg0 scope link $ iperf3 -c 192.168.0.2 # no vpn Connecting to host 192.168.0.2, port 5201 [ 5] local 192.168.0.4 port 37382 connected to 192.168.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 115 MBytes 961 Mbits/sec 0 571 KBytes [ 5] 1.00-2.00 sec 111 MBytes 929 Mbits/sec 0 571 KBytes [ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 0 571 KBytes [ 5] 3.00-4.00 sec 111 MBytes 929 Mbits/sec 0 571 KBytes [ 5] 4.00-5.00 sec 112 MBytes 938 Mbits/sec 0 571 KBytes [ 5] 5.00-6.00 sec 111 MBytes 929 Mbits/sec 0 571 KBytes [ 5] 6.00-7.00 sec 112 MBytes 938 Mbits/sec 0 571 KBytes [ 5] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 571 KBytes [ 5] 8.00-9.00 sec 111 MBytes 929 Mbits/sec 0 571 KBytes [ 5] 9.00-10.00 sec 112 MBytes 938 Mbits/sec 0 571 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.09 GBytes 937 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.09 GBytes 934 Mbits/sec receiver iperf Done. $ iperf3 -c 192.168.1.1 # vpn Connecting to host 192.168.1.1, port 5201 [ 5] local 192.168.1.5 port 60358 connected to 192.168.1.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 108 MBytes 905 Mbits/sec 2 274 KBytes [ 5] 1.00-2.00 sec 106 MBytes 890 Mbits/sec 0 274 KBytes [ 5] 2.00-3.00 sec 107 MBytes 895 Mbits/sec 0 274 KBytes [ 5] 3.00-4.00 sec 107 MBytes 895 Mbits/sec 0 289 KBytes [ 5] 4.00-5.00 sec 107 MBytes 895 Mbits/sec 0 289 KBytes [ 5] 5.00-6.00 sec 107 MBytes 896 Mbits/sec 0 290 KBytes [ 5] 6.00-7.00 sec 104 MBytes 874 Mbits/sec 0 290 KBytes [ 5] 7.00-8.00 sec 106 MBytes 885 Mbits/sec 0 290 KBytes [ 5] 8.00-9.00 sec 105 MBytes 885 Mbits/sec 0 290 KBytes [ 5] 9.00-10.00 sec 107 MBytes 896 Mbits/sec 0 290 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.04 GBytes 892 Mbits/sec 2 sender [ 5] 0.00-10.00 sec 1.04 GBytes 891 Mbits/sec receiver iperf Done.
- tachion 6y agoI would assume you're testing that on the stock kernel settings that aren't really prepared for the highest network throughtput. There's a lot that can be done in the kernel sysctl's tuning for saturating the NIC and I'd expect you to see a bit better results when doing so. This is a very nice starting point for those interested: https://calomel.org/freebsd_network_tuning.html https://calomel.org/freebsd_network_tuning.html
- sigwinch28 6y agoI am running stock kernel settings. I would naively expect that the default kernel settings for both Linux and FreeBSD would allow me to saturate a 1Gbit link in a LAN. Anyway, this looks like one of those things I could go down the rabbit hole of tuning (so I'm not just copy-pasting swathes of configuration without understanding it), but this was just a quick demo which shows that: "basically, the userspace implementation isn't too slow".
- tachion 6y agoAt least in case of FreeBSD, the network saturation isn't an active goal of default kernel settings, hence the link I've pasted. It's especially nice, as it explains a lot of things it proposes so that the blind copy&paste wouldn't be so blind. It's really a good read. And I do get a point of your test and I agree with the anecdotal conclusion :)
- justin66 6y agoJudging from the "no vpn" set of numbers, the stock kernel is admirably prepared for the highest network throughput.
- pacija 6y agoSome quite knowledgeable people in the field of BSD networking, including Henning Brauer, maintainer of OpenBSD's PF, have little love for instruction given on site you are linking to: https://marc.info/?l=openbsd-misc&m=130105013025396&w=2 https://marc.info/?l=openbsd-misc&m=130105013025396&w=2
- craftkiller 6y agoNetgate[1] is sponsoring the development of an in-kernel implementation of wireguard for FreeBSD: https://forum.netgate.com/post/891869 https://forum.netgate.com/post/891869 [1]: Netgate is the company behind pfsense, a router/firewall distro of FreeBSD
- kitotik 6y agoOh this is awesome! A kernel space wireguard implementation and something like beehyve are the last 2 things I need to be able to start using fbsd a lot more.
- mdtancsa 6y agoAny idea whats the target date for it in the tree ?
- gonzo 6y agowhen it's ready, of course. It just started passing packets (ping) last week. It would have been at this point weeks ago, had Jason not baked his e-mail address into the handshake protocol. (Harumph.)
- ta1771 6y agoElaborate?
- loeg 6y agoMatt changing random algorithm parameters he didn't understand is kind of on him, sorry. I'm glad of the work he's doing, and your funding of FreeBSD native wireguard work, but just changing random cryptographic parameters before had had packets passing was an exercise in foot-shooting.
- joeschmooo 6y agoIt was certainly naive of him. However, can't deny that baking the authors e-mail & domain name in to the protocol is pretty narcissistic.
- loeg 6y agoIt can saturate 1Gbps with the TUN driver, sure. 10Gb is harder with TUN. Linux's native driver is lower overhead, although as siblings point out, there is work in progress on a native FreeBSD kernel driver.
- rubatuga 6y agoAnybody know what the benefits of a BSD jail might be over a VM?
- simcop2387 6y agolighter weight than a VM. It uses the same kernel as the rest of the system, but sandboxes the userspace to limit what impact it can have on the rest of the system. They're not quite equivalent to a container on Linux but they're very similar in functionality.
- Lammy 6y agoVNET jails, as the author is using, also have a completely stand-alone network stack from the host-OS!
- nine_k 6y agoAren't cgroups / containers a very close equivalent? In either case, it's namespacing, of a tree of userspace processes to make them invisible from the rest of the userspace.
- pacija 6y agoFreeBSD jails are additional jailed (chrooted) userlands using kernel of the host. They can be as fat as complete FreeBSD userland, or as thin as just a few binaries and libraries necessary to run particular service. https://www.freebsd.org/doc/handbook/jails.html https://www.freebsd.org/doc/handbook/jails.html
- jxy 6y agoWhat's the difference between the epair and the tap interfaces?
- sigwinch28 6y agoAn epair [0] is a pair of virtual ethernet interfaces which are connected together and have network addresses: ethernet frames that go in one interface come out of the other. A tap interface can be used by software to make ethernet traffic appear on an interface, i.e. software can write to a tap interface to simulate ethernet traffic being received on that interface, and an application can receive ethernet traffic from it. From the manpage [1]: "A write(2) call passes an Ethernet frame in to be "received" on the pseudo-interface. Each write() call supplies exactly one frame; the frame length is taken from the amount of data provided to write()." [0]: https://www.freebsd.org/cgi/man.cgi?query=epair&sektion=4&manpath=FreeBSD+12.1-RELEASE https://www.freebsd.org/cgi/man.cgi?query=epair&sektion=4&ma... [1]: https://www.freebsd.org/cgi/man.cgi?query=tap&sektion=4&manpath=FreeBSD+12.1-RELEASE https://www.freebsd.org/cgi/man.cgi?query=tap&sektion=4&manp...
- ignoramous 6y agoThanks. So, is it possible that wireguard writes to tap and the bridge or another interface on the host reads from it? Why would one prefer epairs over tap, if that's the case? And, I don't really get why wg-jail also needs default-router to be pointed to bridge0 when the author addms epair-b to bridge0 on the host (which file are the following lines added to anyway?): ---- cloned_interfaces="bridge0 epair0" ifconfig_bridge0="inet 192.168.20.1/24 addm epair0b up" ifconfig_epair0b="up" ---- And the explicit default-router definition for wg-jail, in vm/wg/etc/rc.conf: ---- defaultrouter="192.168.20.1" ----
- jlgaddis 6y ago> I don't really get why wg-jail also needs default-router to be pointed to bridge0 when the author addms epair-b to bridge0 on the host. The epair0 interfaces provide the layer 2 (Ethernet) connection between the jail and the host. The jail still needs a default IPv4 (layer 3) gateway so that it can route the traffic coming fron the WireGuard clients back out to the network/Interet (same as any other "router"). (Note: With just a single jail -- such as in this case -- the bridge0 interface isn't actually necessary (and the 192.168.20.1 address would then be assigned to the epair0b, not bridge0, interface on the host). The author went ahead and created a bridge with the intention to create additional jails in the future. This way, multiple jails can all be connected to the same internal "jail network". This is all mentioned in TFA, by the way.) > which file are the following lines added to anyway? cloned_interfaces="bridge0 epair0" ifconfig_bridge0="inet 192.168.20.1/24 addm epair0b up" ifconfig_epair0b="up" Those go in /etc/rc.conf on the host. defaultrouter="192.168.20.1" This goes in /etc/rc.conf on the jail (which corresponds to /vm/wg/etc/rc.conf on the host).
- rsync 6y agoA few different articles this week about spinning up a wireguard container/jail/VM ... But it's far, far easier to just fire up an sshd somewhere and 'sshuttle' makes it possible to turn any ssh server that you have a login on into a VPN endpoint: https://sshuttle.readthedocs.io/en/stable/ https://sshuttle.readthedocs.io/en/stable/ You don't even need to be a privileged user - just any old user login, over ssh, and you need python to exist on the remote system.
- bambambazooka 6y agoI absolutely love sshuttle but IMHO nothing beats WireGuard‘s availability and simplicity. I have it setup on my laptops, iPhone and iPad. It works transparently and I can access all the stuff in my homenetwork. I use sshuttle for situations where I don’t have root on a jumphost or only need the tunneling sometimes.
- nine_k 6y agoDid you not need to set up your home router, too?
- bambambazooka 6y agoNo, I have a small VPS that functions as my central WireGuard node.
- deleted 6y ago[deleted]
- tw04 6y agoI can't believe the performance of sshuttle is anywhere near Wireguard...?
- yjftsjthsd-h 6y agoIt wasn't when I used it. It's been too long to remember numbers, but I got a significant speed improvement by switching from sshuttle to wireguard.
- stiray 6y agoAs people are again asking about vm/jails/docker/... https://blog.jessfraz.com/post/containers-zones-jails-vms/ https://blog.jessfraz.com/post/containers-zones-jails-vms/ It is great article. Bottom line, jails are "in kernel" primitives. Containers are not (or at least they werent when I last checked).
- TsiCClawOfLight 6y agoContainers are basically a very thin layer on top of namespaces and cgroups, which are most definitely in kernel primitives. Docker adds packaging and distribution, but the hard work is in kernel.
- cpuguy83 6y agoIt's all hard work. The primitives of containerization are in the kernel, but executing and managing them, especially securely, takes a fair amount of trial and error to do it right.
- stiray 6y agoWhy is so hard to read an article from someone that is developing containers? Quote: "Again, containers were not a top level design, they are something we build from Linux primitives. Zones, Jails, and VMs are designed as top level isolation."
- heavyset_go 6y agoThe tl;dr of the post is that jails are a top-down solution whereas "containers" are just a bunch of kernel features and primitives used together.
- ggm 6y agoNot seeking a v4/v6 flame war, It would be interesting to see the IPv6 version of this for people who want to use WireGuard to protect IPv6 flows back to "inside" and so much of this is NAT related, its not generally applicable to that case.
- loeg 6y agoYou can NAT IPv6.
- ggm 6y agoThe NAT exists because it has to. You could NAT IPv6 to use the model as-is, but you could also re-do the model to not need NAT but still use wireguard to get through an ACL state which protects your boundary.
- whalesalad 6y agoIf you’re versed enough in the differences between ipv6 and ipv4 doing the delta from this post shouldn’t be too hard.