5 ms·

In the self-hosted space, I've been really enjoying playing around with decentralized encrypted overlay mesh networks like Nebula. Here's the current list of my

• brendoncarroll 3y ago
  I'll throw in INET256 https://github.com/inet256 https://github.com/inet256 It's a specification for identity based networking. There is a meshnet and a centralized implementation. You can layer IPv6, IPv4, or application traffic on top of any compatible implementation.
• imiric 3y ago
  No love for tinc[1]? It's the granddaddy of mesh networking, long before Wireguard, and while it's not quite zeroconf, it's very simple to setup and maintain. It also runs on everything. [1]: https://tinc-vpn.org/ https://tinc-vpn.org/
  • wkat4242 3y ago
    Yeah it's still my go-to because it's fully self hosted. Nebula can be too but it uses certs that can expire whereas tinc just uses keys. And really I've been using tinc for almost a decade and I didn't really see the benefit of changing. It's rock-solid. With the exception of one thing: I use some central nodes on cloud VPSes and they can access everything. As far as I know a nebula lighthouse can't access any of the clients. So I've been meaning to give nebula another try. But zerotier and tailscale aren't options for me because they rely on their cloud infrastructure. I only want stuff that's fully self-hosted. There's a great tinc android client these days too.
    • nh2 3y ago
      > It's rock-solid. Unfortunately, I cannot confirm. Sharing my experience: I used tinc over multiple years on production servers and it would sometimes create netsplits that did not recover. I also suspect that there's a race or bug in re-keying, which also causes disconnects. On the netsplit issue, it was me posting alone on the relevant issue [1] over multiple years without response. (I don't expect to get any from free-time maintainers, especially on hard-to-reproduce issues, but it's still important to know that such unsolvable hurdles exist.) When I switched to Nebula, it improved this situation. But both Nebula and tinc max out at around 1 Gbit/s on my Hetzner servers, thus not using most of my 10 Gbit/s connectivity. This is because they cap out at 100% of 1 CPU. The Nebula issue about that was closed due to "inactivity" [2]. I also observed that when Nebula operates at 100% CPU usage, you get lots of package loss. This causes software that expects reasonable timings on ~0.2ms links to fail (e.g. consensus software like Consul, or Ceph). This in turn led to flakiness / intermittent outages. I had to resolve to move the big data pushing softwares like Ceph outside of the VPN to get 10 Gbit/s speed for those, and to avoid downtimes due to the packet loss. Such software like Ceph has its own encryption, but I don't trust it, and that mistrust was recently proven right again [3]. So I'm currently looking to move the Ceph into WireGuard. Summary: For small-data use, tinc and Nebula are fine, but if you start to push real data, they break. [1]: https://github.com/gsliepen/tinc/issues/218 https://github.com/gsliepen/tinc/issues/218 [2]: https://github.com/slackhq/nebula/issues/637 https://github.com/slackhq/nebula/issues/637 [3]: https://github.com/google/security-research/security/advisories/GHSA-jg27-jx6w-xwph https://github.com/google/security-research/security/advisor...
      • imiric 3y ago
        That's good to know, thanks. My tinc usage was far less demanding. Just a handful of nodes and light traffic, so I didn't experience any issues. I did migrate to WireGuard about a year ago, and instead of a mesh network, I ended up with a hub-and-spoke configuration[1] which worked fine for my humble needs. [1]: https://www.procustodibus.com/blog/2020/11/wireguard-hub-and-spoke-config/ https://www.procustodibus.com/blog/2020/11/wireguard-hub-and...
      • radlad 3y ago
  • lenova 3y ago
    > No love for tinc[1]? I actually looked into tinc, and really wanted to love it because of its simplicity. Unfortunately though, it seems that the development scene around it has been stalled for several years now, and benchmark reports show that it can't keep up with cloud speed tests when compared with Wireguard: - Wireguard: 390.3 Mbps - Netmarker: 369.3 Mbps - Tailscale: 62.5 Mbps - ZeroTier: 56.8 Mbps - Nebula: 38.4 Mbps - Tinc: 34.7 Mbps - OpenVPN: 22.3 Mbps Source: [disclaimer, article written by Netmaker CEO, so implicit bias] (https://medium.com/netmaker/battle-of-the-vpns-which-one-is-fastest-speed-test-21ddc9cd50db https://medium.com/netmaker/battle-of-the-vpns-which-one-is-...)
    • imiric 3y ago
      Hhmm I don't remember it being that slow, but then again, I didn't need much throughput for my use cases. Frankly, I'm surprised that Zerotier and Tailscale are that slow as well, so I'm not sure how trustworthy that article is. I found this other benchmark[1] that places them much closer to native WG performance, with Netmaker still in the lead, and somehow faster than a direct connection. This is probably due more to Netmaker using the in-kernel WG, instead of the userspace implementation. I'm kind of curious to give it a try, TBH. :) Hell, even OpenVPN is _much_ faster than that[2,3]. Also, yeah, tinc development has been slow for years now, but I didn't experience any issues with the prerelease versions. It's certainly behind the times now, but I also enjoyed how easy it was to configure and use. [1]: https://techoverflow.net/2022/08/19/iperf-benchmark-of-zerotier-vs-netmaker-vs-tailscale-vs-direct-switched-connection/ https://techoverflow.net/2022/08/19/iperf-benchmark-of-zerot... [2]: https://www.zerotier.com/blog/benchmarking-zerotier-vs-openvpn-and-linux-ipsec/ https://www.zerotier.com/blog/benchmarking-zerotier-vs-openv... [3]: https://www.wireguard.com/performance/ https://www.wireguard.com/performance/
      • BitPirate 3y ago
        Tailscale is using GSO/GRO now. They blogged about reaching the 10Gbit/s milestone. https://tailscale.com/blog/more-throughput/ https://tailscale.com/blog/more-throughput/
    • lenova 3y ago
• khimaros 3y ago
  filling a slightly different niche, there's also yggdrasil, which can be used to create private overlay network taking advantage of public relay nodes.
• RealStickman_ 3y ago
  There's also Yggdrasil [0], an ipv6 based completely decentralised network. You can join the public network or restrict access to known keys for a private network. [0] https://yggdrasil-network.github.io/ https://yggdrasil-network.github.io/