3 ms·
> It's rock-solid. Unfortunately, I cannot confirm. Sharing my experience: I used tinc over multiple years on production servers and it would sometimes create
by nh2 3y ago
> It's rock-solid.
Unfortunately, I cannot confirm. Sharing my experience:
I used tinc over multiple years on production servers and it would sometimes create netsplits that did not recover.
I also suspect that there's a race or bug in re-keying, which also causes disconnects.
On the netsplit issue, it was me posting alone on the relevant issue [1] over multiple years without response. (I don't expect to get any from free-time maintainers, especially on hard-to-reproduce issues, but it's still important to know that such unsolvable hurdles exist.)
When I switched to Nebula, it improved this situation.
But both Nebula and tinc max out at around 1 Gbit/s on my Hetzner servers, thus not using most of my 10 Gbit/s connectivity. This is because they cap out at 100% of 1 CPU. The Nebula issue about that was closed due to "inactivity" [2].
I also observed that when Nebula operates at 100% CPU usage, you get lots of package loss. This causes software that expects reasonable timings on ~0.2ms links to fail (e.g. consensus software like Consul, or Ceph). This in turn led to flakiness / intermittent outages.
I had to resolve to move the big data pushing softwares like Ceph outside of the VPN to get 10 Gbit/s speed for those, and to avoid downtimes due to the packet loss.
Such software like Ceph has its own encryption, but I don't trust it, and that mistrust was recently proven right again [3].
So I'm currently looking to move the Ceph into WireGuard.
Summary: For small-data use, tinc and Nebula are fine, but if you start to push real data, they break.
[1]: https://github.com/gsliepen/tinc/issues/218 https://github.com/gsliepen/tinc/issues/218
[2]: https://github.com/slackhq/nebula/issues/637 https://github.com/slackhq/nebula/issues/637
[3]: https://github.com/google/security-research/security/advisories/GHSA-jg27-jx6w-xwph https://github.com/google/security-research/security/advisor...
- imiric 3y agoThat's good to know, thanks. My tinc usage was far less demanding. Just a handful of nodes and light traffic, so I didn't experience any issues. I did migrate to WireGuard about a year ago, and instead of a mesh network, I ended up with a hub-and-spoke configuration[1] which worked fine for my humble needs. [1]: https://www.procustodibus.com/blog/2020/11/wireguard-hub-and-spoke-config/ https://www.procustodibus.com/blog/2020/11/wireguard-hub-and...
- radlad 3y agoHey there - sorry for closing your issue! I saw Nate had recommended some commands to run and didn't see a response 8 months later, so I assumed you had moved on. I've reopened the issue so you can continue to document and look for solutions. :)