5 ms·

Congrats on the launch! Could you expand a bit on how you differentiate your product from other products in the space like Tailscale and Nebula? Edit - I see y

by jwcrux 5y ago

Congrats on the launch! Could you expand a bit on how you differentiate your product from other products in the space like Tailscale and Nebula? Edit - I see you mention that Tailscale uses userland WireGuard. Is that the biggest difference between the two? Do you foresee yourselves running into issues by not using the userland implementation?

afeiszli 5y ago
The biggest differences with Tailscale are that you can use the kernel version (speed) and that we are self-hostable. We actually noticed a major contributor to the speed difference between us and them was a lot of times, they'll route traffic through their relays which can eat up a good amount of time. With Nebula, they're a lot closer on speed, but also we've got a management GUI which makes things a lot easier. We were very close to using Nebula in our early days. The main thing that stopped us was, we decided WireGuard was going to be the standard in the future, and wanted to be based on WireGuard. That leads to a bit more fundamental of a difference which is a bit harder to quantify. Our aim is to really be a "WireGuard controller." You should be able to shut down our server and agents and your network should still run fine, and you should be able to manually modify all your WireGuard interfaces if necessary. We're getting close to that vision but aren't quite there yet. That last point leads back to the kernel thing. We use kernel by default, but really, Netmaker can use any WireGuard implementation. If users are scared of the security implications of using kernel, they can use the userland version, and Netmaker should be able to pick that up just fine. They can even run it in a docker container on their machine.
- gunapologist99 5y ago
  This is great! Can you also compare to Headscale and Zerotier? How do you handle NAT compared to these other options? I've run into issues with WireGuard when both sides are behind residential NATs (or AWS EC2 IGW); as you know, Nebula solves this with "lighthouse" servers (which are self-hosted but externally accessible), and Tailscale uses its third-party intro devices. Do the Gravitl servers have to always listen on an exposed port?
  - afeiszli 5y ago
    It is way, way faster than Zerotier. Headscale I am less familiar with, and they do make Tailscale self-hostable, but they're a community-driven project I'm not sure how reliable it'll be long term (can't speak for the author). On the NAT side, we provide 3 layers of traversal options: #1 (default) port forwarding: This actually works in a surprisingly well, for about 90% of environments, but does require an exposed port. #2 UDP Hole Punching: The server acts similarly to a Nebula "lighthouse" and will tell clients where to reach each other. This covers that small situation of dualing NAT's, and doesn't require the exposed port. #3 Relay: In situations where neither works such as CGNAT, you can set a public node as a relay to route traffic to the "hidden" node
    - gunapologist99 5y ago
      Awesome, thanks!
    - easton 5y ago
      Tailscale has some functionality to automatically determine what flavor of NAT traversal is necessary, this isn’t a big deal but do I have to configure the nodes to use these methods or will your client figure this out itself?
      - afeiszli 5y ago
        Right now you need to choose your own option. We're planning to automate this in the next couple of months. I will say, this isn't always a bad thing. We have a lot of users who switched from Tailscale because it was frequently falling back to relay, often via public relays that were hundreds of miles away from them, causing big increases in latency. We want to give people the automation, but also the choice.
        easton 5y ago
        Good idea! I’ll have to give this a spin, I like the idea of “nebula but wireguard”.
- lwhsiao 5y ago
  Can you compare to innernet from tonari?
  - afeiszli 5y ago
    Mainly we give a lot more configuration options as well as a GUI. I'm also not sure if they run on anything other than Linux (I think not?). We can run on Mac, FreeBSD, and Windows. If you're just looking to create a pure mesh network with linux-based devices, they're definitely comparable and I think you'd just have to try out both and see what you like more.
- anderspitman 5y ago
  > You should be able to shut down our server and agents and your network should still run fine, and you should be able to manually modify all your WireGuard interfaces if necessary. This is excellent.
- rhuber 5y ago
  (coauthor of Nebula) We briefly considered building something atop Wireguard in the early days of Nebula, but decided not to do so because of scaling. Wireguard's protocol necessitates that all nodes have existing keypairs for each other ahead of time. At Slack's scale, that means every time a fresh node is launched, you would have to tell 50,000 other nodes it exists. Obviously you can smarten this up and tell only hosts it might talk to. But this adds complexity. Using PKI eliminates this key distribution problem and means that you don't have the same scaling limitations as something built on WG. Wireguard is a very very good VPN, but I cannot imagine trying to run something on the scale of tens of thousands of nodes when you need such complex coordination systems to exchange keys/trust, especially in a dynamic environment where nodes are coming and going all the time. I totally get that it solvable overall, but Slack has had 4 years of nearly perfect uptime on Nebula, whilst using it to pass >95% of all backend traffic. These considerations may seem simple to address, but there are fundamentals that mattered and led us to writing Nebula. We didn't want to create something new, but to do what Slack needed, we had to.
deleted 5y ago
[deleted]

by jwcrux 5y ago

afeiszli 5y ago

The biggest differences with Tailscale are that you can use the kernel version (speed) and that we are self-hostable. We actually noticed a major contributor to the speed difference between us and them was a lot of times, they'll route traffic through their relays which can eat up a good amount of time. With Nebula, they're a lot closer on speed, but also we've got a management GUI which makes things a lot easier. We were very close to using Nebula in our early days. The main thing that stopped us was, we decided WireGuard was going to be the standard in the future, and wanted to be based on WireGuard. That leads to a bit more fundamental of a difference which is a bit harder to quantify. Our aim is to really be a "WireGuard controller." You should be able to shut down our server and agents and your network should still run fine, and you should be able to manually modify all your WireGuard interfaces if necessary. We're getting close to that vision but aren't quite there yet. That last point leads back to the kernel thing. We use kernel by default, but really, Netmaker can use any WireGuard implementation. If users are scared of the security implications of using kernel, they can use the userland version, and Netmaker should be able to pick that up just fine. They can even run it in a docker container on their machine.

gunapologist99 5y ago
This is great! Can you also compare to Headscale and Zerotier? How do you handle NAT compared to these other options? I've run into issues with WireGuard when both sides are behind residential NATs (or AWS EC2 IGW); as you know, Nebula solves this with "lighthouse" servers (which are self-hosted but externally accessible), and Tailscale uses its third-party intro devices. Do the Gravitl servers have to always listen on an exposed port?
- afeiszli 5y ago
  It is way, way faster than Zerotier. Headscale I am less familiar with, and they do make Tailscale self-hostable, but they're a community-driven project I'm not sure how reliable it'll be long term (can't speak for the author). On the NAT side, we provide 3 layers of traversal options: #1 (default) port forwarding: This actually works in a surprisingly well, for about 90% of environments, but does require an exposed port. #2 UDP Hole Punching: The server acts similarly to a Nebula "lighthouse" and will tell clients where to reach each other. This covers that small situation of dualing NAT's, and doesn't require the exposed port. #3 Relay: In situations where neither works such as CGNAT, you can set a public node as a relay to route traffic to the "hidden" node
  - gunapologist99 5y ago
    Awesome, thanks!
  - easton 5y ago
    Tailscale has some functionality to automatically determine what flavor of NAT traversal is necessary, this isn’t a big deal but do I have to configure the nodes to use these methods or will your client figure this out itself?
    - afeiszli 5y ago
      Right now you need to choose your own option. We're planning to automate this in the next couple of months. I will say, this isn't always a bad thing. We have a lot of users who switched from Tailscale because it was frequently falling back to relay, often via public relays that were hundreds of miles away from them, causing big increases in latency. We want to give people the automation, but also the choice.
      - easton 5y ago
        Good idea! I’ll have to give this a spin, I like the idea of “nebula but wireguard”.
lwhsiao 5y ago
Can you compare to innernet from tonari?
- afeiszli 5y ago
  Mainly we give a lot more configuration options as well as a GUI. I'm also not sure if they run on anything other than Linux (I think not?). We can run on Mac, FreeBSD, and Windows. If you're just looking to create a pure mesh network with linux-based devices, they're definitely comparable and I think you'd just have to try out both and see what you like more.
anderspitman 5y ago
> You should be able to shut down our server and agents and your network should still run fine, and you should be able to manually modify all your WireGuard interfaces if necessary. This is excellent.
rhuber 5y ago
(coauthor of Nebula) We briefly considered building something atop Wireguard in the early days of Nebula, but decided not to do so because of scaling. Wireguard's protocol necessitates that all nodes have existing keypairs for each other ahead of time. At Slack's scale, that means every time a fresh node is launched, you would have to tell 50,000 other nodes it exists. Obviously you can smarten this up and tell only hosts it might talk to. But this adds complexity. Using PKI eliminates this key distribution problem and means that you don't have the same scaling limitations as something built on WG. Wireguard is a very very good VPN, but I cannot imagine trying to run something on the scale of tens of thousands of nodes when you need such complex coordination systems to exchange keys/trust, especially in a dynamic environment where nodes are coming and going all the time. I totally get that it solvable overall, but Slack has had 4 years of nearly perfect uptime on Nebula, whilst using it to pass >95% of all backend traffic. These considerations may seem simple to address, but there are fundamentals that mattered and led us to writing Nebula. We didn't want to create something new, but to do what Slack needed, we had to.

deleted 5y ago

[deleted]