3 ms·

ZeroTier One 1.0.0: Better, Faster, Cheaper

6 points by api 12y ago

wmf 12y ago
Congrats. Is there a writeup of the old and new multicast algorithms? It sounds like the old one was building a spanning tree and the new one does sender-side replication; is that close?
- api 12y ago
  (I'm the author/founder.) Old one did distributed breadth-first graph exploration: https://github.com/zerotier/ZeroTierOne/wiki/OLD-Multicast-Algorithm-Notes https://github.com/zerotier/ZeroTierOne/wiki/OLD-Multicast-A... It was nifty and cool and worked pretty well for small to mid tier stuff, but biggest disadvantages in practice were: (1) Peers seem to drop the potato more often than I predicted when I simulated it, leading to a lot of truncated propagations. It also has no geo-awareness, so if your propagation graph goes through sub-Saharan Africa you're likely to see a lot of packet loss. I think my simulations implicitly assumed all high-end first-world broadband links, when reality includes the whole world and a lot of slow/crummy cellular. (2) High latency, which caused problems for people who wanted low-latency multicast for SDN-type applications (enterprise stuff). In the beginning I never considered those applications, but they're emerging as a major user base. (3) Imposed a high CPU cost, since to prevent flooding every packet had to be cryptographically signed by its originating node. That's a lot of ECC256 signatures and verifications. (4) Mobile-unfriendly, since mobile devices really want to be able to enter long-term hibernation and not have to wake to help others propagate their multicasts. Mobile might even want to turn off most multicast, only allowing ARP and NDP or even emulating these for no multicast at all. (Currently working on iOS and Android ZT1 ports.) This also applies to tiny low power embedded devices, and I've had some interest from Internet-of-things experimenters. There's no write-up on the new algorithm yet, but it's basically sender-side replication with lazy but deterministic gathering. It gathers recipients up to the limit from supernodes and from peers. (In reality supernodes are peers-- they're merely peers designated as such.) I remember initially passing on that idea since I was afraid it would be too heavy on the sender, but observing real traffic flows shows that in aggregate the multicast traffic load ends up close to the same. Sender-side replication is also inherently fair -- sender pays the price -- so you don't need such heavy signature/verification checks to ensure that somebody can't flood the network. This way it's no easier to flood with multicast than with unicast. (Obviously anyone can DDOS but that's another matter.) In the longer term I plan to add some support for merging of multicasts going to the same local network or site, but I'm not sure how yet... it's probably going to intersect with R&D work I'm doing right now on enterprise supernode federation for SDN applications and similar stuff.
hxm 12y ago
Hi, Basic (newbie) questions: 1. Once on the network, does one have a conventional IPv4 address ? 2. What IP address range should one best choose for a small network? 3. How does this compare (technically and/or practically) to the use of 25.x.x.x addresses by Hamachi ? Thanks!
tekacs 12y ago
This is why I'm glad I read https://news.ycombinator.com/classic https://news.ycombinator.com/classic [1][2]. Just another recommendation for anyone who stumbles across this article later - it currently seems unlikely to reach the front page :( P.S. This is hugely useful - I have an immediate use for it in connecting many disparate daemons of the project I'm currently working on (I'd played with Hamachi, Docker/Weave and full custom on OpenVPN until now). :) [1]: even if I'm not amongst its voters. :P [2]: this is on the third page of /news at the time of writing.
- api 12y ago
  Thanks! May I ask: given the freemium model here, what features or services would make you more likely to sign up for a paid account? I'm looking for ways to add more value to the professional side of things.
  - tekacs 12y ago
    Replied by e-mail. Best of luck!

6 points by api 12y ago

wmf 12y ago

Congrats. Is there a writeup of the old and new multicast algorithms? It sounds like the old one was building a spanning tree and the new one does sender-side replication; is that close?

api 12y ago
(I'm the author/founder.) Old one did distributed breadth-first graph exploration: https://github.com/zerotier/ZeroTierOne/wiki/OLD-Multicast-Algorithm-Notes https://github.com/zerotier/ZeroTierOne/wiki/OLD-Multicast-A... It was nifty and cool and worked pretty well for small to mid tier stuff, but biggest disadvantages in practice were: (1) Peers seem to drop the potato more often than I predicted when I simulated it, leading to a lot of truncated propagations. It also has no geo-awareness, so if your propagation graph goes through sub-Saharan Africa you're likely to see a lot of packet loss. I think my simulations implicitly assumed all high-end first-world broadband links, when reality includes the whole world and a lot of slow/crummy cellular. (2) High latency, which caused problems for people who wanted low-latency multicast for SDN-type applications (enterprise stuff). In the beginning I never considered those applications, but they're emerging as a major user base. (3) Imposed a high CPU cost, since to prevent flooding every packet had to be cryptographically signed by its originating node. That's a lot of ECC256 signatures and verifications. (4) Mobile-unfriendly, since mobile devices really want to be able to enter long-term hibernation and not have to wake to help others propagate their multicasts. Mobile might even want to turn off most multicast, only allowing ARP and NDP or even emulating these for no multicast at all. (Currently working on iOS and Android ZT1 ports.) This also applies to tiny low power embedded devices, and I've had some interest from Internet-of-things experimenters. There's no write-up on the new algorithm yet, but it's basically sender-side replication with lazy but deterministic gathering. It gathers recipients up to the limit from supernodes and from peers. (In reality supernodes are peers-- they're merely peers designated as such.) I remember initially passing on that idea since I was afraid it would be too heavy on the sender, but observing real traffic flows shows that in aggregate the multicast traffic load ends up close to the same. Sender-side replication is also inherently fair -- sender pays the price -- so you don't need such heavy signature/verification checks to ensure that somebody can't flood the network. This way it's no easier to flood with multicast than with unicast. (Obviously anyone can DDOS but that's another matter.) In the longer term I plan to add some support for merging of multicasts going to the same local network or site, but I'm not sure how yet... it's probably going to intersect with R&D work I'm doing right now on enterprise supernode federation for SDN applications and similar stuff.

hxm 12y ago

Hi, Basic (newbie) questions: 1. Once on the network, does one have a conventional IPv4 address ? 2. What IP address range should one best choose for a small network? 3. How does this compare (technically and/or practically) to the use of 25.x.x.x addresses by Hamachi ? Thanks!

tekacs 12y ago

This is why I'm glad I read https://news.ycombinator.com/classic https://news.ycombinator.com/classic [1][2]. Just another recommendation for anyone who stumbles across this article later - it currently seems unlikely to reach the front page :( P.S. This is hugely useful - I have an immediate use for it in connecting many disparate daemons of the project I'm currently working on (I'd played with Hamachi, Docker/Weave and full custom on OpenVPN until now). :) [1]: even if I'm not amongst its voters. :P [2]: this is on the third page of /news at the time of writing.

api 12y ago
Thanks! May I ask: given the freemium model here, what features or services would make you more likely to sign up for a paid account? I'm looking for ways to add more value to the professional side of things.
- tekacs 12y ago
  Replied by e-mail. Best of luck!