4 ms·

The first thing I tried to find on their website and their GitHub was a protocol specification, to be able to implement it independently from the reference impl

The first thing I tried to find on their website and their GitHub was a protocol specification, to be able to implement it independently from the reference implementation. I thought this would be straightforward since it’s advertised as a scheme/protocol, but such a spec isn’t referenced anywhere! Digging on my own I eventually found [1] on a side-branch of one of their other GitHub projects. Kudos to the author: I think it actually covers a lot of what you’d need to know: crypto identities, message formats, wire protocols, peering and stream semantics, spanning tree updates and root selection, the DHT, forwarding logic, sessions, etc. A couple things are TODOs like how to verify and sign root updates, and there’s some ambiguity in the tiebreaker algorithm for next-hop selection. It seems to be very tightly coupled to TCP as the transport layer though, since all packets need to be delivered reliably and in the order they were sent, and need to be capable of being fragmented into smaller packets for varying MTU sizes. [1] https://github.com/yggdrasil-network/yggdrasil-specs/blob/ys001/ys001-yggdrasil-core-specification.md https://github.com/yggdrasil-network/yggdrasil-specs/blob/ys...

neilalexander 2y ago
We did spend a little bit of time documenting the earlier v0.3 protocol, as you have linked, but the protocol has changed significantly in design twice since then. v0.4 changed the DHT quite a bit and v0.5 removed the DHT altogether. As a research project it likely will continue to change until we settle on a design we are happier with, at which point we will definitely spend more time documenting it. The need for ordered/reliable links is mostly for convenience of development at this stage, but that can be fixed for sure.
- Rhapso 2y ago
  Look at https://arxiv.org/pdf/1502.06461 https://arxiv.org/pdf/1502.06461 if you want to try a chord dht again. Kademlia is a lot less intuitive, but by not ever assuming it's tables are correct, it handles and corrects inconsistency (and malicious nodes) better. Chapter 6 of this pile of (my) crap https://scholarworks.gsu.edu/cs_diss/106/ https://scholarworks.gsu.edu/cs_diss/106/ talks about doing latency optimization on dht routing. Basically just embedding then network graph into a metric space.
- godelski 2y ago
  Some documentation can help with those issues though. I find it helps more because you’re writing to yourself why you’re making certain decisions and it helps when you decide to make others. It just so happens that it’s also a great way to onboard people.
colordrops 2y ago
Is coupling with TCP a problem? Does it do anything that goes against their goal of full decentralization?
- blurbleblurble 2y ago
  Makes it hard to do hole punching I think? At any rate, direct connections currently cannot be established between multi-hop peers, traffic gets routed through peers instead. I think this has something to do with the TCP choice.
  - foundry27 2y ago
    Yeaaah. TCP hole punching is goofy and unreliable, last I checked. You have to do some arcane ritual of having both peers start a three-way handshake to each others’s public endpoints simultaneously, relying on NATs to accept inbound SYN packets if they match the outgoing SYN. And nobody’s NAT devices implement simultaneous-open the same way, so all your connections just fail. Naturally this leads to slapping even more arcane fixes on top of that, like NAT port assignment oracles to adversarial interoperate with different port allocation strategies (random, sequential, single, etc.) by analyzing patterns in previous port assignments. Networking sucks.
    - beeflet 2y ago
      https://xkcd.com/2044/ https://xkcd.com/2044/
      - paulddraper 2y ago
        Actuate
        ionspin 2y ago
        I presume you meant to say "Accurate", but it made me think of a off-brand Picard that says "Actuate" instead of "Engage".

neilalexander 2y ago

We did spend a little bit of time documenting the earlier v0.3 protocol, as you have linked, but the protocol has changed significantly in design twice since then. v0.4 changed the DHT quite a bit and v0.5 removed the DHT altogether. As a research project it likely will continue to change until we settle on a design we are happier with, at which point we will definitely spend more time documenting it. The need for ordered/reliable links is mostly for convenience of development at this stage, but that can be fixed for sure.

Rhapso 2y ago
Look at https://arxiv.org/pdf/1502.06461 https://arxiv.org/pdf/1502.06461 if you want to try a chord dht again. Kademlia is a lot less intuitive, but by not ever assuming it's tables are correct, it handles and corrects inconsistency (and malicious nodes) better. Chapter 6 of this pile of (my) crap https://scholarworks.gsu.edu/cs_diss/106/ https://scholarworks.gsu.edu/cs_diss/106/ talks about doing latency optimization on dht routing. Basically just embedding then network graph into a metric space.
godelski 2y ago
Some documentation can help with those issues though. I find it helps more because you’re writing to yourself why you’re making certain decisions and it helps when you decide to make others. It just so happens that it’s also a great way to onboard people.

colordrops 2y ago

Is coupling with TCP a problem? Does it do anything that goes against their goal of full decentralization?

blurbleblurble 2y ago
Makes it hard to do hole punching I think? At any rate, direct connections currently cannot be established between multi-hop peers, traffic gets routed through peers instead. I think this has something to do with the TCP choice.
- foundry27 2y ago
  Yeaaah. TCP hole punching is goofy and unreliable, last I checked. You have to do some arcane ritual of having both peers start a three-way handshake to each others’s public endpoints simultaneously, relying on NATs to accept inbound SYN packets if they match the outgoing SYN. And nobody’s NAT devices implement simultaneous-open the same way, so all your connections just fail. Naturally this leads to slapping even more arcane fixes on top of that, like NAT port assignment oracles to adversarial interoperate with different port allocation strategies (random, sequential, single, etc.) by analyzing patterns in previous port assignments. Networking sucks.
  - beeflet 2y ago
    https://xkcd.com/2044/ https://xkcd.com/2044/
    - paulddraper 2y ago
      Actuate
      - ionspin 2y ago
        I presume you meant to say "Accurate", but it made me think of a off-brand Picard that says "Actuate" instead of "Engage".