4 ms·
Years back someone proposed a cute algorithm for erasure codes that depended not on spinning rust but on multipath networking. I believe they called it network
by hinkley 2y ago
Years back someone proposed a cute algorithm for erasure codes that depended not on spinning rust but on multipath networking.
I believe they called it network coding and the idea was in a network with multiple routes I might get a file faster by pulling an erasure code that used two parts of the file or even two files from one upstream instead of waiting for the entire file from the primary server.
- nullc 2y agoNetwork coding is more than that, participants in the graph can synthesize new parts on the fly from parts they just got without having the whole thing. FWIW, freenet at least uses fec-coded files so that you can have some flexibility in what parts you get and durability against a file becoming broken just because a single part gets lost.
- StillBored 2y agoand usenet binaries with https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive and even earlier with RAR's recovery block, and probably even earlier with BBSes, but my memory is failing me what I was using before it in the 1990s. edit: I see immediately below while I was composing this, someone mentioned pararchive..
- phonon 2y agoWith something like https://github.com/cberner/raptorq https://github.com/cberner/raptorq you can do several gbits/s over high latency/lossy UDP.
- sva_ 2y agoI thought about something like that to make video calls more stable. For example I'll get completely different routes (sometimes noticeably lower/more predictable latency) when I use a VPN to connect to some peer in the US (from Europe.) Would be cool to combine different routes.
- supertrope 2y agoIt’s called SD-WAN
- kayg04 2y agocan you explain? I tried looking it up but I didn't quite understand how it is called SD-WAN.
- toast0 2y agoOne difficulty with using multiple routes is you'll probably need to spend a lot of bytes on active probing, because the quality of a connection may change during the call and when the active connection loses quality it's not apparent what the other connections will do, so you need recent probing from them as well to make a smart change. But in theory, you should have several routing options in a well supported calling service. I'll illustrate one direction, but the same options apply in the other direction, and there's no need for both peers to use the same connection to send (although WebRTC will) Peer A -> Peer B Peer A -> Relay near A -> Peer B Peer A -> Relay near B -> Peer B Peer A -> Relay near A -> Relay near B -> Peer B If at least two of the four hosts mentioned have IPv4 and IPv6, you can also add those permutations. It's pretty typical to have different routing for v4 and v6.
- epistasis 2y agoThis has been used in Ceph for a long time: https://docs.ceph.com/en/latest/rados/operations/erasure-code/ https://docs.ceph.com/en/latest/rados/operations/erasure-cod... I would not be surprised if there was a lot of stuff like this behind S3 and other cloud storage systems too, at least in the lower-access tiers of storage, but I have no actual knowledge of AWS or GCP systems.
- preisschild 2y agoYeah, I use it in my homelab and it is really awesome to have "RAID(5/6)" basically work over the network.