4 ms·
Have you tried https://github.com/openrq-team/OpenRQ https://github.com/openrq-team/OpenRQ or https://github.com/harmony-one/go-raptorq https://github.com/harm
by phonon 7y ago
Have you tried https://github.com/openrq-team/OpenRQ https://github.com/openrq-team/OpenRQ or https://github.com/harmony-one/go-raptorq https://github.com/harmony-one/go-raptorq or https://github.com/cberner/raptorq https://github.com/cberner/raptorq (the last seems to have excellent decoding/encoding performance)?
(Raptor codes (a type of FEC) can essentially transfer over UDP at the underlying line rate, even if packet drop is high. In other words, if it is a 1 gbit/s link, with 5% packet loss, you will be able to send over 900 mbits/s without needing any TCP features). This is also helps high latency scenarios, as you don't need test/back off how much data to send...you just send as much Raptor encoded data as you want down the pipe, and as long as you send enough recovery packets with it, you will be able to reconstruct the original flawlessly).
(Also see https://par.nsf.gov/servlets/purl/10066600 https://par.nsf.gov/servlets/purl/10066600
http://www1.icsi.berkeley.edu/~pooja/HowUseRaptorQ.pdf http://www1.icsi.berkeley.edu/~pooja/HowUseRaptorQ.pdf
https://a1f9fb7d-b120-4c73-98e3-5cdb4ec8a2ab.filesusr.com/ugd/4c5b11_93bc499bed254bf29057bfb5f8c44867.pdf https://a1f9fb7d-b120-4c73-98e3-5cdb4ec8a2ab.filesusr.com/ug...
https://youtu.be/bYPbat-FFTo https://youtu.be/bYPbat-FFTo
https://github.com/mk-fg/python-libraptorq https://github.com/mk-fg/python-libraptorq)
- Scaevolus 7y agoYou don't need 5% packet loss on a high latency 1Gbps link to trash performance. Even .1% packet loss is enough to cut throughput massively on long fat networks (LFNs). Erasure coding helps, but it demands much more CPU than simply having a very wide window for retransmits. For bulk data transfer of files, retransmitting any part of a 1GB window is trivial.
- traverseda 7y agoYou mean with raptorQ 5% loss trashes performance by more than 5%?
- phonon 7y agoNo, the overhead is fairly minimal. A 5% packet loss (which is very bad) on a gigabit link where the sender is thousands of miles away might have an effective throughput of 930 mbits per second with RaptorQ. With regular TCP you would get only a few megabits/s, as it would back down until the loss rate was nearly zero.
- phonon 7y agoYes, of course--even a low rate of packet loss where the bandwidth delay product is high will cause you to end up with throughout dropping to critically terrible levels. Tweaking TCP can help, but a high performance RaptorQ system can work at packet loss levels that TCP could never handle.
- matsur 7y agoI’m on the product team at Cloudflare. If you run a throughput sensitive workload on Aspera or similar, I’d love to chat — rustam@cloudflare.com