4 ms·

If you're looking for a proven protocol that's better suited to transferring vast amounts of data across the globe, while being based on UDP, and you have been

by cm3 10y ago

If you're looking for a proven protocol that's better suited to transferring vast amounts of data across the globe, while being based on UDP, and you have been using rsync or bittorrent, here's something that's less known but offers a good speed advantage: https://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protocol https://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protoc... Please don't complain about Sourceforge, UDT is an old project: http://udt.sourceforge.net/ http://udt.sourceforge.net/ And here's their distributed filesystem built on top of UDT: http://sector.sourceforge.net/ http://sector.sourceforge.net/ Edit: Protocol-level security is being worked on for UDT5, but in the meantime there's a Rust-based experimental attempt at replacing SCP with UDT-based solution: https://github.com/mcginty/shoop https://github.com/mcginty/shoop

_asummers 10y ago
Re: SourceForge, my understanding is that the previous owners sold SF in the past few months. The new owners have apparently taken down the malware and are working to try and restore it to its former name, despite the large incline of that hill. https://www.reddit.com/r/sysadmin/comments/4n3e1s/the_state_of_sourceforge_since_its_acquisition_in/ https://www.reddit.com/r/sysadmin/comments/4n3e1s/the_state_...
- cm3 10y ago
  Given that gforge is a fork of the original sourceforge code, it's probably possible to host a mirror to not repeat the loss of projects as happened with Google Code. There's just too much abandoned code on there, and most of it is still valuable. And even stuff not useful today is still valuable for archival purposes. Is there already a neocities like attempt at doing that?
Scaevolus 10y ago
I found UDT-rsync (the only file copying program that seems to incorporate it) to reliably segfault on files >1GB. I use uftp[1] now for bulk transfers across oceans-- it's a little confusing to use, but it generally accomplishes the goal of pushing packets at 70Mbps from Europe to Seattle. [1]: http://uftp-multicast.sourceforge.net/ http://uftp-multicast.sourceforge.net/
- cm3 10y ago
  Sounds like a bug/limitation in the code and goes against what UDT was designed for. Is there a ticket open? Thanks for the link to uftp, didn't know about. Also, UDT works in a LAN too, of course, and before finding UDT, I had been wondering how Weta Digital shares huge media files with Universal Studios in LA. Seems like UDT is built into a lot of commercial products.
  - mevodig 10y ago
    The "industry standard" software for this would be Aspera, now owned by IBM. They use their own UDP-based protocol called FASP.

by cm3 10y ago

_asummers 10y ago

Re: SourceForge, my understanding is that the previous owners sold SF in the past few months. The new owners have apparently taken down the malware and are working to try and restore it to its former name, despite the large incline of that hill. https://www.reddit.com/r/sysadmin/comments/4n3e1s/the_state_of_sourceforge_since_its_acquisition_in/ https://www.reddit.com/r/sysadmin/comments/4n3e1s/the_state_...

cm3 10y ago
Given that gforge is a fork of the original sourceforge code, it's probably possible to host a mirror to not repeat the loss of projects as happened with Google Code. There's just too much abandoned code on there, and most of it is still valuable. And even stuff not useful today is still valuable for archival purposes. Is there already a neocities like attempt at doing that?

Scaevolus 10y ago

I found UDT-rsync (the only file copying program that seems to incorporate it) to reliably segfault on files >1GB. I use uftp[1] now for bulk transfers across oceans-- it's a little confusing to use, but it generally accomplishes the goal of pushing packets at 70Mbps from Europe to Seattle. [1]: http://uftp-multicast.sourceforge.net/ http://uftp-multicast.sourceforge.net/

cm3 10y ago
Sounds like a bug/limitation in the code and goes against what UDT was designed for. Is there a ticket open? Thanks for the link to uftp, didn't know about. Also, UDT works in a LAN too, of course, and before finding UDT, I had been wondering how Weta Digital shares huge media files with Universal Studios in LA. Seems like UDT is built into a lot of commercial products.
- mevodig 10y ago
  The "industry standard" software for this would be Aspera, now owned by IBM. They use their own UDP-based protocol called FASP.