5 ms·

Ask HN: Wanted: reliable, very slow file copying over UDP

I have a bizarre requirement that I'm about to have to code, but I was wondering if someone already has what I need. I have two machines separated by a network connection that only supports UDP. The network is heavily loaded, but I can put a few packets per second on it, perhaps up to 10. I need to reliably copy a file from one machine to the other. I don't have RDP (who does!) and I don't want (or need) to replicate all of TCP. Any suggestions?

jacquesm 17y ago
Tunnel tcp over udp. http://code.google.com/p/udptunnel/ http://code.google.com/p/udptunnel/ http://thebends.org/~allen/utunnel/ http://thebends.org/~allen/utunnel/
brk 17y ago
I would suggest a forward error correction model. Your problem is not unlike satellite radio transmission (essentially a 1-way UDP transmission) and FEC is the method used in Sirius/XM to minimize audio garbling from brief signal loss. Google should yield many results and more info. If the connection is mostly reliable (just slow) you shouldn't have to chew up a lot of wasted packets with error-correction data.
- RiderOfGiraffes 17y ago
  I'm not sure how FEC is going to help, unless I do this. Send lots of packets, bit the first bit of each packet is an error corrected byte of the file to be transmitted. Then if a packet gets lost I simply make up a random one and let the error correction deal with it. If I use 2 bit correction then I can lose two packets and still have uncorrupted transmission. Hmm. I still need to know how many packets I expect, how big the file is, and some sort of checksum to ensure that I haven't error "corrected" to the wrong value anywhere. It seems very complex compared with just a simple re-try of packets that haven't arrived. Advantage is that it's forward only, but that doesn't help much since I do have a two-way connection. I'll think about that though, because although it's non-obvious, I can see that it does potentially have advantages. Not necessarily in this specific case, though.
Rust 17y ago
If I recall, the old "Hotline" protocol is UDP based. There should be a Java implementation of that somewhere, and there is a VB6 one somewhere as well. KDX is also UDP based for file transfers, but I don't know if it's OSS or not.
wmf 17y ago
TFTP was mentioned in the netboot discussion. http://tools.ietf.org/html/rfc1350 http://tools.ietf.org/html/rfc1350
evgen 17y ago
Go back to the old skool... FSP. Back in the days when gopher was the go-to tool for document sharing and network pipes were smaller and less reliable we used this for lightweight anonymous ftp. It was very resilient in the face of random packet loss (a requirement since it only uses udp.)
omellet 17y ago
Use sequence numbers, and issue replay requests for sequence gaps.
retrogradeorbit 17y ago
Is ICMP allowed? I remember an old colleague writing a steganographic file transfer agent that transfered files slowly with ping packets. An ICMP ECHO_REQUEST of 56 bytes was a zero. And an ICMP ECHO_REQUEST of 64 bytes was a one. ICMP ECHO_REPLY was confirmation of receipt! Would that be slow enough? :P
- RiderOfGiraffes 17y ago
  Interesting idea! And it would certainly be slow enough. However, it's not really what I'm looking for ... It looks like I'll implement something similar to FSP - there doesn't seem to be anything like what I need "off-the-shelf" Much appreciation to those who replied - I've learned a lot that may be useful.

jacquesm 17y ago

Tunnel tcp over udp. http://code.google.com/p/udptunnel/ http://code.google.com/p/udptunnel/ http://thebends.org/~allen/utunnel/ http://thebends.org/~allen/utunnel/

brk 17y ago

I would suggest a forward error correction model. Your problem is not unlike satellite radio transmission (essentially a 1-way UDP transmission) and FEC is the method used in Sirius/XM to minimize audio garbling from brief signal loss. Google should yield many results and more info. If the connection is mostly reliable (just slow) you shouldn't have to chew up a lot of wasted packets with error-correction data.

RiderOfGiraffes 17y ago
I'm not sure how FEC is going to help, unless I do this. Send lots of packets, bit the first bit of each packet is an error corrected byte of the file to be transmitted. Then if a packet gets lost I simply make up a random one and let the error correction deal with it. If I use 2 bit correction then I can lose two packets and still have uncorrupted transmission. Hmm. I still need to know how many packets I expect, how big the file is, and some sort of checksum to ensure that I haven't error "corrected" to the wrong value anywhere. It seems very complex compared with just a simple re-try of packets that haven't arrived. Advantage is that it's forward only, but that doesn't help much since I do have a two-way connection. I'll think about that though, because although it's non-obvious, I can see that it does potentially have advantages. Not necessarily in this specific case, though.

Rust 17y ago

If I recall, the old "Hotline" protocol is UDP based. There should be a Java implementation of that somewhere, and there is a VB6 one somewhere as well. KDX is also UDP based for file transfers, but I don't know if it's OSS or not.

wmf 17y ago

TFTP was mentioned in the netboot discussion. http://tools.ietf.org/html/rfc1350 http://tools.ietf.org/html/rfc1350

evgen 17y ago

Go back to the old skool... FSP. Back in the days when gopher was the go-to tool for document sharing and network pipes were smaller and less reliable we used this for lightweight anonymous ftp. It was very resilient in the face of random packet loss (a requirement since it only uses udp.)

omellet 17y ago

Use sequence numbers, and issue replay requests for sequence gaps.

retrogradeorbit 17y ago

Is ICMP allowed? I remember an old colleague writing a steganographic file transfer agent that transfered files slowly with ping packets. An ICMP ECHO_REQUEST of 56 bytes was a zero. And an ICMP ECHO_REQUEST of 64 bytes was a one. ICMP ECHO_REPLY was confirmation of receipt! Would that be slow enough? :P

RiderOfGiraffes 17y ago
Interesting idea! And it would certainly be slow enough. However, it's not really what I'm looking for ... It looks like I'll implement something similar to FSP - there doesn't seem to be anything like what I need "off-the-shelf" Much appreciation to those who replied - I've learned a lot that may be useful.