5 ms·

Are you familiar with UDT (http://udt.sourceforge.net/ http://udt.sourceforge.net/) ? How does it compare to Aspera ? Thanks!

ncmncm 7y ago
I was not familiar with it. It looks like you could build a good-enough file transfer system on top of UDT, but probably not with the rate control protocols they provide. One problem with UDP protocols over the open internet is that you cannot trust UDP packet contents, which could come from anywhere.
- whatshisface 7y ago
  Step one: send a file that's cryptographically signed. Step two: the client solves the problem of "which of these duplicate and conflicting packets do I need to assemble together to make a message that is appropriately signed?"
  - ncmncm 7y ago
    I solved it by cryptographically signing each packet. The signature was checked before decrypting the packet. Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks.
    - light_hue_1 7y ago
      > Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks. Wow. I had no idea. This may be one of the dumbest things I've ever read. Encryption is free.
      - ncmncm 7y ago
        Didn't use to be. VIA/centaur was the first to add AES instructions, later picked up by Intel and AMD. Its not free, but not expensive. A brilliant woman from, where, Slovenia? figured a way to use SSE instructions to encrypt just as fast as the built-in instructions, I think 2 cycles per byte. I guess DJB's ciphers run much the same way, nowadays.
        exikyut 7y ago
        If you ever track down a citation for that I'd love to know more.
        ncmncm 7y ago
        Might be Emilia Kasper, now apparently at Google. Probably it was 2 cycles per bit, not byte. There are pretty good references starting from https://crypto.stanford.edu/vpaes/ https://crypto.stanford.edu/vpaes/ .
        ncmncm 7y ago
        https://news.ycombinator.com/item?id=13364762 https://news.ycombinator.com/item?id=13364762 has interesting discussion.
      - sneak 7y ago
        Bandwidth for signature values in each packet is not. Additionally, as is pointed out down-thread, crypto wasn’t always free either.
        ncmncm 7y ago
        Actually the signature was almost free, because it was also the nonce for the header, which was the nonce for the rest of the packet. IIRC.
    - keredson 7y ago
      not true, although i'm sure there are one-off instances where there's no encryption. but in general there's lots of encryption on each of the different levels of secure government networks.
      - ncmncm 7y ago
        Right, they didn't want us encrypting what they were going to encrypt in their routers anyway. "Trusted" is a technical term. So they started using Aspera to send video from the UAVs over Iraq etc. to the Pentagon. Before that, they were literally flying boxes of videotapes (or CDs? Tapes seem hard to believe, except this was govt.) to a warehouse, where analysts could check them out, like from your neighborhood library. Probably now it's all realtime. But they have to stream it, somehow, so it is likely still going via Aspera to Nevada or wherever they work the remote pilots.
        exikyut 7y ago
        Sounds like this was a (little?) while ago. On first read I thought you meant Aspera ran on the UAVs :) but I now realize the high unlikeliness of that...
        ncmncm 7y ago
        At the time (mid-late 2000s) the video output from the UAVs was sent in the clear, in analog format, and recorded in analog format. So they had to start digitizing it in the field to transfer as files. Jihadists could tune in an ordinary scanner, and watch the video coming from them. Embarrassing.

ncmncm 7y ago

I was not familiar with it. It looks like you could build a good-enough file transfer system on top of UDT, but probably not with the rate control protocols they provide. One problem with UDP protocols over the open internet is that you cannot trust UDP packet contents, which could come from anywhere.

whatshisface 7y ago
Step one: send a file that's cryptographically signed. Step two: the client solves the problem of "which of these duplicate and conflicting packets do I need to assemble together to make a message that is appropriately signed?"
- ncmncm 7y ago
  I solved it by cryptographically signing each packet. The signature was checked before decrypting the packet. Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks.
  - light_hue_1 7y ago
    > Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks. Wow. I had no idea. This may be one of the dumbest things I've ever read. Encryption is free.
    - ncmncm 7y ago
      Didn't use to be. VIA/centaur was the first to add AES instructions, later picked up by Intel and AMD. Its not free, but not expensive. A brilliant woman from, where, Slovenia? figured a way to use SSE instructions to encrypt just as fast as the built-in instructions, I think 2 cycles per byte. I guess DJB's ciphers run much the same way, nowadays.
      - exikyut 7y ago
        If you ever track down a citation for that I'd love to know more.
        ncmncm 7y ago
        Might be Emilia Kasper, now apparently at Google. Probably it was 2 cycles per bit, not byte. There are pretty good references starting from https://crypto.stanford.edu/vpaes/ https://crypto.stanford.edu/vpaes/ .
        ncmncm 7y ago
        https://news.ycombinator.com/item?id=13364762 https://news.ycombinator.com/item?id=13364762 has interesting discussion.
    - sneak 7y ago
      Bandwidth for signature values in each packet is not. Additionally, as is pointed out down-thread, crypto wasn’t always free either.
      - ncmncm 7y ago
        Actually the signature was almost free, because it was also the nonce for the header, which was the nonce for the rest of the packet. IIRC.
  - keredson 7y ago
    not true, although i'm sure there are one-off instances where there's no encryption. but in general there's lots of encryption on each of the different levels of secure government networks.
    - ncmncm 7y ago
      Right, they didn't want us encrypting what they were going to encrypt in their routers anyway. "Trusted" is a technical term. So they started using Aspera to send video from the UAVs over Iraq etc. to the Pentagon. Before that, they were literally flying boxes of videotapes (or CDs? Tapes seem hard to believe, except this was govt.) to a warehouse, where analysts could check them out, like from your neighborhood library. Probably now it's all realtime. But they have to stream it, somehow, so it is likely still going via Aspera to Nevada or wherever they work the remote pilots.
      - exikyut 7y ago
        Sounds like this was a (little?) while ago. On first read I thought you meant Aspera ran on the UAVs :) but I now realize the high unlikeliness of that...
        ncmncm 7y ago
        At the time (mid-late 2000s) the video output from the UAVs was sent in the clear, in analog format, and recorded in analog format. So they had to start digitizing it in the field to transfer as files. Jihadists could tune in an ordinary scanner, and watch the video coming from them. Embarrassing.