3 ms·

This is a pretty cool idea. I absolutely detest how much apps don't properly handle network degradation correctly, so anything resumable is great in my opinion.

This is a pretty cool idea. I absolutely detest how much apps don't properly handle network degradation correctly, so anything resumable is great in my opinion. I have a question though... regarding the resuming you do the HEAD request to see what has been uploaded: > HTTP/1.1 200 Ok > Content-Length: 100 > Content-Type: image/jpg > Content-Disposition: attachment; filename="cat.jpg"' > Range: bytes=0-69 Is it possible that the data that is already there could be corrupt? I'm also wondering how things like proxies deal with this. A lot of mobile networks have nasty transparent caching proxies in their network. Also when uploading a file through Nginx (when the upload works correctly) it won't send anything to the backend until it has the complete data, is this the same if the connection cut half way through?

lgierth 14y ago
The server could send a Content-MD5 header, which the client could compare to the md5sum of its own bytes 0 to 69.
- iso-8859-1 14y ago
  Why use hashing when you only need checksumming?
  - felixge 14y ago
    Depending on the use case, man in the middle attacks could be a good reason for hashing.
    - Filligree 14y ago
      A man in the middle could just as easily change the hash header, and MD5 isn't cryptographically secure anyway. It is still a very good checksum.
madisp 14y ago
TCP means that it shouldn't be corrupt. I would like the feature of having MD5 hash of the data sent so far though, with this the clients would be able to validate that the data up to N bytes is most likely not corrupt.
felixge 14y ago
> Is it possible that the data that is already there could be corrupt? TCP error correction should take care of this for most parts, but we're actively discussing adding support for stronger guarantees: https://github.com/tus/tus-resumable-upload-protocol/issues/7 https://github.com/tus/tus-resumable-upload-protocol/issues/... > I'm also wondering how things like proxies deal with this. A lot of mobile networks have nasty transparent caching proxies in their network. That's a good question. A http proxy could always cause issues, but most proxies should leave POST/PUT/HEAD requests untouched. That being said, we won't freeze the protocol until we had a chance to try it against a variety of mobile networks, which is why we're already starting to implement an initial iOS client over here: https://github.com/tus/tus-ios-client https://github.com/tus/tus-ios-client (not ready yet, but keep an eye on it). > Also when uploading a file through Nginx (when the upload works correctly) it won't send anything to the backend until it has the complete data, is this the same if the connection cut half way through? NGINX is somewhat unsuitable as a proxy for file uploads due to the buffering you mentioned. Ideally people will implement the tus protocol as an NGINX Addon like this one: http://www.grid.net.ru/nginx/resumable_uploads.en.html http://www.grid.net.ru/nginx/resumable_uploads.en.html Meanwhile clients are free to choose a small chunk size for individual PUT requests (e.g. 1 MB), which will allow them to still have resumability (in 1 MB intervals) without changing their architecture. Last but not least, we'll implement the tus protocol for our commercial uploading service, transloadit.com. So I'm reasonably optimistic that NGINX won't be a major hurdle for the adoption of the protocol.
- lucaspiller 14y ago
  Thanks for the quick response. Everything you say makes sense. Best of luck with this, I look forward to seeing where it goes!
- rogerbinns 14y ago
  > TCP error correction should take care of this for most parts ... Oh no it doesn't! We have an analytics service receiving HTTP posts from browsers all over the world as JSON. There is an astonishing amount of single bit errors going on. Usually the initial 20 bytes are okay, but after that we see all sorts of patterns including a bit flip every 8 bytes or so. Note that these will have been received at Google's appengine servers with the correct checksum. I believe that much of the cause is intermediary devices (eg performing NAT or routers) that are responsible for the corruption and recalculate the checksum putting a good checksum on what is now corrupted data. For that service we have to use HTTP (grumble grumble IE grumble). For our regular stuff we use HTTPS where we do still see the problem but it is considerably rarer. In that case the cause is most likely the client device having problems (eg RAM bit flips, cosmic rays, overclocked/overheated CPUs etc) All else being equal I'd recommend you add a layer of checksums as a helpful sanity check. Using SSL also does that for you, but it sees the data late.

lgierth 14y ago

The server could send a Content-MD5 header, which the client could compare to the md5sum of its own bytes 0 to 69.

iso-8859-1 14y ago
Why use hashing when you only need checksumming?
- felixge 14y ago
  Depending on the use case, man in the middle attacks could be a good reason for hashing.
  - Filligree 14y ago
    A man in the middle could just as easily change the hash header, and MD5 isn't cryptographically secure anyway. It is still a very good checksum.

madisp 14y ago

TCP means that it shouldn't be corrupt. I would like the feature of having MD5 hash of the data sent so far though, with this the clients would be able to validate that the data up to N bytes is most likely not corrupt.

felixge 14y ago

> Is it possible that the data that is already there could be corrupt? TCP error correction should take care of this for most parts, but we're actively discussing adding support for stronger guarantees: https://github.com/tus/tus-resumable-upload-protocol/issues/7 https://github.com/tus/tus-resumable-upload-protocol/issues/... > I'm also wondering how things like proxies deal with this. A lot of mobile networks have nasty transparent caching proxies in their network. That's a good question. A http proxy could always cause issues, but most proxies should leave POST/PUT/HEAD requests untouched. That being said, we won't freeze the protocol until we had a chance to try it against a variety of mobile networks, which is why we're already starting to implement an initial iOS client over here: https://github.com/tus/tus-ios-client https://github.com/tus/tus-ios-client (not ready yet, but keep an eye on it). > Also when uploading a file through Nginx (when the upload works correctly) it won't send anything to the backend until it has the complete data, is this the same if the connection cut half way through? NGINX is somewhat unsuitable as a proxy for file uploads due to the buffering you mentioned. Ideally people will implement the tus protocol as an NGINX Addon like this one: http://www.grid.net.ru/nginx/resumable_uploads.en.html http://www.grid.net.ru/nginx/resumable_uploads.en.html Meanwhile clients are free to choose a small chunk size for individual PUT requests (e.g. 1 MB), which will allow them to still have resumability (in 1 MB intervals) without changing their architecture. Last but not least, we'll implement the tus protocol for our commercial uploading service, transloadit.com. So I'm reasonably optimistic that NGINX won't be a major hurdle for the adoption of the protocol.

lucaspiller 14y ago
Thanks for the quick response. Everything you say makes sense. Best of luck with this, I look forward to seeing where it goes!
rogerbinns 14y ago
> TCP error correction should take care of this for most parts ... Oh no it doesn't! We have an analytics service receiving HTTP posts from browsers all over the world as JSON. There is an astonishing amount of single bit errors going on. Usually the initial 20 bytes are okay, but after that we see all sorts of patterns including a bit flip every 8 bytes or so. Note that these will have been received at Google's appengine servers with the correct checksum. I believe that much of the cause is intermediary devices (eg performing NAT or routers) that are responsible for the corruption and recalculate the checksum putting a good checksum on what is now corrupted data. For that service we have to use HTTP (grumble grumble IE grumble). For our regular stuff we use HTTPS where we do still see the problem but it is considerably rarer. In that case the cause is most likely the client device having problems (eg RAM bit flips, cosmic rays, overclocked/overheated CPUs etc) All else being equal I'd recommend you add a layer of checksums as a helpful sanity check. Using SSL also does that for you, but it sees the data late.