3 ms·
> Is it possible that the data that is already there could be corrupt? TCP error correction should take care of this for most parts, but we're actively discuss
by felixge 14y ago
> Is it possible that the data that is already there could be corrupt?
TCP error correction should take care of this for most parts, but we're actively discussing adding support for stronger guarantees: https://github.com/tus/tus-resumable-upload-protocol/issues/7 https://github.com/tus/tus-resumable-upload-protocol/issues/...
> I'm also wondering how things like proxies deal with this. A lot of mobile networks have nasty transparent caching proxies in their network.
That's a good question. A http proxy could always cause issues, but most proxies should leave POST/PUT/HEAD requests untouched. That being said, we won't freeze the protocol until we had a chance to try it against a variety of mobile networks, which is why we're already starting to implement an initial iOS client over here: https://github.com/tus/tus-ios-client https://github.com/tus/tus-ios-client (not ready yet, but keep an eye on it).
> Also when uploading a file through Nginx (when the upload works correctly) it won't send anything to the backend until it has the complete data, is this the same if the connection cut half way through?
NGINX is somewhat unsuitable as a proxy for file uploads due to the buffering you mentioned. Ideally people will implement the tus protocol as an NGINX Addon like this one: http://www.grid.net.ru/nginx/resumable_uploads.en.html http://www.grid.net.ru/nginx/resumable_uploads.en.html
Meanwhile clients are free to choose a small chunk size for individual PUT requests (e.g. 1 MB), which will allow them to still have resumability (in 1 MB intervals) without changing their architecture.
Last but not least, we'll implement the tus protocol for our commercial uploading service, transloadit.com.
So I'm reasonably optimistic that NGINX won't be a major hurdle for the adoption of the protocol.
- lucaspiller 14y agoThanks for the quick response. Everything you say makes sense. Best of luck with this, I look forward to seeing where it goes!
- rogerbinns 14y ago> TCP error correction should take care of this for most parts ... Oh no it doesn't! We have an analytics service receiving HTTP posts from browsers all over the world as JSON. There is an astonishing amount of single bit errors going on. Usually the initial 20 bytes are okay, but after that we see all sorts of patterns including a bit flip every 8 bytes or so. Note that these will have been received at Google's appengine servers with the correct checksum. I believe that much of the cause is intermediary devices (eg performing NAT or routers) that are responsible for the corruption and recalculate the checksum putting a good checksum on what is now corrupted data. For that service we have to use HTTP (grumble grumble IE grumble). For our regular stuff we use HTTPS where we do still see the problem but it is considerably rarer. In that case the cause is most likely the client device having problems (eg RAM bit flips, cosmic rays, overclocked/overheated CPUs etc) All else being equal I'd recommend you add a layer of checksums as a helpful sanity check. Using SSL also does that for you, but it sees the data late.