7 ms·
Tus - an Open Source File Upload Protocol
- andyking 14y agoCall me a prude, but I saw the F-word and hit the back button.
- nodata 14y agoI didn't notice it until I read your comment.
- Andrenid 14y agoSame here. I think in that context it's fine. I didn't even notice it until I saw this comment here and went back to look.
- pow-tac 14y agoI don't get it...
- vincentkriek 14y agoThe little "about" text on the landing page has the word "fucking" in it.
- gmaslov 14y agoPrude.
- shin_lao 14y agoI got exactly the same reaction.
- felixge 14y agoPoint taken. We'll try to hit a better balance between emotionalizing the subject without being too exclusive when iterating the text for this again. https://github.com/tus/tus.io/issues/14 https://github.com/tus/tus.io/issues/14
- PommeDeTerre 14y agoI don't think it's an issue with "emotionalizing the subject", nor one of people getting offended, but rather one of professionalism. That sort of language, in a context like that, reminds me of a Zed-style rant. It makes it harder for me to take it seriously, you know? The whole project ends up coming off as an amateur effort, even if that may not be the case.
- felixge 14y agoI don't consider professionalism and the word "fuck" to be mutually exclusive, but at the end of the day we'll focus on what attracts people. Our current choice of words clearly fails at this goal, so we'll consider replacing it.
- lttlrck 14y agoIt clearly fails but you'll only consider changing it? That's more worrying than choosing the word in the first place...
- olalonde 14y agoYou mean "fork"?
- scjr 14y agoIs there any reason why you don't use a profanity filter, like this one - https://chrome.google.com/webstore/detail/simple-profanity-filter/ackkocjhcalcpgpfjcoinogdejibgbho?hl=en https://chrome.google.com/webstore/detail/simple-profanity-f... . It is an issue where you object to a certain subset of language, language that is used to express strong or passionate feelings about things. I have never given swearing much thought and I don't understand what makes you object to it. I guess I have always felt that the less people have to sensor themselves them better. The wikipedia article on the subject [http://en.wikipedia.org/wiki/Profanity http://en.wikipedia.org/wiki/Profanity] is not as informative as I'd hoped.
- icebraining 14y agoSeems fine as a best practice for using HTTP for file uploads. I feel the requirement for the server to have fixed URLs for uploading to be limiting, but then again, I'm one of those HATEOAS freaks.
- felixge 14y agoThe urls don't have to be fixed. I'll clarify this in the docs: https://github.com/tus/tus-resumable-upload-protocol/issues/11 https://github.com/tus/tus-resumable-upload-protocol/issues/... Also: The protocol is still under heavy development, so please post any additional ideas, issues, patches or feedback you may have!
- gabipurcaru 14y agoShameless plug, but HTML5 + CORS + S3 can enable resumable file uploads. I've written a library that uploads to S3, and can resume uploads (think internet going down for a while, force-closing the tab, etc.): https://github.com/cinely/mule-uploader https://github.com/cinely/mule-uploader . There's a demo available, I suggest you test it with bigger files (>100MB)
- felixge 14y agoYour library looks great - thanks for releasing it! S3 is an incredible offering, but since I'm working on tus.io, I'll focus on what's wrong with it : ) - Multipart chunks need to be 5 MB at least. An interrupted part cannot be resumed. This kills the mobile use case. - Throughput to S3 is bad from outside of EC2, uploads often start at very slow speeds and won't reach the capacity of the connection in many cases. - S3 does not let you stream/access an upload in progress easily, so you can't start to transcode a video while it's still uploading. - The S3 API is the opposite of RESTful. - S3 is a proprietary service, their protocols are not intendent/documented for adoption, and IMO they don't deserve great people like you making free contributions to their ecosystem. edit: I'm not trying to say S3 isn't a good choice for many people. But our goal is to bring resumable file uploads to every iOS, jQuery, Wordpress, Drupal, Rails, etc. application in the world - S3 is not the right starting point for that.
- gabipurcaru 14y agoThanks for your insight, you make good points; it seems to me you don't really like S3, can you elaborate why?
- felixge 14y agoI love S3, and we use it all over the place at transloadit. I realized my comment sounded overly negative, so I added a clarification to my comment: Our goal is to bring resumable file uploads to the entire planet, S3 or any other proprietary protocol should not be the base for that.
- nanoman 14y agoI've been a user of their product https://transloadit.com/ https://transloadit.com/ for about a year now for video encoding and am very happy with it. The API is elegant, the whole service is reliable and fast. These guys know what they're doing.
- felixge 14y agoThanks for the kind words! Tus.io the beginning of a larger commitment to open source here at Transloadit, which will also translate into many new features (resumable uploads being the next one).
- karterk 14y agoFelix, given that the rest of your stack is on Node, what made you choose Go?
- felixge 14y agoNode.js was a good choice for Transloadit when we started (2009). By good I mean better than PHP which we tried using for transloadit before. I started playing with Go late last year, so far I'm under the impression that it's easier to write reliable software with it than node.js. Callbacks and exception handling are a huge PITA in node.js, and the community has chosen to refuse improvements that would help with some of the issues (promises). Go is also an incredible joy to work with given the modern nature of the standard library, static typing, gofmt, built-in testing, and many other small things that the Go team has done right. That being said, tus.io is not a Go project. Our first server implementation (tusd) is written in Go, but we're working on support for other plattforms like node.js as well. Generally speaking node.js will continue to be part of our toolkit at Transloadit (for the quick & dirty), but I suspect that we'll use Go for the more criticial parts we work on going forward.
- lucaspiller 14y agoThis is a pretty cool idea. I absolutely detest how much apps don't properly handle network degradation correctly, so anything resumable is great in my opinion. I have a question though... regarding the resuming you do the HEAD request to see what has been uploaded: > HTTP/1.1 200 Ok > Content-Length: 100 > Content-Type: image/jpg > Content-Disposition: attachment; filename="cat.jpg"' > Range: bytes=0-69 Is it possible that the data that is already there could be corrupt? I'm also wondering how things like proxies deal with this. A lot of mobile networks have nasty transparent caching proxies in their network. Also when uploading a file through Nginx (when the upload works correctly) it won't send anything to the backend until it has the complete data, is this the same if the connection cut half way through?
- lgierth 14y agoThe server could send a Content-MD5 header, which the client could compare to the md5sum of its own bytes 0 to 69.
- iso-8859-1 14y agoWhy use hashing when you only need checksumming?
- madisp 14y agoTCP means that it shouldn't be corrupt. I would like the feature of having MD5 hash of the data sent so far though, with this the clients would be able to validate that the data up to N bytes is most likely not corrupt.
- j4_james 14y agoI think some of your responses aren't quite right. For example, in response to the first PUT, you have: HTTP/1.1 200 Ok Range: bytes=0-99 Content-Length: 0 But the Range header surely can't be used here, since it's a request header and this is a response. A Content-Range header wouldn't be any more appropriate, since you're not actually returning any content (of any amount). Do you really need this info in the response anyway? The sender knows what they sent, and either it was entirely successful (a 2xx response) or it wasn't. Also, if you're going to return a zero-length 200 response, you might as well use 204 No Content instead. Then, when resuming an upload, you send a HEAD that returns the following: HTTP/1.1 200 Ok Content-Length: 100 Content-Type: image/jpg Content-Disposition: attachment; filename="cat.jpg"' Range: bytes=0-69 Again, you can't use the Range request header in a response. And the Content-Length should surely be 70, since that's how much content would be returned if this was a GET request. You could possibly include a Content-Range of 0-69/100 if the server wanted to communicate the expected file size, but I'm not convinced that's necessary and seems something of an abuse of that header. Finally, the response to the resumed PUT has the same problems as the first PUT response. It should probably just be a 204 No Content response - no Content-Length or Range headers required.
- felixge 14y ago> But the Range header surely can't be used here, since it's a request header and this is a response. We're currently discussing how to interpret RFC 2616 (http 1.1) for this here: https://github.com/tus/tus-resumable-upload-protocol/issues/2 https://github.com/tus/tus-resumable-upload-protocol/issues/... If you have a better suggestions than using the Range header that will still allow clients to send multiple file chunks in parallel, I'd be very interested in it! > Do you really need this info in the response anyway? The sender knows what they sent, and either it was entirely successful (a 2xx response) or it wasn't. We don't need it for the PUT request, but we do need it for HEAD. Adding it to PUT is redundant, but simplifies the logic for clients who choose to upload multiple chunks in parallel. > Also, if you're going to return a zero-length 200 response, you might as well use 204 No Content instead. Good point, I'll investigate on that: https://github.com/tus/tus-resumable-upload-protocol/issues/12 https://github.com/tus/tus-resumable-upload-protocol/issues/... > And the Content-Length should surely be 70, since that's how much content would be returned if this was a GET request. It's 100. We haven't specified GET requests yet, but a server could stream an upload in this case until all bytes have been received. Anyway, this is awesome feedback - thank you so much!
- nnnnni 14y agoAs usual, relevant xkcd: http://xkcd.com/927/ http://xkcd.com/927/
- felixge 14y ago: ) - hopefully that's not the case here, afaik our protocol is the first proposal in this space meant for general adoption. All prior art is tied to specific libraries and services.
- nnnnni 14y agoI hope so too =-) It looked nice from my quick skim, it just brought that classic xkcd to mind! If this can be put into any page/service, it'd be a huge contribution.
- raimue 14y agoIt seems like the handling of concurrent access has been neglected in this protocol. What if multiple clients try to resume uploading the same file?
- felixge 14y agoSo far we have this: > Servers MUST handle overlapping PUT requests in an idempotent fashion given that the overlapping data is identical. Otherwise the behavior is undefined. Does that adress your concern?
- shabble 14y agoSome of hte ideas from http://www.w3.org/1999/04/Editing/ http://www.w3.org/1999/04/Editing/ might come in handy there. What realistic usage scenarios are there for multiple clients concurrently uploading (parts of) the same file?
- tsuraan 14y agoI'm confused on what information the HEAD request gives after a chunk has failed. Suppose a client concurrently uploads chunks 1, 2, 3, 4 and 5; chunks 2 and 4 fail, and the rest work. What information does the HEAD give to tell the client that it needs to re-send the data that was in chunks 2 and 4? Wouldn't it make more sense for the client to store the success of each of its chunk uploads? I'm also not seeing how the client indicates that the upload is complete. It could be done server-side, by just detecting when a file has no more holes in it, but that seems hacky. Holes can also be useful; suppose I make a 32GB .vmdk file (non-sparse) and put 2GB of data on it. If the server can support holes, then I can upload (and the server only has to store) about 2GB of data; if the server can't support holes, then I'll have to upload a bit more data (assuming compression), and the server will have to store a lot more data. If there were some final message the client could submit to the resource saying "I'm done, commit it!", I think the protocol would be a bit more complete.
- nateabele 14y agoDisclaimer: I'm a friend of the main author and have been peripherally involved (mostly watching from the sidelines) in this project. Valid points, IMO. Those sound like use cases that might not have been contemplated originally (the idea for the spec grew out of the author's work here: https://transloadit.com https://transloadit.com). That said, the spec, and the code around it are both still very much evolving, and are welcoming of input. You can join us on GitHub or in #tusio on Freenode.
- j4_james 14y agoLets say the whole file is 1000 bytes and each of the five chunks is 200 bytes. If 2 and 4 have failed, then the HEAD would return with a Range header like this: Range: bytes=0-199,400-599,800-999 The client would then know it had to resend bytes 200-399 and 600-799 (namely parts 2 and 4). If the chunks only partially failed (say 100 bytes of each chunk was received), the HEAD might even return something like this: Range: bytes=0-299,400-699,800-999 So now the client knows it only has to resend bytes 300-399 and 700-799 (only the last 100 bytes of chunks 2 and 4). Technically the Range header isn't valid in an HTTP response (something they are aware of), but conceptually I think the idea works fairly well.