7 ms·
How to directly upload files to Amazon S3 from your client side web app
- brettcvz 14y agoAlternatively, use Filepicker.io. filepicker.store(myFile, {location: "S3"}, function(FPFile){console.log("File stored in S3 at "+FPFile.key+" and available at "+FPFile.url);});
- tadruj 14y agofilepicker.io is a little bit more heavy-weight on the browser. But I really like it. It's THE start-up of 2012 for me. This post was intended for the people who want to have a specific and stand-alone building block just for S3.
- philfreo 14y agounfortunately they don't return the full url to the file on YOUR s3 bucket (but you can piece it together)
- brettcvz 14y agoOur concern is that if we return the direct S3 url, people less experienced will assume that the S3 url should work by default, when it will 403. So we return Filepicker.io urls which alias to your S3 bucket
- philfreo 14y agothen you should have an example IAM Policy in a very easily accessible location overall as someone who is currently using filepicker, i would recommend people avoid it if all you want to do is upload to files to your own bucket without any transformations like image resizing
- matb33 14y agoYou could leverage zencoder to generate your thumbnails. Send the job as a test so it's free (limited to 5 seconds but that's ok since we only want thumbnails) and specify your thumbnail settings as desired. You could even set the clip length to one frame so the video encoding step is nearly skipped. Since zencoder runs on AWS, you should get near-LAN transfers between S3 and zencoder
- templaedhel 14y agoI came here to say this exactly, and was a little disappointed to see the mixed bag of filepicker impressions. Filepicker is awesome. At my job I just had to implement direct to s3 uploads, and filepicker cut down the work by 10x. It's basically a one line implementation to save the contents of a file input to s3. That's awesome. The pricing is extremely liberal, and the docs are good. I haven't used many of the more advanced features, but even at its simplest form it's an excellent tool. As for the complaints here, they seem to be mostly pedantic - I would much rather concat a few strings to build the proper s3 URL when the alternative is building the whole upload infrastructure, and once again, that's without considering all the advanced features filepicker gives you.
- JuDue 14y agoFilepicker is a great idea, especially if you want to alter the image size or make thumbnails without the hassle of setting up the processor. Just with their image manipulations were a bit more fully featured, like imagemagic
- modarts 14y agoTangentially related, but does anyone know of an easy way to push a github repo to an S3 bucket (configured as a public site)? I'm pretty new at git/github, but understand that the solution may involve a post commit hook that somehow calls an S3 file upload service.
- kmfrk 14y agoOne way is to use something like vaporfile (https://github.com/EnigmaCurry/Vaporfile https://github.com/EnigmaCurry/Vaporfile) to create a post commit hook that lets vaporfile push the new files to S3. It doesn't do delta updates, which could be a problem depending on the kind of project, though.
- gdubs 14y agoI was actually reading up on this earlier; I was led to jgit http://stackoverflow.com/questions/7031729/publish-to-s3-using-git http://stackoverflow.com/questions/7031729/publish-to-s3-usi...
- modarts 14y agoSorry, I should have clarified a little better. I'd like to avoid installing anything on a server to make this happen. (My development workflow/environment is completely decoupled from any servers I control: I use Parse.com as my backend with static HTML pages served from S3, and cloud9 as my IDE, with Github as my source host)
- vinhboy 14y agoLast week I was trying to figure out how to do this with dropbox. Anyone know a way?
- tadruj 14y agoDropbox has a filepicker https://www.dropbox.com/developers/chooser https://www.dropbox.com/developers/chooser , you can also use filepicker.io for more services support. But Dropbox doesn't support per-request CORS as I read on their Forums.
- vinhboy 14y agoG@#$D!T I wished I had known about this last week. I already implemented my own file browser! Thanks for the tip though.
- mjtokelly 14y agoDirect uploading like this has been possible for at least two years. However, in our tests it turned out that Amazon was silently losing some files >100MB (and almost all files ~1GB). On the AWS forums it was clear that AWS was aware of this consistent problem, but was not able to fix it, and not willing to document it. They eventually released "multipart upload", which remains the only reliable way to get large files to S3. Unfortunately, multipart upload is nearly impossible to implement as a web app (short of resorting to e.g. Java).
- kzahel 14y agoMultipart is not at all impossible from a webapp (depending on your browser requirements). FileReader API is available across all browsers (except IE < version 10, which I admit is a big problem). For IE, you can resort to some other solution (perhaps Silverlight, or Flash?)
- carson 14y agoI will second this. At the bottom of this article there is a reference the article I wrote a while ago when Amazon released this feature. I was planing on doing multipart uploading at that time using the FileReader but there was a bug in the way S3 did CORS so I didn't want to continue until that was fixed. They fixed it and I never came back to it. Maybe be a good time to try it again. Resuming a partial upload seemed like a good win to me.
- tadruj 14y agoOur video bucket is a few TB in size right now, file-sizes ranging from 10MB to 1GB. Not one file/upload broken until now. So I guess they fixed it.
- akoumjian 14y agoDirect uploading like this has not been possible through XHR as CORS support was only added this past year. However, you could mitigate it by handling the CORS headers through a proxy, using flash, or a couple other methods. You could, however, perform a traditional POST.
- 14y ago
- philfreo 14y agoThanks for posting... I was looking for something just like this: https://github.com/tadruj/s3upload-coffee-javascript https://github.com/tadruj/s3upload-coffee-javascript
- tlrobinson 14y agoI really like CoffeeScript, but do we really need to advertise every JavaScript library as a CoffeeScript library as well? Feel free to use CoffeeScript to implement your libraries, but lets just call it a JavaScript library and assume everyone who uses CoffeeScript will understand they can use any JavaScript library.
- tadruj 14y agoI think we need to. It deserves a credit, because we use it and we didn't pay for it. CoffeeScript makes our lives easier, almost eliminates bad practices and removes brace clutter from the screen, making the code more readable, understandable, better. When having to choose between a library written in JavaScript and a library written in CoffeeScript I choose the latter, because it statistically has a probability to be a better library, just because the coder didn't have a chance to use bad JavaScript practices, when writing it. Assuming I usually don't have time to read the code of the libraries I'm using.
- pud 14y agoAny recommendations for how to show a progress-indicator to the user during a direct-to-S3 upload?
- carson 14y agoI wrote the example referenced at the bottom of the article and it has an example of indicating progress to the user. You can see it in app.js here: https://github.com/carsonmcdonald/direct-browser-s3-upload-example https://github.com/carsonmcdonald/direct-browser-s3-upload-e...
- pswenson 14y agoCoolest would be an ability to have 1) a great filepicker UI (drag target + multi file upload w/ progress bar) 2) resizing/thumbnails in the browser 3) no server side other than s3 guessing #2 isn't possible?
- carson 14y ago#2 is possible but #3 isn't unless you have a completely open bucket and that wouldn't be good.
- brettcvz 14y agoI hate to be so self-promoting, but http://jsfiddle.net/GTYX5/2/ http://jsfiddle.net/GTYX5/2/ should do everything you're asking for
- pswenson 14y agofilepicker.io is cool. But your backend service wouldn't be needed if the image uploading & resizing straight to s3 could be done in the browser. Your product would simply be a javascript lib.
- pswenson 14y agohow do you do #2?
- ayushgta 14y agoThanks for posting this. I was looking at adding uploads just this morning, so this is very timely. It looks like if you combine the direct upload to S3 as outlined in this article with bitline (http://www.blitline.com/ http://www.blitline.com/), you get some pretty cool abilities. I especially like that bitline supports batch operation (for example create five sizes of my image) in one call.
- secabeen 14y agoHere's the other page on the AWS documentation on this: http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentic...
- dgoodlad 14y agoI did a demo similar to this a few months ago: https://github.com/dgoodlad/s3-photo-upload-demo https://github.com/dgoodlad/s3-photo-upload-demo http://s3-photo-upload-demo.throwawayapp.com http://s3-photo-upload-demo.throwawayapp.com