15 ms·

Ask HN: How to handle user file uploads?

214 points by matusgallik008 2y ago

hey, i work as an SRE for a company where we allow users to upload media files (e.g. profile picture, attach docs or videos to tasks..the usual). We currently just take a S3 pre-signed URL and let the user upload stuff. Occasionally, limits are set on the <input/> element for file types. I don't feel this is safe enough. I also feel we could do better by optimizing images on the BE, or creating thumbnails for videos. But then there is the question of cost on AWS side. Anybody have experience with any of this? I imagine having a big team and dedicated services for media processing could work, but what about small teams? All thoughts/discussions are welcome.

p2hari 2y ago
Look at https://uppy.io/ https://uppy.io/ open source and lot of integrations. You can keep moving to different levels of abstraction as required and see some good practices of how things are done.
junto 2y ago
You should quarantine them until you’ve analyzed them. Like you stated, an async process using a function would suffice. Previously used ClamAV for this in a private cloud solution, I’ve also used the built in anti-virus support on Azure Blob Storage if you don’t mind multi-cloud, plus an Azure Function has the ability to support blob triggers, which is a nice feature. The file types scan is relatively simple. You just need a list of known “magic string” header values to do a comparison, and for that you only need a max of 40 bytes of the beginning of the file to do the check (from memory). Depending on your stack, there are usually some libraries already available to perform the matching. And it goes without saying, but never trust the client, and always generate your own filenames. https://en.m.wikipedia.org/wiki/List_of_file_signatures https://en.m.wikipedia.org/wiki/List_of_file_signatures
- dividuum 2y ago
  I'm pretty sure ClamAv would add more vulnerabilities to your stack than it might prevent from being exploited.
  - junto 2y ago
    It’s an old stack. Can you suggest an alternative self hosted option?
    - dividuum 2y ago
      I consider the approach of using virus scanning inherently flawed: They rely on heuristics and rules to essentially create a blacklist. And they add a ton of complexity (more code -> usually more bugs) which is not something you want if you work with untrustworthy data. What you instead want is a whitelist: Only allow properly formatted images and videos and ruthlessly reject anything else. I wrote about how I implemented this for my service in another response.
stevoski 2y ago
My SaaS uses Cloudinary for uploading and storing images. It’s not particularly cheap. But it is fast and flexible and safe.
- sairamkunala 2y ago
  My previous company used Cloudinary for the purpose and amazingly lot of time was saved.
arcza 2y ago
Kinda incredible how nearly every comment here mentions S3. Cloud storage is not the only backend in existence :)
- Dowwie 2y ago
  This is your moment. Let's hear about alternatives.
- hnlmorg 2y ago
  That’s because the requester is already using AWS and S3 specifically. One of the most important, yet oft ignored on HN, principles of architecting solutions is having an unbiased view of products and then choosing what will fit within an existing organisation’s architecture. Given the specifications shared, utilising S3 further is the correct advice.
- morbicer 2y ago
  It gives you at least some basic security. Nothing can get executed on S3, there is no lateral move. If you are using your own servers for user uploads you have to do a damn good job of quarantining them.
ytch 2y ago
https://github.com/google/magika https://github.com/google/magika Magika is a novel AI powered file type detection tool that relies on the recent advance of deep learning to provide accurate detection. It's not a sliver bullet, but I use it recently for inspecting file type instead of magic file. One advantage is that detecting composed files. Take pdf+exe file for example, the library will report something like 70% pdf and 30% pebin file.
- mceachen 2y ago
  I don't think this project was actually meant for production use, and especially not under decidedly hostile conditions. I'd suggest existing tooling like `exiftool` to do mimetype detection and metadata stripping.
PaywallBuster 2y ago
2 buckets: - upload bucket - processed bucket upload bucket has an event triggered on new file upload which triggers a lambda, the lambda will re-encode and do wtv you deem fit and upload to new bucket your app will use the processed bucket
sim7c00 2y ago
as poijted out already magika is useful for good file type analysis. also, scan thoroughly for viruses, potebtially using multiple engines. now this can be tricky depending on the confidentiality of the uploaded files so take good care not to submit them to sandboxes, av platforms etc. if not allowed. i would really recommend it though. if you want to get into the nitty grit of filetype abuse, to learn how to possibly detect that good. ange albertini's work on polymorphic filetypes and the ezine poc||gtfo as well as lots of articles by AV code devs are available. its really hard problem and also depends a lot on what program is interpreting the files submitted. if its some custom tool.there might even be unique vectors to take into account. fuzzing and penteting the upload form and any tools handing these files can shed light on those issues potentially. (edit: fat fingers)
marcpaq 2y ago
Have you considered a commercial solution? https://developer.massive.io/js-uploader/ https://developer.massive.io/js-uploader/ Just point it to your local files, as big as you want, and it does the rest. It handles the complexities of S3 juggling and browser constraints. Your users pay nothing to upload, you pay for egress. Full disclosure: I'm MASV's developer advocate.
fy20 2y ago
Lots of other comments give good suggestions on how to handle uploading and processing, but nothing mentions serving resulting content, so let me chime in: Do not serve content from S3 directly. ISPs often deprioritize traffic from S3, so downloading assets can be very slow. I've seen kbytes/s on a connection that Speedtest.net says has a download speed of 850 mbit. Putting Cloudfront in front of S3 solves that.
- throwaway984393 2y ago
  Not to mention using Cloudfront in front of S3 can lower cost, due to caching.
- maayank 2y ago
  > ISPs often deprioritize traffic from S3 I wonder why
- bagels 2y ago
  Cloudfront does not solve it. Comcast deprioritizes any content that comes from Amazon.
- 65 2y ago
  Also a good idea to stream large files instead of trying to directly serve the file as-is. For video you essentially have to stream the file in order for it to load on Safari.
- sakopov 2y ago
  You should never be serving content from S3 directly anyway. Always setup a cloudfront distribution for your buckets and allow origin access identity (OAI) access to the bucket.
grishka 2y ago
In my fediverse server project[1], I convert all user-uploaded images to high-quality webp and store them like that. I discard the original files after that's done. I use imgproxy[2] to further resize and convert them on the fly for actual display. In general, I try my best to treat the original user-uploaded files like they're radioactive, getting rid of them as soon as possible. I don't do videos yet, but I'm kinda terrified of the idea of putting user-uploaded files through ffmpeg if/when I'll support them. [1] https://github.com/grishka/Smithereen https://github.com/grishka/Smithereen [2] https://github.com/imgproxy/imgproxy https://github.com/imgproxy/imgproxy
JohnCClarke 2y ago
What do you do with the uploaded images? You could be exposed to risks that may not be immediately obvious. I have seen a team struggle for over a month to eliminate NSFW content - avatars - uploaded by a user that lead to their site being demonetised.
- PaulHoule 2y ago
  I had a site demonetized because about 1 in 10,000 pictures were of deformed penises, dead nazis, things like that.
- samus 2y ago
  I think these days you'd have to deploy an NSFW detector model. And you'd have to silently reject the upload* and limit profile picture change rate, else the attacker could use the endpoint as an oracle to build an adversarial model, which can eventually generate inputs that the detector won't flag. *: act as if successful, including processing time, but throw away the upload
santiagobasulto 2y ago
It’s amazing that this was an issue in 2004 and it’s still an issue today. I don’t have much to add aside from what was already said. There are services like uppy, transload it, etc that simplify this, but might be more expensive than S3+CF.
dividuum 2y ago
I recently redesigned my stack of validating uploaded files and creating thumbnails from them. My approach is to have different binaries per file type (currently images JPEG/PNG, videos H264/265 and truetype fonts). Each of them is implemented as in a way that they receive the raw data stream via stdin and then either generate an error message or a raw planar RGBA data stream via stdout. The validation and thumbnail process is triggered after first locking in the process in a seccomp strict mode jail before touching any of the untrustworthy data. Seccomp prevents them from basically every syscall except read/write. Even if there would be an exploit in the format parser, it would very likely not get anywhere as there’s literally nothing it could do except write to stdout. Outside a strict time limit is enforced. The raw RGBA output is then received and converted back into PNG or similar. It was a bit tricky to get everything working without additional allocation and using syscalls triggered by glibc somewhere, but works pretty well now and is fast enough for my use case (around 20ms/item).
- artwr 2y ago
  Oh could you expand briefly on what the stack looks like to accomplish this? Or do you have a write up on a blog/site you could share?
  - dividuum 2y ago
    I wrote a little bit more about this here: https://community.info-beamer.com/t/an-updated-approach-to-content-sandboxing/1277 https://community.info-beamer.com/t/an-updated-approach-to-c... I’m a huge fan of building minimal self-contained tools, so all of the C programs statically link in the required parser libs (libavcodec/wuffs/freetype) so the resulting binaries don’t require additional dependencies on the target machine. The python wrapping code is rather straightforward as well and is only like 300 lines of code.
TacticalCoder 2y ago
> I don't feel this is safe enough. I also feel we could do better by optimizing images on the BE, or creating thumbnails for videos. Yeah definitely. Even optimizing the vids. I just spend time writing scripts to convert, in parallel, a massive amount of JPG, PNG, PDFs, mp4 videos and even some HEIC files customers sent of their ID (identity card or passport, basically). I did shrink them all to reasonable size. The issue is: if you let user do anything, you'll have that one user, once in a while, that shall send a 30 MB JPG of his ID. Recto. Than Verso. Then the signed contracts: imagine a user printing a 15 pages contract, signing/paraphing every single page, then not scanning it but taking a 30 MB picture, with his phone, in diagonal, in perspective. And sending all the files individually. After a decade, this represented a massive amount of data. It was beautiful to crush that data to anywhere from 1/4th to 1/10th of its size and see all the cores working at full speed, compressing everything to reasonable sizes. Many sites and 3rd party identity verification services (whatever these are called) do put limit on the allowed size per document, which already helps. In my case I simply used ImageMagick (mogrify), ffmpeg (to convert to x265) and... GhostScript (good old gs command). PDFs didn't have to be searchable for text so there's that too (and often already weren't at least not easily, due to users taking pictures then creating a PDF out of the picture). This was not in Amazon S3 but basically all in Google Workspace: it was for a SME to make everything leaner, snapper, quicker, smaller. Cheaper too (no need to buy additional storage). Backups of all the originals, full size, files were of course made too but these shall probably never be needed. In my case I downloaded everything. Both to create backups (offsite, offline) and to crush everything locally (simply on an AMD 7700X: powerful enough as long as you don't have months of videos to encode). > Anybody have experience with any of this? I imagine having a big team and dedicated services for media processing could work, but what about small teams? I did it as a one-person job. Putting limits in place or automatically resizing, right after upload, a 30 MB JPG file which you know if of an ID card to a 3 MB JPG file doesn't require a team. Same for invoking the following to downsize vids: ffmpeg -i input.mp4 -vcodec libx265 -crf 28 output.mp4 (I think that's what I'm using) My script's logic were quite simple: files above a certain size were candidates for downsizing then downsizing then if the output was successful and took less than a certain amount of time, use that, otherwise keep the original. I didn't bother verifying that the files visually matched (once again: all the originals are available on offline backups in case something went south and some file is really badly needed) but I could have done that too. There was a blog post posted here a few years ago where a book author would visually compare thumbnails of different revisions of his book, to make sure that nothing changed too much between two minor revisions. I considered doing something similar but didn't bother. Needless to say my client is very happy with the results and the savings. YMMV but worked for me and worked for my client.
mattpavelle 2y ago
If you're looking for a good image optimization product, I've had excellent results from ImageOptim (I have no affiliation with them). They have a free Mac App, they have an API service, but also they kindly link to lots of other free similar products and services: https://imageoptim.com/versions.html https://imageoptim.com/versions.html If you can spare the CPU cycles (depending on how you optimize, they can actually be expensive) and if your images will be downloaded frequently, your users will thank you.
4RealFreedom 2y ago
Read through the comments and was surprised no one mentioned libvips - https://github.com/libvips/libvips https://github.com/libvips/libvips. At my current small company we were trying to allow image uploads and started with imagemagick but certain images took too long to process and we were looking for faster alternatives. It's a great tool with minimum overhead. For video thumbnails, we use ffmpeg which is really heavy. We off-load video thumbnail generation to a queue. We've had great luck with these tools.
- gh123man 2y ago
  +1 to vips! It's amazingly fast and stable. I even wrote (some minimal) Swift bindings for it to be used with a Swift backend: https://github.com/gh123man/SwiftVips https://github.com/gh123man/SwiftVips
- gopher_space 2y ago
  Thanks for the suggestion, and Meson keeps popping up. I vaguely remember flattening images in some manner so imagemagick wouldn't choke. Something about setting key tones? It's been a while.
  - 4RealFreedom 2y ago
    Didn't know if you were asking but Meson is used to build libvips from source. It took a couple days for me to get up to speed on it but wasn't difficult to understand. https://mesonbuild.com/ https://mesonbuild.com/ https://www.libvips.org/install.html https://www.libvips.org/install.html As for imagemagick, I just couldn't get the speed I wanted out of it. This is partly due to the fact that the system I was building allows many different image types. I want to say gifs were almost always an issue but I'm reaching here. Been a bit for me as well.
    - gopher_space 2y ago
      Yep, I keep wondering why I'm not just using make and a text editor on a lot of projects, but make feels old and creaky for some unexamined reason so I'm looking around. One of the major lessons I pulled out of that time is that most library authors will happily fix bugs or add features if you give them money. You can also just ask them to sell you a different license if FOSS ruins your plan.
tylergetsay 2y ago
I have used https://github.com/cshum/imagor https://github.com/cshum/imagor infront of S3 before and liked it, there is many (some commercial) offerings for this
jfengel 2y ago
I don't understand the details of cloud stuff, and it took me a heck of a long time to google just what a "pre-signed URL" was. In case anybody else is in the same bucket (ahem): Users can't upload to your S3 storage because they lack credentials. (It would be dangerous to make it public.) But you can give them access with a specially-generated URL (generated for each time they want to upload). So your server makes a special URL, "signed" with its own authorization. That lets them upload one file, to that specific URL. (I dunno about anybody else, but I find working with AWS always involves cramming in a lot of security-related concepts that I had never had to think about before. It puts a lot of overhead on simple operations, but presumably is mandatory if you want an application accessible to the world.)
- ilogik 2y ago
  you can use an EBS volume attached to the pod, or shared EFS volume as well. but for certain applications, s3 is simpler to use, especially when you need to scale
  - jfengel 2y ago
    I have zero idea what an EBS, EFS, or pod is. Not that I'm asking for an explanation. Just illustrating how much stuff there is to learn for the basic operation of "run this on your computer", even for an experienced developer. (I've been doing this for nearly 40 years.)
    - danrob 2y ago
      You should probably learn.
      - roland35 2y ago
        I took a class for AWS certification. Even though I never ended up taking the test, it has been extremely helpful throughout my career.
      - jfengel 2y ago
ben_jones 2y ago
Assuming you serve that content out through a CDN a lot of optimization work will be handled there and customization should also be handled there. I’d be shocked if CDNs don’t allow you to do much/all of that out of the box. Honestly though if this is an authenticated function and you have a small user base… who cares? Is there a reasonable chance at this disrupting any end user services? Maybe it’s not the best way to spend hundreds of hours and thousands of dollars. Granted you’re an SRE so it’s your job to ideas this. I’d just push back on defaulting to dropping serious resources on a process that might be entirely superfluous for your use case.
- tootie 2y ago
  If it's AWS there's basically nothing like this available as default. There are transcoding services and edge functions but you need to set them up yourself.
j45 2y ago
I'm not affiliated but a cloud file service from someone like backblaze may interest you
eastoeast 2y ago
Following a similar stack, has anyone found success handling iPhone HDR and Live Photos? Both seem to give issues to standard HTML formats. I believe we’re using an AWS service to convert videos to various qualities (maybe Elastic Transcoder or Media Convert), and those iPhone video formats causes the service to error out.
squigz 2y ago
Is HN turning into StackOverflow now?
- lobito14 2y ago
  This would probably be closed on SO as too broad.
  - whoknowsidont 2y ago
    Good. It doesn't work here either.
giaour 2y ago
> Occasionally, limits are set on the <input/> element for file types. Since this isn't enforced by the presigned PUT URL, you can't trust that the limits have been respected without inspecting whatever was uploaded to S3. You can get a lot more flexibility in what is allowed if you use an S3 presigned POST, which lets you set minimum and maximum allowed content lengths. [0]: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPO...
sigil 2y ago
Like you, we use pre-signed S3 upload urls. From there we use Transloadit [0] to crop and sanitize and convert and generate thumbnails. Transloadit is basically ImageMagick-as-a-Service. Running ImageMagick yourself on a huge variety of untrusted user input would be terrifying. [0] https://transloadit.com/ https://transloadit.com/
b-karl 2y ago
I was part of designing a user file upload, it was a B2B product,limited number of users and in principle trusted users but similar to other comments we did something like: - some file type and size checks in web app - pre-signed URL - upload bucket - lambdas for processing and sanity checks - processed bucket
atonse 2y ago
We use Cloudflare Images and Cloudflare Stream (Video) to process images and video that are uploaded to our site. Both have worked well for us so far but I don't know about your scale and impact on pricing (we're small scale so far). Cloudflare Images lets you auto resize images to generate thumbnails, etc. Same with video, where they will auto-encode the video based on who's watching it where. So for us it's just a matter of uploading it, getting an identifier, and storing that identifier.
hamandcheese 2y ago
For what its worth, processing the files is probably more risky for your internal infra than doing nothing. I've seen a RCE exploit from resizing profile images before. On the other hand, not processing/scanning your uploads is probably more risky for your users/the rest of the internet.
efxhoy 2y ago
We run an image scaler on aws lambda based on libvips. We cache the responses from it with cloudflare. We compared to letting cloudflare handle the scaling and the lambda was several times cheaper.
time0ut 2y ago
We map the TUS[0] protocol to S3 multipart upload operations. This lets us obscure the S3 bucket from the client and perform authorize each interaction. The TUS operations are handled by a dedicated micro-service. It could be done in a Lambda or anything. Once the upload completes we kick off a workflow to virus scan, unzip, decrypt, and process the file depending on what it is. We do some preliminary checks in the service looking at the file name, extension, magic bytes, that sort of stuff and reject anything that is obviously wrong. For virus scanning, we started with ClamAV[1], but eventually bought a Trend Micro product[2] for reasons that may not apply to you. It is serverless based on SQS, Lambda, and SNS. Works fine. Once scanned, we do a number of things. For images that you are going to serve back out, you for sure want to re-encode those and strip metadata. I haven't worked directly on that part in years, but my prototype used ImageMagick[3] to do this. I remember being annoyed with a Java binding for it. [0] https://tus.io/ https://tus.io/ [1] https://www.clamav.net/ https://www.clamav.net/ [2] https://cloudone.trendmicro.com/ https://cloudone.trendmicro.com/ [3] https://imagemagick.org/index.php https://imagemagick.org/index.php
whoknowsidont 2y ago
Is this really the state of the industry? Where an SRE is asking how to handle user media on the web? I'm not diminishing asking the question in principle, I'm questioning the role and forum that the question is being asked on.
dgoldstein0 2y ago
Some application security thoughts for serving untrusted content. Not all are required but the main thing is that you don't want the user to be able to serve html or similar (pdf, SVG?) file formats that can use your origin and therefore gain access to anything your origin can do: - serve on a different top level domain, ideally with random subdomains per uploaded file or user who provides the content. This is really most important for serving document types and probably not for images though SVG I think is the exception as it can have scripting and styling within when loaded outside of an IMG tag - set "content-security-policy: sandbox" (don't allow scripts and definitely don't allow same origin) - set "X-Content-Type-Options: no sniff" - disabling sniffing makes it a lot harder to submit an image that's actually interpreted as html or js later. Transforming the uploaded file also would defeat most exploit paths that depend on sniffing the content type.
- bawolff 2y ago
  Pdf isnt too bad, it has javascript but generally is sandboxed so you cant do anything bad with it. SVG is basically the same as html, and you can do standard XSS stuff. Hosting on separate domain (not just subdomain but actual separate domain) is a must if you allow formats like svg, and a good idea generally.
- technion 2y ago
  As a side issue here I can't believe sniffing is still the default. From basically the day it existed it's been considered a best practise to send that header with every request, because there might be a vulnerability introduced otherwise . I've never heard of anyone relying on it. Why can't modern browsers flip the default on this internet Explorer era magic footgun?
  - bawolff 2y ago
    > As a side issue here I can't believe sniffing is still the default It isn't the default in modern browsers, or at least not in the sense you mean. The header still controls some things, but even without it, browsers will not sniff something served as content type image/png as html or anything like that. How modern browsers interpret that is different than IE.
bagels 2y ago
If you are a big enough target, people will try to compromise your infrastructure or your users through these uploads. Some problems you can run in to: Exploiting image or video processing tools with crafted inputs, leading to server compromise when encoding/resizing. Having you host illegal or objectionable material. Causing extreme resource consumption, especially around video processing and storage. Having you host material that in some way compromises your clients (exploits bugs in jpeg libraries, cross site scripting, etc.) I can't really talk about what is done at the FAANG that I worked at on this stuff, but if you are a large enough target, this is a huge threat vector.
qingcharles 2y ago
I remember one major web host in 2004 .. I noticed they weren't checking the extension of profile pic uploads, so I uploaded a .aspx file that I wrote a file tree explorer into. From there I could browse through all of their customer's home directories; eventually I found the SQL database admin password, which turned out to be the same as their administrator password for the Windows server it was running on: "internet". This was a big lesson for me in upload sanitizing.
- freedomben 2y ago
  '04 was good times for that indeed lol. Early enough that almost no regular developers knew anything about security. SQL injection vulnerabilities were the norm lol. It's really a shame that things turned nasty around then as well. Most people hacked stuff for lulz in those days, but now they do extreme damage.
  - qingcharles 2y ago
    Yeah, I should have added a caveat that I emailed the admins immediately after I found the administrator password and let them know what was going on. I did set my profile pic to goatse on the way out though, just to put a fire under them to fix it.
nickjj 2y ago
Beyond taking advantage of validations that are enforced with IAM policies, you can also have a background job handle making thumbnails or whatever you want. Also I don't think the Content-Type is actually verified by S3 so technically users can still upload malicious files such as an executable with a png extension. On the bright side, S3 supports requesting a range of bytes. You can use that to perform validation server side afterwards to enforce it's really a png, jpg or whatever format you want. Here's examples in Python and Ruby to verify common image types by reading bytes: https://nickjanetakis.com/blog/validate-file-types-by-reading-the-first-few-bytes-of-a-file https://nickjanetakis.com/blog/validate-file-types-by-readin...
erhaetherth 2y ago
I tried to do the pre-signed URL thing but gave up quickly. I don't know how you'd do it properly. You're going to want a record of that in your database, right? So what, you have the client upload the image and then send a 2nd request to your server to tell you they uploaded it? I ended up piping it through my server. I can limit file size, authenticate, insert it into my DB and do what not this way.
- brianhama 2y ago
  Usually it’s the opposite order. Client requests an upload, an entry is made in the database and a pre signed url is generated and returned to the client. The client uploads the image. Optionally, the client could then tell the server the upload was completed, although I’ve never done that final step.
jftuga 2y ago
Slight OT... I created a program for profile pictures. It uses face recognition technology as to not deform faces when resizing photos. This may be useful to you. https://github.com/jftuga/photo_id_resizer https://github.com/jftuga/photo_id_resizer
time4tea 2y ago
A lot of people here are telling you to do stuff, but not really explaining "why". A different suggestion would be to build a threat model. Who are your uploaders? Who are your viewers? What are the threats? Only when you've figured out what the threats are, can the solutions make sense. Internal site? Internal users? Publicly available upload link? What media types? .... Threat modelling is a thing that adds lots of value (imho) when thinking about adding new capabilities to a system.
andersa 2y ago
> But then there is the question of cost on AWS side. Use Cloudflare R2 for storage, public bucket for delivery, and a $10/month Hetzner server to run conversions. You don't have to take your users' money just to burn it on AWS... If you are using AWS and are worried about cost, the first step is to not use AWS.
- cqqxo4zV46cp 2y ago
  I can think of 100 circumstances off the top of my head where this “cost saving” wouldn’t be net worth it. It very well may be sometimes, too. But your comment is very absolute, and very obviously intentionally so, so you could use this lecturing tone.
- byyll 2y ago
  As someone outside of the AWS trap, I am amazed at how many services people are saying you must use to have file uploads. 2 storage buckets that are charged in 10 different ways, a CDN, a processing server, a "lambda", "media convert", virus scanning and a third party service for resizing.. Extremely unnecessarily over complicated and probably overly expensive.
  - andersa 2y ago
    Yeah, companies really seem to like pissing away their users money on AWS. And then people wonder why everything is an expensive subscription now and somehow Amazon keeps getting richer...
tempaccount1234 2y ago
It depends on how small. I’m working with really small (like less than 5mb per upload) and use a FastAPI endpoint on my API to receive the file and then have other Python code save it (or reject it)
taitems 2y ago
Ensure you have adequate cleanup procedures too. I heard from ex-employees of a major car reselling platform that CSAM was distributed by creating a draft car ad, never publishing it and the CSAM was now hosted and accessible via direct image URLs. The records were orphaned, not sure how they got the tip-off.
sakopov 2y ago
This is how I would build this in AWS. Upload to S3 via pre-signed URLs. Create a notification on the bucket which publishes new objects to an SNS topic. Then create a lambda function with its own dedicated SQS queue subscribing to the SNS topic mentioned earlier. This setup would allow you to post-process new uploads without data loss (and at scale) especially if you use a DLQ. The lambda would drop all post-processed images into another bucket from which you can serve them safely via cloudfront distribution. Beware, if you have a high traffic site this will probably cost you an arm and a leg in S3 costs and Lambda executions. You could consider aging out items in the upload bucket to save some moneys. Similarly, you could use an auto-scaled ECS task instead of lambda, which would scale out (and in) based on the number of items in the queue. Not for the faint of heart, I know :))
samslade 2y ago
Hi! I've been working on this exact problem myself recently and decided to build a product out of it. Take a look: www.bucketscan.com The intention is to develop an API-driven approach that can be easily integrated into your own products as part of your file upload mechanism. We're really early stage, so we're looking for businesses and individuals to help us define what the product should look like. If you'd be up for sharing your thoughts, you can email us at info@bucketscan.com and/or complete this short product survey: https://forms.gle/rywgnQ7zqsPuLdMd6 https://forms.gle/rywgnQ7zqsPuLdMd6
cebert 2y ago
We allow uploads with pre-singed URLs at my work too. We added a few more constraints. First, we only allow clients to upload files with extensions that are reasonable for our apps. Files uploaded to S3 are quarantined with tags until we validate the binary contents appears to match the signature for a given extension/mime-type with a Lambda. Second, we use ClamAV to scan the uploaded files. Once these checks have completed, we generate a thumbnail with a Lambda and then make the file available to users. I’m honestly surprise this isn’t a value-added capability offered by AWS S3 because it’s such a common need and undifferentiated work.
- dividuum 2y ago
  Is using ClamAv because of some compliance checklist that forces you to include a virus scanner in there? Otherwise I don’t see how adding a ton of additional complexity with a proven track record of CVEs would add any security at all.
  - cebert 2y ago
    No compliance reason other than adding it in couldn’t hurt.
    - robertlagrant 2y ago
      How does > a ton of additional complexity with a proven track record of CVEs become: > adding it in couldn’t hurt ?
      - matja 2y ago
        By running it in an ephemeral container with no network access for each file you want to check?
- Kinrany 2y ago
  If S3 did this, what would that look like?
- thebeardedone 2y ago
brudgers 2y ago
I don't feel this is safe enough. I also feel we could do better... What is the business case for making the necessary changes? Good luck.
- matusgallik008 2y ago
  Well, security for starts. Having user A upload a malicious_file.png and then user B download it would not be ideal. Secondly, UX - having thumbnails for files can increase the loading speed. Arguably not the most important UX feature, but we have the resources to spend on this, so why not.
  - julik 2y ago
    > Having user A upload a malicious_file.png and then user B download it You will usually want the profile image of user A to be shown to user B. Same for videos and other files - users likely upload them so that others can download them / consume them in some way, right? I think where you need to start is doing some threat analysis, and proceed from there. Hosting user content can be built out from "very small" to "very big", depending on the particular threat scenario/use cases/your particular userbase. With the description that you are giving, I would say you are more at risk of building an overcomplicated ("oversecured") solution which might compromise UX for the sake of some protection that is not necessarily needed. If you are a small team, likely you could use an image resizing / video thumbnailing proxy server such as https://www.imgix.com/ https://www.imgix.com/ https://imgproxy.net/ https://imgproxy.net/ etc. You generate a signed URL to it and then the service picks up the file from S3 and does $thing to it. https://www.thumbor.org/ https://www.thumbor.org/ is another such tool. There are quite a few. Re. uploads and downloads - you have quite some options with S3. You can generate the presigned upload URL on the server (in fact: you should do just that), make it time limited and add the Content-Length of the upload to the signed headers - this way the server may restrict the size of the upload. Similarly, access to display the images can be done via a CDN or using low-TTL signed URLs... plenty of things to do.
jjice 2y ago
You'll probably have no issue sending the image to you backend directly, doing whatever you want to it (compression, validation, etc), and then uploading it to S3 from there. It's not a lot of overhead (and I'd argue more testable and easy to run locally). You can do the math on the ingress to your service (let's say it's EC2), and then the upload from EC2 to S3. It appears that AWS doesn't charge for ingress [0]. "There is no charge for inbound data transfer across all services in all Regions". S3 is half a cent per 1000 PUT requests [1], but you were already going to pay that anyway. Storage costs would also be paid anyway with a presigned URL. You'll have more overhead on your server, but how often do people upload files? It'll depend on your application. Generally, I'd lean towards sending it to your backend until it becomes an issue, but I doubt it ever will. Having the ability to run all the post processing you want and validate it is a big win. It's also just so much easier to test when everything is right there in your application. [0] https://aws.amazon.com/blogs/architecture/overview-of-data-transfer-costs-for-common-architectures/ https://aws.amazon.com/blogs/architecture/overview-of-data-t... [1] https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
- JimDabell 2y ago
  Aside from the extra load (and long requests, which is a bigger factor for non-async app servers), you need to take into account the physical location of your app servers. Depending on where the user is, there can be drastic differences in performance uploading to, say, us-east-1 vs ap-southeast-1. With S3, you can enable transfer acceleration so that the user uploads to whichever is their nearest region and then AWS handles it from there. This can give huge speedups.
moomoo11 2y ago
I use lambda for image uploads. The function ensures images aren’t bigger than 10mb, the image is compressed to our sizing, and put into s3.
- pier25 2y ago
  With Google Cloud Storage you can simply set a header with the max size allowed when signing the upload URL.
  - matusgallik008 2y ago
    I believe AWS S3 allows setting the Content-Length header as well. So you might not need the lambda
    - moomoo11 2y ago
      Can you explain what you mean? I'm confused about the "might not need the lambda". Thanks! In my case we allow up to 10mb file upload. For example just testing something right now from my iPhone I selected a 3.7mb image which was uploaded and resized to the "frame" it will go into in the UI and its now 288kb in S3. We have a few "frame" sizes which are consistent throughout our application. Now, my cloudfront is serving 288kb file instead of 3.7mb, which is good for me because I want to avoid the bandwidth costs and users honestly can't tell the difference, and then also we aren't a photo gallery app. So in my case I need the lambda to resize the image to our "frame" sizes. I wonder if I didn't explain right. I'm ESL so just want to make sure since I'm confused about the your last sentence.
      - aprilnya 2y ago
        They mean you might not need the lambda if all you want to do is enforce a max size; if you want to do resizing/other processing then you’d still need the lambda
      - matusgallik008 2y ago
        You're good mate. I meant exaclty what aprilnya said. I thought you are using the Lambda just to validate the 10mb limit - and just for that you don't need the lambda :) and could save yourself some cents & effort. But if you do resizing as well, then ofc you need the Lambda (or some other compute engine).
pier25 2y ago
For images we simply use Cloudflare Images which takes care of everything. Images are easy to display but for other media files you will probably need some streaming solution, a player, etc. We're in the audio hosting and processing space. We still don't have an api though. For video maybe look into Wistia, Cloudflare, BunnyCdn, etc.
kevincox 2y ago
I would encourage not directly using the user-uploaded images. But uploading directly to S3 is probably fine. I just wouldn't use the raw file. 1. Re-encoding the image is a good idea to make it harder to distribute exploits. For example imaging the recent WebP vulnerability. A malicious user could upload a compromised image as their profile picture and pwn anyone who saw that image in the app. There is a chance that the image survives the re-encoding but it is much less likely and at the very least makes your app not the easiest channel to distribute it. 2. It gives a good place to strip metadata. For example you should almost certianlly be stripping geo location. But in general I would recommend stripping everything non-essential. 3. Generating different sizes as you mentioned can be useful. 4. Allows accepting a variety of formats without requiring consumers to support them all. As you just transcode in one place. I don't know much about the cost on the AWS side, but it seems like you are always at some sort of risk given that if the user knows the bucket name they can create infinite billable requests. Can you create a size limit on the pre-signed URL? That would be a basic line of defence. But you probably also want to validate once the URL expires the data uploaded and decide if it conforms to your expectations (and delete it if you aren't interested in preserving the original data).
- hju22_-3 2y ago
  Do also note that re-encoding can also be used as part of the exploit. E.g. Team Fortress 2 recently had one that exploited a similar system.
  - toast0 2y ago
    There'a been explpots in image, video, and audio codecs... Which is why it's important to protect your users, but also your servers... Best to sandbox/jail/etc as tightly as possible, and limit the codecs to only what you need. You can configure the ffmpeg builds pretty granularly... default will include too much.
    - beeboobaa3 2y ago
      I care more about protecting my servers than my users. If one user attacks another that's not really my problem, blame can easily be shifted to the browser. But if someone hacks my servers and leaks everyone's data, that's my problem. So not encoding is probably the safer way to go for the business.

p2hari 2y ago

Look at https://uppy.io/ https://uppy.io/ open source and lot of integrations. You can keep moving to different levels of abstraction as required and see some good practices of how things are done.

junto 2y ago

You should quarantine them until you’ve analyzed them. Like you stated, an async process using a function would suffice. Previously used ClamAV for this in a private cloud solution, I’ve also used the built in anti-virus support on Azure Blob Storage if you don’t mind multi-cloud, plus an Azure Function has the ability to support blob triggers, which is a nice feature. The file types scan is relatively simple. You just need a list of known “magic string” header values to do a comparison, and for that you only need a max of 40 bytes of the beginning of the file to do the check (from memory). Depending on your stack, there are usually some libraries already available to perform the matching. And it goes without saying, but never trust the client, and always generate your own filenames. https://en.m.wikipedia.org/wiki/List_of_file_signatures https://en.m.wikipedia.org/wiki/List_of_file_signatures

dividuum 2y ago
I'm pretty sure ClamAv would add more vulnerabilities to your stack than it might prevent from being exploited.
- junto 2y ago
  It’s an old stack. Can you suggest an alternative self hosted option?
  - dividuum 2y ago
    I consider the approach of using virus scanning inherently flawed: They rely on heuristics and rules to essentially create a blacklist. And they add a ton of complexity (more code -> usually more bugs) which is not something you want if you work with untrustworthy data. What you instead want is a whitelist: Only allow properly formatted images and videos and ruthlessly reject anything else. I wrote about how I implemented this for my service in another response.

stevoski 2y ago

My SaaS uses Cloudinary for uploading and storing images. It’s not particularly cheap. But it is fast and flexible and safe.

sairamkunala 2y ago
My previous company used Cloudinary for the purpose and amazingly lot of time was saved.

arcza 2y ago

Kinda incredible how nearly every comment here mentions S3. Cloud storage is not the only backend in existence :)

Dowwie 2y ago
This is your moment. Let's hear about alternatives.
hnlmorg 2y ago
That’s because the requester is already using AWS and S3 specifically. One of the most important, yet oft ignored on HN, principles of architecting solutions is having an unbiased view of products and then choosing what will fit within an existing organisation’s architecture. Given the specifications shared, utilising S3 further is the correct advice.
morbicer 2y ago
It gives you at least some basic security. Nothing can get executed on S3, there is no lateral move. If you are using your own servers for user uploads you have to do a damn good job of quarantining them.

ytch 2y ago

https://github.com/google/magika https://github.com/google/magika Magika is a novel AI powered file type detection tool that relies on the recent advance of deep learning to provide accurate detection. It's not a sliver bullet, but I use it recently for inspecting file type instead of magic file. One advantage is that detecting composed files. Take pdf+exe file for example, the library will report something like 70% pdf and 30% pebin file.

mceachen 2y ago
I don't think this project was actually meant for production use, and especially not under decidedly hostile conditions. I'd suggest existing tooling like `exiftool` to do mimetype detection and metadata stripping.

PaywallBuster 2y ago

2 buckets: - upload bucket - processed bucket upload bucket has an event triggered on new file upload which triggers a lambda, the lambda will re-encode and do wtv you deem fit and upload to new bucket your app will use the processed bucket

sim7c00 2y ago

as poijted out already magika is useful for good file type analysis. also, scan thoroughly for viruses, potebtially using multiple engines. now this can be tricky depending on the confidentiality of the uploaded files so take good care not to submit them to sandboxes, av platforms etc. if not allowed. i would really recommend it though. if you want to get into the nitty grit of filetype abuse, to learn how to possibly detect that good. ange albertini's work on polymorphic filetypes and the ezine poc||gtfo as well as lots of articles by AV code devs are available. its really hard problem and also depends a lot on what program is interpreting the files submitted. if its some custom tool.there might even be unique vectors to take into account. fuzzing and penteting the upload form and any tools handing these files can shed light on those issues potentially. (edit: fat fingers)

marcpaq 2y ago

Have you considered a commercial solution? https://developer.massive.io/js-uploader/ https://developer.massive.io/js-uploader/ Just point it to your local files, as big as you want, and it does the rest. It handles the complexities of S3 juggling and browser constraints. Your users pay nothing to upload, you pay for egress. Full disclosure: I'm MASV's developer advocate.

fy20 2y ago

Lots of other comments give good suggestions on how to handle uploading and processing, but nothing mentions serving resulting content, so let me chime in: Do not serve content from S3 directly. ISPs often deprioritize traffic from S3, so downloading assets can be very slow. I've seen kbytes/s on a connection that Speedtest.net says has a download speed of 850 mbit. Putting Cloudfront in front of S3 solves that.

throwaway984393 2y ago
Not to mention using Cloudfront in front of S3 can lower cost, due to caching.
maayank 2y ago
> ISPs often deprioritize traffic from S3 I wonder why
bagels 2y ago
Cloudfront does not solve it. Comcast deprioritizes any content that comes from Amazon.
65 2y ago
Also a good idea to stream large files instead of trying to directly serve the file as-is. For video you essentially have to stream the file in order for it to load on Safari.
sakopov 2y ago
You should never be serving content from S3 directly anyway. Always setup a cloudfront distribution for your buckets and allow origin access identity (OAI) access to the bucket.

grishka 2y ago

In my fediverse server project[1], I convert all user-uploaded images to high-quality webp and store them like that. I discard the original files after that's done. I use imgproxy[2] to further resize and convert them on the fly for actual display. In general, I try my best to treat the original user-uploaded files like they're radioactive, getting rid of them as soon as possible. I don't do videos yet, but I'm kinda terrified of the idea of putting user-uploaded files through ffmpeg if/when I'll support them. [1] https://github.com/grishka/Smithereen https://github.com/grishka/Smithereen [2] https://github.com/imgproxy/imgproxy https://github.com/imgproxy/imgproxy

JohnCClarke 2y ago

What do you do with the uploaded images? You could be exposed to risks that may not be immediately obvious. I have seen a team struggle for over a month to eliminate NSFW content - avatars - uploaded by a user that lead to their site being demonetised.

PaulHoule 2y ago
I had a site demonetized because about 1 in 10,000 pictures were of deformed penises, dead nazis, things like that.
samus 2y ago
I think these days you'd have to deploy an NSFW detector model. And you'd have to silently reject the upload* and limit profile picture change rate, else the attacker could use the endpoint as an oracle to build an adversarial model, which can eventually generate inputs that the detector won't flag. *: act as if successful, including processing time, but throw away the upload

santiagobasulto 2y ago

It’s amazing that this was an issue in 2004 and it’s still an issue today. I don’t have much to add aside from what was already said. There are services like uppy, transload it, etc that simplify this, but might be more expensive than S3+CF.

dividuum 2y ago

I recently redesigned my stack of validating uploaded files and creating thumbnails from them. My approach is to have different binaries per file type (currently images JPEG/PNG, videos H264/265 and truetype fonts). Each of them is implemented as in a way that they receive the raw data stream via stdin and then either generate an error message or a raw planar RGBA data stream via stdout. The validation and thumbnail process is triggered after first locking in the process in a seccomp strict mode jail before touching any of the untrustworthy data. Seccomp prevents them from basically every syscall except read/write. Even if there would be an exploit in the format parser, it would very likely not get anywhere as there’s literally nothing it could do except write to stdout. Outside a strict time limit is enforced. The raw RGBA output is then received and converted back into PNG or similar. It was a bit tricky to get everything working without additional allocation and using syscalls triggered by glibc somewhere, but works pretty well now and is fast enough for my use case (around 20ms/item).

artwr 2y ago
Oh could you expand briefly on what the stack looks like to accomplish this? Or do you have a write up on a blog/site you could share?
- dividuum 2y ago
  I wrote a little bit more about this here: https://community.info-beamer.com/t/an-updated-approach-to-content-sandboxing/1277 https://community.info-beamer.com/t/an-updated-approach-to-c... I’m a huge fan of building minimal self-contained tools, so all of the C programs statically link in the required parser libs (libavcodec/wuffs/freetype) so the resulting binaries don’t require additional dependencies on the target machine. The python wrapping code is rather straightforward as well and is only like 300 lines of code.

TacticalCoder 2y ago

> I don't feel this is safe enough. I also feel we could do better by optimizing images on the BE, or creating thumbnails for videos. Yeah definitely. Even optimizing the vids. I just spend time writing scripts to convert, in parallel, a massive amount of JPG, PNG, PDFs, mp4 videos and even some HEIC files customers sent of their ID (identity card or passport, basically). I did shrink them all to reasonable size. The issue is: if you let user do anything, you'll have that one user, once in a while, that shall send a 30 MB JPG of his ID. Recto. Than Verso. Then the signed contracts: imagine a user printing a 15 pages contract, signing/paraphing every single page, then not scanning it but taking a 30 MB picture, with his phone, in diagonal, in perspective. And sending all the files individually. After a decade, this represented a massive amount of data. It was beautiful to crush that data to anywhere from 1/4th to 1/10th of its size and see all the cores working at full speed, compressing everything to reasonable sizes. Many sites and 3rd party identity verification services (whatever these are called) do put limit on the allowed size per document, which already helps. In my case I simply used ImageMagick (mogrify), ffmpeg (to convert to x265) and... GhostScript (good old gs command). PDFs didn't have to be searchable for text so there's that too (and often already weren't at least not easily, due to users taking pictures then creating a PDF out of the picture). This was not in Amazon S3 but basically all in Google Workspace: it was for a SME to make everything leaner, snapper, quicker, smaller. Cheaper too (no need to buy additional storage). Backups of all the originals, full size, files were of course made too but these shall probably never be needed. In my case I downloaded everything. Both to create backups (offsite, offline) and to crush everything locally (simply on an AMD 7700X: powerful enough as long as you don't have months of videos to encode). > Anybody have experience with any of this? I imagine having a big team and dedicated services for media processing could work, but what about small teams? I did it as a one-person job. Putting limits in place or automatically resizing, right after upload, a 30 MB JPG file which you know if of an ID card to a 3 MB JPG file doesn't require a team. Same for invoking the following to downsize vids: ffmpeg -i input.mp4 -vcodec libx265 -crf 28 output.mp4 (I think that's what I'm using) My script's logic were quite simple: files above a certain size were candidates for downsizing then downsizing then if the output was successful and took less than a certain amount of time, use that, otherwise keep the original. I didn't bother verifying that the files visually matched (once again: all the originals are available on offline backups in case something went south and some file is really badly needed) but I could have done that too. There was a blog post posted here a few years ago where a book author would visually compare thumbnails of different revisions of his book, to make sure that nothing changed too much between two minor revisions. I considered doing something similar but didn't bother. Needless to say my client is very happy with the results and the savings. YMMV but worked for me and worked for my client.

mattpavelle 2y ago

If you're looking for a good image optimization product, I've had excellent results from ImageOptim (I have no affiliation with them). They have a free Mac App, they have an API service, but also they kindly link to lots of other free similar products and services: https://imageoptim.com/versions.html https://imageoptim.com/versions.html If you can spare the CPU cycles (depending on how you optimize, they can actually be expensive) and if your images will be downloaded frequently, your users will thank you.

4RealFreedom 2y ago

Read through the comments and was surprised no one mentioned libvips - https://github.com/libvips/libvips https://github.com/libvips/libvips. At my current small company we were trying to allow image uploads and started with imagemagick but certain images took too long to process and we were looking for faster alternatives. It's a great tool with minimum overhead. For video thumbnails, we use ffmpeg which is really heavy. We off-load video thumbnail generation to a queue. We've had great luck with these tools.

gh123man 2y ago
+1 to vips! It's amazingly fast and stable. I even wrote (some minimal) Swift bindings for it to be used with a Swift backend: https://github.com/gh123man/SwiftVips https://github.com/gh123man/SwiftVips
gopher_space 2y ago
Thanks for the suggestion, and Meson keeps popping up. I vaguely remember flattening images in some manner so imagemagick wouldn't choke. Something about setting key tones? It's been a while.
- 4RealFreedom 2y ago
  Didn't know if you were asking but Meson is used to build libvips from source. It took a couple days for me to get up to speed on it but wasn't difficult to understand. https://mesonbuild.com/ https://mesonbuild.com/ https://www.libvips.org/install.html https://www.libvips.org/install.html As for imagemagick, I just couldn't get the speed I wanted out of it. This is partly due to the fact that the system I was building allows many different image types. I want to say gifs were almost always an issue but I'm reaching here. Been a bit for me as well.
  - gopher_space 2y ago
    Yep, I keep wondering why I'm not just using make and a text editor on a lot of projects, but make feels old and creaky for some unexamined reason so I'm looking around. One of the major lessons I pulled out of that time is that most library authors will happily fix bugs or add features if you give them money. You can also just ask them to sell you a different license if FOSS ruins your plan.

tylergetsay 2y ago

I have used https://github.com/cshum/imagor https://github.com/cshum/imagor infront of S3 before and liked it, there is many (some commercial) offerings for this

jfengel 2y ago

I don't understand the details of cloud stuff, and it took me a heck of a long time to google just what a "pre-signed URL" was. In case anybody else is in the same bucket (ahem): Users can't upload to your S3 storage because they lack credentials. (It would be dangerous to make it public.) But you can give them access with a specially-generated URL (generated for each time they want to upload). So your server makes a special URL, "signed" with its own authorization. That lets them upload one file, to that specific URL. (I dunno about anybody else, but I find working with AWS always involves cramming in a lot of security-related concepts that I had never had to think about before. It puts a lot of overhead on simple operations, but presumably is mandatory if you want an application accessible to the world.)

ilogik 2y ago
you can use an EBS volume attached to the pod, or shared EFS volume as well. but for certain applications, s3 is simpler to use, especially when you need to scale
- jfengel 2y ago
  I have zero idea what an EBS, EFS, or pod is. Not that I'm asking for an explanation. Just illustrating how much stuff there is to learn for the basic operation of "run this on your computer", even for an experienced developer. (I've been doing this for nearly 40 years.)
  - danrob 2y ago
    You should probably learn.
    - roland35 2y ago
      I took a class for AWS certification. Even though I never ended up taking the test, it has been extremely helpful throughout my career.
    - jfengel 2y ago

ben_jones 2y ago

Assuming you serve that content out through a CDN a lot of optimization work will be handled there and customization should also be handled there. I’d be shocked if CDNs don’t allow you to do much/all of that out of the box. Honestly though if this is an authenticated function and you have a small user base… who cares? Is there a reasonable chance at this disrupting any end user services? Maybe it’s not the best way to spend hundreds of hours and thousands of dollars. Granted you’re an SRE so it’s your job to ideas this. I’d just push back on defaulting to dropping serious resources on a process that might be entirely superfluous for your use case.

tootie 2y ago
If it's AWS there's basically nothing like this available as default. There are transcoding services and edge functions but you need to set them up yourself.

j45 2y ago

I'm not affiliated but a cloud file service from someone like backblaze may interest you

eastoeast 2y ago

Following a similar stack, has anyone found success handling iPhone HDR and Live Photos? Both seem to give issues to standard HTML formats. I believe we’re using an AWS service to convert videos to various qualities (maybe Elastic Transcoder or Media Convert), and those iPhone video formats causes the service to error out.

squigz 2y ago

Is HN turning into StackOverflow now?

lobito14 2y ago
This would probably be closed on SO as too broad.
- whoknowsidont 2y ago
  Good. It doesn't work here either.

giaour 2y ago

> Occasionally, limits are set on the <input/> element for file types. Since this isn't enforced by the presigned PUT URL, you can't trust that the limits have been respected without inspecting whatever was uploaded to S3. You can get a lot more flexibility in what is allowed if you use an S3 presigned POST, which lets you set minimum and maximum allowed content lengths. [0]: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPO...

sigil 2y ago

Like you, we use pre-signed S3 upload urls. From there we use Transloadit [0] to crop and sanitize and convert and generate thumbnails. Transloadit is basically ImageMagick-as-a-Service. Running ImageMagick yourself on a huge variety of untrusted user input would be terrifying. [0] https://transloadit.com/ https://transloadit.com/

b-karl 2y ago

I was part of designing a user file upload, it was a B2B product,limited number of users and in principle trusted users but similar to other comments we did something like: - some file type and size checks in web app - pre-signed URL - upload bucket - lambdas for processing and sanity checks - processed bucket

atonse 2y ago

We use Cloudflare Images and Cloudflare Stream (Video) to process images and video that are uploaded to our site. Both have worked well for us so far but I don't know about your scale and impact on pricing (we're small scale so far). Cloudflare Images lets you auto resize images to generate thumbnails, etc. Same with video, where they will auto-encode the video based on who's watching it where. So for us it's just a matter of uploading it, getting an identifier, and storing that identifier.

hamandcheese 2y ago

For what its worth, processing the files is probably more risky for your internal infra than doing nothing. I've seen a RCE exploit from resizing profile images before. On the other hand, not processing/scanning your uploads is probably more risky for your users/the rest of the internet.

efxhoy 2y ago

We run an image scaler on aws lambda based on libvips. We cache the responses from it with cloudflare. We compared to letting cloudflare handle the scaling and the lambda was several times cheaper.

time0ut 2y ago

We map the TUS[0] protocol to S3 multipart upload operations. This lets us obscure the S3 bucket from the client and perform authorize each interaction. The TUS operations are handled by a dedicated micro-service. It could be done in a Lambda or anything. Once the upload completes we kick off a workflow to virus scan, unzip, decrypt, and process the file depending on what it is. We do some preliminary checks in the service looking at the file name, extension, magic bytes, that sort of stuff and reject anything that is obviously wrong. For virus scanning, we started with ClamAV[1], but eventually bought a Trend Micro product[2] for reasons that may not apply to you. It is serverless based on SQS, Lambda, and SNS. Works fine. Once scanned, we do a number of things. For images that you are going to serve back out, you for sure want to re-encode those and strip metadata. I haven't worked directly on that part in years, but my prototype used ImageMagick[3] to do this. I remember being annoyed with a Java binding for it. [0] https://tus.io/ https://tus.io/ [1] https://www.clamav.net/ https://www.clamav.net/ [2] https://cloudone.trendmicro.com/ https://cloudone.trendmicro.com/ [3] https://imagemagick.org/index.php https://imagemagick.org/index.php

whoknowsidont 2y ago

Is this really the state of the industry? Where an SRE is asking how to handle user media on the web? I'm not diminishing asking the question in principle, I'm questioning the role and forum that the question is being asked on.

dgoldstein0 2y ago

Some application security thoughts for serving untrusted content. Not all are required but the main thing is that you don't want the user to be able to serve html or similar (pdf, SVG?) file formats that can use your origin and therefore gain access to anything your origin can do: - serve on a different top level domain, ideally with random subdomains per uploaded file or user who provides the content. This is really most important for serving document types and probably not for images though SVG I think is the exception as it can have scripting and styling within when loaded outside of an IMG tag - set "content-security-policy: sandbox" (don't allow scripts and definitely don't allow same origin) - set "X-Content-Type-Options: no sniff" - disabling sniffing makes it a lot harder to submit an image that's actually interpreted as html or js later. Transforming the uploaded file also would defeat most exploit paths that depend on sniffing the content type.

bawolff 2y ago
Pdf isnt too bad, it has javascript but generally is sandboxed so you cant do anything bad with it. SVG is basically the same as html, and you can do standard XSS stuff. Hosting on separate domain (not just subdomain but actual separate domain) is a must if you allow formats like svg, and a good idea generally.
technion 2y ago
As a side issue here I can't believe sniffing is still the default. From basically the day it existed it's been considered a best practise to send that header with every request, because there might be a vulnerability introduced otherwise . I've never heard of anyone relying on it. Why can't modern browsers flip the default on this internet Explorer era magic footgun?
- bawolff 2y ago
  > As a side issue here I can't believe sniffing is still the default It isn't the default in modern browsers, or at least not in the sense you mean. The header still controls some things, but even without it, browsers will not sniff something served as content type image/png as html or anything like that. How modern browsers interpret that is different than IE.

bagels 2y ago

If you are a big enough target, people will try to compromise your infrastructure or your users through these uploads. Some problems you can run in to: Exploiting image or video processing tools with crafted inputs, leading to server compromise when encoding/resizing. Having you host illegal or objectionable material. Causing extreme resource consumption, especially around video processing and storage. Having you host material that in some way compromises your clients (exploits bugs in jpeg libraries, cross site scripting, etc.) I can't really talk about what is done at the FAANG that I worked at on this stuff, but if you are a large enough target, this is a huge threat vector.

qingcharles 2y ago

I remember one major web host in 2004 .. I noticed they weren't checking the extension of profile pic uploads, so I uploaded a .aspx file that I wrote a file tree explorer into. From there I could browse through all of their customer's home directories; eventually I found the SQL database admin password, which turned out to be the same as their administrator password for the Windows server it was running on: "internet". This was a big lesson for me in upload sanitizing.

freedomben 2y ago
'04 was good times for that indeed lol. Early enough that almost no regular developers knew anything about security. SQL injection vulnerabilities were the norm lol. It's really a shame that things turned nasty around then as well. Most people hacked stuff for lulz in those days, but now they do extreme damage.
- qingcharles 2y ago
  Yeah, I should have added a caveat that I emailed the admins immediately after I found the administrator password and let them know what was going on. I did set my profile pic to goatse on the way out though, just to put a fire under them to fix it.

nickjj 2y ago

Beyond taking advantage of validations that are enforced with IAM policies, you can also have a background job handle making thumbnails or whatever you want. Also I don't think the Content-Type is actually verified by S3 so technically users can still upload malicious files such as an executable with a png extension. On the bright side, S3 supports requesting a range of bytes. You can use that to perform validation server side afterwards to enforce it's really a png, jpg or whatever format you want. Here's examples in Python and Ruby to verify common image types by reading bytes: https://nickjanetakis.com/blog/validate-file-types-by-reading-the-first-few-bytes-of-a-file https://nickjanetakis.com/blog/validate-file-types-by-readin...

erhaetherth 2y ago

I tried to do the pre-signed URL thing but gave up quickly. I don't know how you'd do it properly. You're going to want a record of that in your database, right? So what, you have the client upload the image and then send a 2nd request to your server to tell you they uploaded it? I ended up piping it through my server. I can limit file size, authenticate, insert it into my DB and do what not this way.

brianhama 2y ago
Usually it’s the opposite order. Client requests an upload, an entry is made in the database and a pre signed url is generated and returned to the client. The client uploads the image. Optionally, the client could then tell the server the upload was completed, although I’ve never done that final step.

jftuga 2y ago

Slight OT... I created a program for profile pictures. It uses face recognition technology as to not deform faces when resizing photos. This may be useful to you. https://github.com/jftuga/photo_id_resizer https://github.com/jftuga/photo_id_resizer

time4tea 2y ago

A lot of people here are telling you to do stuff, but not really explaining "why". A different suggestion would be to build a threat model. Who are your uploaders? Who are your viewers? What are the threats? Only when you've figured out what the threats are, can the solutions make sense. Internal site? Internal users? Publicly available upload link? What media types? .... Threat modelling is a thing that adds lots of value (imho) when thinking about adding new capabilities to a system.

andersa 2y ago

> But then there is the question of cost on AWS side. Use Cloudflare R2 for storage, public bucket for delivery, and a $10/month Hetzner server to run conversions. You don't have to take your users' money just to burn it on AWS... If you are using AWS and are worried about cost, the first step is to not use AWS.

cqqxo4zV46cp 2y ago
I can think of 100 circumstances off the top of my head where this “cost saving” wouldn’t be net worth it. It very well may be sometimes, too. But your comment is very absolute, and very obviously intentionally so, so you could use this lecturing tone.
byyll 2y ago
As someone outside of the AWS trap, I am amazed at how many services people are saying you must use to have file uploads. 2 storage buckets that are charged in 10 different ways, a CDN, a processing server, a "lambda", "media convert", virus scanning and a third party service for resizing.. Extremely unnecessarily over complicated and probably overly expensive.
- andersa 2y ago
  Yeah, companies really seem to like pissing away their users money on AWS. And then people wonder why everything is an expensive subscription now and somehow Amazon keeps getting richer...

tempaccount1234 2y ago

It depends on how small. I’m working with really small (like less than 5mb per upload) and use a FastAPI endpoint on my API to receive the file and then have other Python code save it (or reject it)

taitems 2y ago

Ensure you have adequate cleanup procedures too. I heard from ex-employees of a major car reselling platform that CSAM was distributed by creating a draft car ad, never publishing it and the CSAM was now hosted and accessible via direct image URLs. The records were orphaned, not sure how they got the tip-off.

sakopov 2y ago

This is how I would build this in AWS. Upload to S3 via pre-signed URLs. Create a notification on the bucket which publishes new objects to an SNS topic. Then create a lambda function with its own dedicated SQS queue subscribing to the SNS topic mentioned earlier. This setup would allow you to post-process new uploads without data loss (and at scale) especially if you use a DLQ. The lambda would drop all post-processed images into another bucket from which you can serve them safely via cloudfront distribution. Beware, if you have a high traffic site this will probably cost you an arm and a leg in S3 costs and Lambda executions. You could consider aging out items in the upload bucket to save some moneys. Similarly, you could use an auto-scaled ECS task instead of lambda, which would scale out (and in) based on the number of items in the queue. Not for the faint of heart, I know :))

samslade 2y ago

Hi! I've been working on this exact problem myself recently and decided to build a product out of it. Take a look: www.bucketscan.com The intention is to develop an API-driven approach that can be easily integrated into your own products as part of your file upload mechanism. We're really early stage, so we're looking for businesses and individuals to help us define what the product should look like. If you'd be up for sharing your thoughts, you can email us at info@bucketscan.com and/or complete this short product survey: https://forms.gle/rywgnQ7zqsPuLdMd6 https://forms.gle/rywgnQ7zqsPuLdMd6

cebert 2y ago

We allow uploads with pre-singed URLs at my work too. We added a few more constraints. First, we only allow clients to upload files with extensions that are reasonable for our apps. Files uploaded to S3 are quarantined with tags until we validate the binary contents appears to match the signature for a given extension/mime-type with a Lambda. Second, we use ClamAV to scan the uploaded files. Once these checks have completed, we generate a thumbnail with a Lambda and then make the file available to users. I’m honestly surprise this isn’t a value-added capability offered by AWS S3 because it’s such a common need and undifferentiated work.

dividuum 2y ago
Is using ClamAv because of some compliance checklist that forces you to include a virus scanner in there? Otherwise I don’t see how adding a ton of additional complexity with a proven track record of CVEs would add any security at all.
- cebert 2y ago
  No compliance reason other than adding it in couldn’t hurt.
  - robertlagrant 2y ago
    How does > a ton of additional complexity with a proven track record of CVEs become: > adding it in couldn’t hurt ?
    - matja 2y ago
      By running it in an ephemeral container with no network access for each file you want to check?
Kinrany 2y ago
If S3 did this, what would that look like?
thebeardedone 2y ago

brudgers 2y ago

I don't feel this is safe enough. I also feel we could do better... What is the business case for making the necessary changes? Good luck.

matusgallik008 2y ago
Well, security for starts. Having user A upload a malicious_file.png and then user B download it would not be ideal. Secondly, UX - having thumbnails for files can increase the loading speed. Arguably not the most important UX feature, but we have the resources to spend on this, so why not.
- julik 2y ago
  > Having user A upload a malicious_file.png and then user B download it You will usually want the profile image of user A to be shown to user B. Same for videos and other files - users likely upload them so that others can download them / consume them in some way, right? I think where you need to start is doing some threat analysis, and proceed from there. Hosting user content can be built out from "very small" to "very big", depending on the particular threat scenario/use cases/your particular userbase. With the description that you are giving, I would say you are more at risk of building an overcomplicated ("oversecured") solution which might compromise UX for the sake of some protection that is not necessarily needed. If you are a small team, likely you could use an image resizing / video thumbnailing proxy server such as https://www.imgix.com/ https://www.imgix.com/ https://imgproxy.net/ https://imgproxy.net/ etc. You generate a signed URL to it and then the service picks up the file from S3 and does $thing to it. https://www.thumbor.org/ https://www.thumbor.org/ is another such tool. There are quite a few. Re. uploads and downloads - you have quite some options with S3. You can generate the presigned upload URL on the server (in fact: you should do just that), make it time limited and add the Content-Length of the upload to the signed headers - this way the server may restrict the size of the upload. Similarly, access to display the images can be done via a CDN or using low-TTL signed URLs... plenty of things to do.

jjice 2y ago

You'll probably have no issue sending the image to you backend directly, doing whatever you want to it (compression, validation, etc), and then uploading it to S3 from there. It's not a lot of overhead (and I'd argue more testable and easy to run locally). You can do the math on the ingress to your service (let's say it's EC2), and then the upload from EC2 to S3. It appears that AWS doesn't charge for ingress [0]. "There is no charge for inbound data transfer across all services in all Regions". S3 is half a cent per 1000 PUT requests [1], but you were already going to pay that anyway. Storage costs would also be paid anyway with a presigned URL. You'll have more overhead on your server, but how often do people upload files? It'll depend on your application. Generally, I'd lean towards sending it to your backend until it becomes an issue, but I doubt it ever will. Having the ability to run all the post processing you want and validate it is a big win. It's also just so much easier to test when everything is right there in your application. [0] https://aws.amazon.com/blogs/architecture/overview-of-data-transfer-costs-for-common-architectures/ https://aws.amazon.com/blogs/architecture/overview-of-data-t... [1] https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/

JimDabell 2y ago
Aside from the extra load (and long requests, which is a bigger factor for non-async app servers), you need to take into account the physical location of your app servers. Depending on where the user is, there can be drastic differences in performance uploading to, say, us-east-1 vs ap-southeast-1. With S3, you can enable transfer acceleration so that the user uploads to whichever is their nearest region and then AWS handles it from there. This can give huge speedups.

moomoo11 2y ago

I use lambda for image uploads. The function ensures images aren’t bigger than 10mb, the image is compressed to our sizing, and put into s3.

pier25 2y ago
With Google Cloud Storage you can simply set a header with the max size allowed when signing the upload URL.
- matusgallik008 2y ago
  I believe AWS S3 allows setting the Content-Length header as well. So you might not need the lambda
  - moomoo11 2y ago
    Can you explain what you mean? I'm confused about the "might not need the lambda". Thanks! In my case we allow up to 10mb file upload. For example just testing something right now from my iPhone I selected a 3.7mb image which was uploaded and resized to the "frame" it will go into in the UI and its now 288kb in S3. We have a few "frame" sizes which are consistent throughout our application. Now, my cloudfront is serving 288kb file instead of 3.7mb, which is good for me because I want to avoid the bandwidth costs and users honestly can't tell the difference, and then also we aren't a photo gallery app. So in my case I need the lambda to resize the image to our "frame" sizes. I wonder if I didn't explain right. I'm ESL so just want to make sure since I'm confused about the your last sentence.
    - aprilnya 2y ago
      They mean you might not need the lambda if all you want to do is enforce a max size; if you want to do resizing/other processing then you’d still need the lambda
    - matusgallik008 2y ago
      You're good mate. I meant exaclty what aprilnya said. I thought you are using the Lambda just to validate the 10mb limit - and just for that you don't need the lambda :) and could save yourself some cents & effort. But if you do resizing as well, then ofc you need the Lambda (or some other compute engine).

pier25 2y ago

For images we simply use Cloudflare Images which takes care of everything. Images are easy to display but for other media files you will probably need some streaming solution, a player, etc. We're in the audio hosting and processing space. We still don't have an api though. For video maybe look into Wistia, Cloudflare, BunnyCdn, etc.

kevincox 2y ago

I would encourage not directly using the user-uploaded images. But uploading directly to S3 is probably fine. I just wouldn't use the raw file. 1. Re-encoding the image is a good idea to make it harder to distribute exploits. For example imaging the recent WebP vulnerability. A malicious user could upload a compromised image as their profile picture and pwn anyone who saw that image in the app. There is a chance that the image survives the re-encoding but it is much less likely and at the very least makes your app not the easiest channel to distribute it. 2. It gives a good place to strip metadata. For example you should almost certianlly be stripping geo location. But in general I would recommend stripping everything non-essential. 3. Generating different sizes as you mentioned can be useful. 4. Allows accepting a variety of formats without requiring consumers to support them all. As you just transcode in one place. I don't know much about the cost on the AWS side, but it seems like you are always at some sort of risk given that if the user knows the bucket name they can create infinite billable requests. Can you create a size limit on the pre-signed URL? That would be a basic line of defence. But you probably also want to validate once the URL expires the data uploaded and decide if it conforms to your expectations (and delete it if you aren't interested in preserving the original data).

hju22_-3 2y ago
Do also note that re-encoding can also be used as part of the exploit. E.g. Team Fortress 2 recently had one that exploited a similar system.
- toast0 2y ago
  There'a been explpots in image, video, and audio codecs... Which is why it's important to protect your users, but also your servers... Best to sandbox/jail/etc as tightly as possible, and limit the codecs to only what you need. You can configure the ffmpeg builds pretty granularly... default will include too much.
  - beeboobaa3 2y ago
    I care more about protecting my servers than my users. If one user attacks another that's not really my problem, blame can easily be shifted to the browser. But if someone hacks my servers and leaks everyone's data, that's my problem. So not encoding is probably the safer way to go for the business.