9 ms·
I would encourage not directly using the user-uploaded images. But uploading directly to S3 is probably fine. I just wouldn't use the raw file. 1. Re-encoding
by kevincox 2y ago
I would encourage not directly using the user-uploaded images. But uploading directly to S3 is probably fine. I just wouldn't use the raw file.
1. Re-encoding the image is a good idea to make it harder to distribute exploits. For example imaging the recent WebP vulnerability. A malicious user could upload a compromised image as their profile picture and pwn anyone who saw that image in the app. There is a chance that the image survives the re-encoding but it is much less likely and at the very least makes your app not the easiest channel to distribute it.
2. It gives a good place to strip metadata. For example you should almost certianlly be stripping geo location. But in general I would recommend stripping everything non-essential.
3. Generating different sizes as you mentioned can be useful.
4. Allows accepting a variety of formats without requiring consumers to support them all. As you just transcode in one place.
I don't know much about the cost on the AWS side, but it seems like you are always at some sort of risk given that if the user knows the bucket name they can create infinite billable requests. Can you create a size limit on the pre-signed URL? That would be a basic line of defence. But you probably also want to validate once the URL expires the data uploaded and decide if it conforms to your expectations (and delete it if you aren't interested in preserving the original data).
- hju22_-3 2y agoDo also note that re-encoding can also be used as part of the exploit. E.g. Team Fortress 2 recently had one that exploited a similar system.
- toast0 2y agoThere'a been explpots in image, video, and audio codecs... Which is why it's important to protect your users, but also your servers... Best to sandbox/jail/etc as tightly as possible, and limit the codecs to only what you need. You can configure the ffmpeg builds pretty granularly... default will include too much.
- beeboobaa3 2y agoI care more about protecting my servers than my users. If one user attacks another that's not really my problem, blame can easily be shifted to the browser. But if someone hacks my servers and leaks everyone's data, that's my problem. So not encoding is probably the safer way to go for the business.
- mthoms 2y agoYikes. What if the other user getting attacked is you or another admin on your team? Now the attacker has admin access and can compromise your servers and “leak everyone’s data” just fine. I don’t think you’ve thought this through.
- bawolff 2y agoTrust me, your website will always get blamed regardless of its a user fault. If websites get blamed when users reuse passwords, they are definitely getting blamed if you distribute malicious files.
- theendisney 2y agoYou can think that but you should never say or write it. :p
- beeboobaa3 2y agoIt was just a joke, your honor.
- theendisney 2y agoI got the picture.
- kevincox 2y agoI don't think the exploit in that case was re-encoding. What happened is an image with very large dimensions was uploaded. When this was decoded into a raw pixel buffer on the client it used tons of memory. It was effectively a zip bomb attack. In fact re-encoding probably would have solved this as the server could enforce the expected dimensions and rescale or reject the image.
- arrowsmith 2y agoSounds complicated. Any recommendations for existing libraries I can use to handle this?
- RobotToaster 2y agoI know lemmy uses pict-rs https://git.asonix.dog/asonix/pict-rs/ https://git.asonix.dog/asonix/pict-rs/
- JimDabell 2y agohttps://www.libvips.org https://www.libvips.org
- tetris11 2y ago> recent WebP vulnerability For anyone wondering: https://blog.cloudflare.com/uncovering-the-hidden-webp-vulnerability-cve-2023-4863/ https://blog.cloudflare.com/uncovering-the-hidden-webp-vulne... > The vulnerability allows an attacker to create a malformed WebP image file that makes libwebp write data beyond the buffer memory allocated to the image decoder. By writing past the legal bounds of the buffer, it is possible to modify sensitive data in memory, eventually leading to execution of the attacker's code.
- chefkd 2y agoI'm not the sharpest tool in the shed just learning python right now can the wizards explain how this would work? Isn't data stored in memory super random so the attacker would need to upload an image multiple times to get sensitive data? Or does the webp image itself have code that will be run and can retrieve the data?
- dmoy 2y agoGenerally with a buffer overrun, you're trying to put executable code in, and get it to a spot where it'll be run, allowing you to do further stuff. Here is a canonical old article on one type of buffer overflow: http://www.phrack.org/issues/49/14.html#article http://www.phrack.org/issues/49/14.html#article (The webp one, iirc, is not a stack buffer overflow, but rather a heap, but that'll give you a general sense of what's going on, even if that article is a bit ... dated)
- vitro 2y ago> 3. Generating different sizes as you mentioned can be useful. For years now I use nginx's image filter [1] which handles file resizing quite nicely. Resized images are cached by nginx. For some usecases it works very vell and I no longer need to specify sizes beforehand, you just ask for the size by crafting your url properly. [1] https://nginx.org/en/docs/http/ngx_http_image_filter_module.html https://nginx.org/en/docs/http/ngx_http_image_filter_module....
- smackeyacky 2y agoThis is good advice. Just pick a resize dimension and try to resize everything that comes in. If it fails it's not an image. You can hang an event off S3 and have a lambda that does the work / warns you of a bad upload.
- kijin 2y agoResizing also helps reduce costs. The latest phones can generate ridiculously large images with 200+ megapixels. You really don't want to dump that kind of behemoth in your S3 bucket and serve it as somebody's profile pic. Videos will add even more to your AWS bill if you're not careful. Re-encode that 4K cat video as soon as it comes in, or wire up a CDN to do it for you.
- mmsc 2y ago>Re-encoding the image is a good idea to make it harder to distribute exploits. For example imaging the recent WebP vulnerability. And now your server which is doing re-encoding is pwned. Gotta segment the server doing that somehow, or use pledge, capsicum, or seccomp I guess.
- afiori 2y agoFor video you might want to use the GPU, but for images this sounds like a good use case for Wasm
- dpkirchner 2y agoThis is a pretty good use case for serverless endpoints, assuming the volume is pretty low.
- hamandcheese 2y agoServerless + making sure the serverless function doesn't have any privileges. More often than not I see people use serverless in the name of security, and then give the function write access to prod resources.
- Hamuko 2y agoI've ran into a security issue where a serverless function had pretty large range of AWS access and a pentester was able to utilise that.
- dpkirchner 2y agoThat's a bad use case for serverless :)
- cyberpunk 2y agoIdeal use case for a cheeky OpenBSD machine... Not that OpenBSD is actually unhackable or anything, but I doubt many attackers would guess you're running imagemagick on OpenBSD in your image pipeline. I rather like it for such use cases; it has the added benefit that it never, ever seems to die. I found a 6.0 machine I setup doing some kind of risky Kafka processing that had an uptime of 6 years the other week (since migrated).
- JimDabell 2y ago> For example you should almost certianlly be stripping geo location. But in general I would recommend stripping everything non-essential. Including animation in most cases. Otherwise somebody can use a single frame with very long duration that will be reviewed by moderators, then follow that frame with a different frame containing objectionable material which will eventually be shown to people.
- joshstrange 2y agoWow, that's something I had never considered. I did run into a bug with someone uploading a gif to our servers and our resize script spitting out a file per resized gif frame (base_1.png, base_2.png, ...) that I had to fix but I'd never have thought of your example. We just took the first gif frame in our case, which would have been safe from this thankfully.
- qingcharles 2y agoI often find you can upload animated GIFs to sites that don't allow them by just renaming them to PNG first.
- busymom0 2y ago> Can you create a size limit on the pre-signed URL Yes, pre-signed URL can have the `Content-Length` set and amazon S3 checks it. However, note that this is true for Amazon S3 but not for others like BackBlaze or R2. Last time I tried, BackBlaze didn't support it.
- gehen88 2y agoOnly with createPresignedPost, not with getSignedUrl.
- perpil 2y agoPresigned post lets you set content-length-range: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html#sigv4-PolicyConditions https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authen... You can specify content length on presigned PUTs, but it needs to be set as a header and added to the signed headers for SIGv4. It can't be set as a query param.
- _8j50 2y agoYou can tell a lot from the exif metadata of images so that's one reason (user privacy) to always re-encode images.
- sdsd 2y ago>Re-encoding the image is a good idea to make it harder to distribute exploits. Famously, the Dangerous Kitten hacking toolset was distributed with the classic zip-in-a-jpeg technique, because imageboards used to not re-encode images. https://web.archive.org/web/20120322025258/http://partyvan.info/wiki/Dangerous_Kitten https://web.archive.org/web/20120322025258/http://partyvan.i...
- smsm42 2y agoThere's also an old bug where browsers re-interpreted image as HTML (even with correct MIME type set) and this allowed to host exploits on user-upload sites. Not sure if any modern browser still has this problem, but it used to be a concern. Recoding the image usually broke those exploits. Though it also could break some metadata - e.g. if you go from JPEG to PNG you could lose EXIF data.
- blincoln 2y agoThis still generally works with SVGs.
- Frost1x 2y agoI’m pretty sure, if memory serves me, some of the older versions of IE (Internet Explorer, I imagine some people here may be young enough to not be aware of that) used to let you pass any sort of file hosted over HTTP with arbitrary extension set (e.g. ‘.jpeg’) however it would actually do some MIME typing and execute the file based on the actual content, so you could literally set a .exe with a jpeg file extension at the time and IE would let you run arbitrary executables on Windows users systems at the time with full permission of the user running it (which was pretty much often admin). You could also do other file types but an executable was the most glaring example of how bad the issue was. Obviously you could embed this in HTML as well…
- perpil 2y ago> Can you create a size limit on the pre-signed URL? Yes, if you use the POST method you can set the content-length-range property in your presigned URL form inputs to limit min and max bytes. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html#sigv4-PolicyConditions https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authen...
- cpeterso 2y agoDidn’t some photo gallery service (Google Photos or Takeout?) make the news because they re-encoded users’ images but didn’t preserve the original files? People who relied on the service as a safe backup lost their original quality images. So in some cases, you may want to re-encode/optimize uploaded images for display but also archive the original files somewhere.
- cqqxo4zV46cp 2y agoIt obviously depends on the use case.
- bufferoverflow 2y ago> Re-encoding the image is a good idea to make it harder to distribute exploits Which immediately makes you vulnerable to zip-bomb attacks. If you want to be more safe, you first have to do all kinds of checks of the image headers.
- deadbabe 2y ago#1 sounds interesting. Any examples of this being used?