6 ms·
Don‘t underestimate the massive cpu power needed for scaling images at their request rate
by tpetry 7y ago
Don‘t underestimate the massive cpu power needed for scaling images at their request rate
- nine_k 7y agoMaybe some caching would help.
- abhiyerra 7y agoWonder if they could use S3 bucket with a lambda to resize. I’m sure their images are a long tail so certain images likely have way more traffic than others. At a previous company where we were also doing a lot of imgix traffic we did something like this but with a go service but now I’d just offload to s3 and use lambda to avoid Devops..
- scarface74 7y agoI’ve done exactly this. But not at any scale. But if they are specializing in this and with low margins, AWS is not the answer. The compute cost would be relatively cheap and possibly even the storage cost. I would think that the bandwidth costs would be expensive.
- philliphaydon 7y agoA friend of mine runs a site which uses aws s3 and lambda resizing, but cache is via BunnyCDN. His costs are low after moving from aws cloudfront, and he’s serving ~400m images a day. (This is run by a company of 2 staff) Just because everything is in AWS doesn’t force you to use all of AWS services.
- superwayne 7y agoDo you happen to have a link?
- manigandham 7y agohey man, nice seeing you here.
- m0zg 7y agoWhy would it depend on request rate rather than upload rate? Just pre-scale several variants on upload. Then run an MR job every now and then and delete the variants that haven't been touched for more than e.g. a month. Scale those on demand, store recents and frequently accessed pre-scaled. This is literally a few days of work.
- manigandham 7y agoIt's much nicer to be able to resize on demand. Many CDNs and image/file hosting services offer transform abilities via URL which makes it easy to get exactly what you need and optimize for every site and device. But since images are immutable, this is a perfect use-case for lambda functions fronted by a CDN.
- puzzle 7y agoI bet there's a very long tail, i.e. the majority of images are rarely accessed, if ever.
- manigandham 7y agoWe run an ad platform that resizes, crops to the important section, transforms with text, and more with billions of requests per day. It's very cacheable, horizontally scalable, and CPU is cheap. The main scaling challenge here is serving and bandwidth, both of which are handled by their CDN. I expect 5 figure costs for that but the rest of the platform is very expensive.