11 ms·
Cost of serving billions of images per month
- no1youknowz 7y agoI like Unsplash. They and Pixabay are what I use for apps I have developed in the past. But simply, this floors me. I looked at their costs and with some developer muscle, you could find savings such as: - Move Fastly to Cloudflare. They don't expressly say what the cost is for that. But moving to CF would eliminate it. - Move Heroku to Digital Ocean. It's not difficult to create a fully redundant solution. - Move from Imgix to having Golang micro services which handle the resizing of images and use something like Belugacdn for the CDN. Beluga is $5k a month for a PB. (Or some other cheaper CDN if you don't like Beluga, but damn... imgix pricing) I'm pretty sure that a savvy CTO could save at least $50k a month with a well designed project that does this over many months and achieves the same result and keeps the redundancy concerns for the small team. I do realise however, why the team have done this. In the same position (with very few resources) I would probably have done the same. But damn, when a cost of a service gets up to a years salary ($120k) for a good developer. Time to seek alternatives.
- qes 7y agoThey say they spend $12,700/month on Fastly. I guess I don't know offhand the difference between Fastly and Imgix (which they also say is their CDN), but the levels of bandwidth they're talking about are not going to be free at Cloudflare. We serve significantly more bandwidth and requests than their 2016 numbers at under $15k compared to their $17.5k, and we're not particularly lean on infra costs being a .Net shop with Win servers and MS SQL on Azure, and I can still bring our monthly a lot closer to $10k if I can find time for some projects. But the reality is that even thousands of dollars per month saved can take a good while to ROI against the costs of development, or more so the opportunity costs.
- judge2020 7y ago> the levels of bandwidth they're talking about are not going to be free at Cloudflare Correct. Cloudflare's terms state that you can't serve a disproportionate/substantial amount of non-HTML content, so they would need to sign up for Enterprise with a full contract to get image serving via CF. Enterprise is generally $5k+/month, but CF contracts can offer a subset of enterprise features from $500-$5k/month, so I would guess ~3k/month just to serve multiple petabytes of images. This doesn't eliminate the bandwidth costs so CF may not be the single solution for the bandwidth costs.
- odensc 7y agoThey specifically say they use Imgix's CDN for the images, which can be confirmed by looking at the response headers. Fastly is used for their actual website, judging by the ASN info. Meaning they wouldn't be serving petabytes of images from CloudFlare, just HTML/CSS/JS, which should be perfectly fine.
- marcinzm 7y ago$50k/month comes out to $600k/year which is the cost around 2-3 engineers counting overhead. To manage infrastructure at that scale themselves they'd need someone who knows DevOps. To write services at that scale they'd need someone who knows backend engineering and micro services. Then they'd need to be able to have 24/7 on call rotations for when things break. And of course, like any rewrite it seems like a "couple month" project until you get into the details and it becomes a two year project. Soon you've got a full 6+ person team and are paying them a lot more than $600k/year.
- manigandham 7y agoAnd some projects really are just simple. Scale makes no difference to their services, it's a basic OLTP + search with resizing image proxy. This doesn't have any SLAs nor is it critical software either. 24/7 oncall is completely unnecessary.
- tpetry 7y agoDon‘t underestimate the massive cpu power needed for scaling images at their request rate
- nine_k 7y agoMaybe some caching would help.
- abhiyerra 7y agoWonder if they could use S3 bucket with a lambda to resize. I’m sure their images are a long tail so certain images likely have way more traffic than others. At a previous company where we were also doing a lot of imgix traffic we did something like this but with a go service but now I’d just offload to s3 and use lambda to avoid Devops..
- scarface74 7y agoI’ve done exactly this. But not at any scale. But if they are specializing in this and with low margins, AWS is not the answer. The compute cost would be relatively cheap and possibly even the storage cost. I would think that the bandwidth costs would be expensive.
- puzzle 7y agoResizing images in Golang is not as trivial as you might imagine at first. You need to be able to handle all sorts of formats and color spaces. You'd be surprised by the kind of weird garbage your users will upload. Then you need to use SIMD, not pure Golang solutions, or performance will suffer. So you end up adopting a wrapper around libvips or similar, at which point you will start to ponder if you should have stuck with C/C++ in the first place. (It all depends on how much of Go's features you use or if it's just a nicer, safer C for you.)
- bdibs 7y agoI’m stunned that a business of this size would still be using Heroku. Heroku’s sweet spot (to me at least) is for anyone with less than 3-4 servers. After that, it just becomes so expensive there’s no rationalization in the world that makes sense.
- quickthrower2 7y agoMoreover what if Heroku goes out of business? Then they’re f’d. Hosting on k8s, using Terraform or whatever would give them a lot of automation and provider independence.
- quickthrower2 7y agoI’m guessing I’m being grayed because I’m suggesting Heroku might go out of business? Ok let me explain. That is one scenario, it’s unlikely but there might be other similar scenarios that pose risk being tied in to one providers stack. DDOS, TOS issues, simply out growing them, Heroku themselves having TOS issues with their cloud. There are almost weekly posts on HN of “this major service is down” and “that major service is down”. I understand a scrappy startup needing to focus on growth and using the most convenient tools, but once you’ve been around a bit it’s time to think of boring stuff like what if stuff goes wrong. It might be technical, security related, business related or political. Eliminating SPOFs is wise. Happy for you to downvote this but if you do please drop a sentence to say why I’m wrong. I’m happy to be proved wrong it helps me learn.
- mypalmike 7y agoI didn't downvote, but... "I understand a scrappy startup needing to focus on growth and using the most convenient tools" Well, there's your answer. As a scrappy startup, you can't afford to plan migrations you aren't planning to execute. It's that simple. Heroku is not going away (they're owned by salesforce) and they have a good reputation for uptime. The "what if" worst case scenarios have to be weighed against other business concerns. And this company appears to have done a good job of navigating such things.
- 7y ago
- blissofbeing 7y agoNow I'm interested to see what their business model is. It seems like they have a lot of money to throw at services instead of developing it themselves, so I'm interested in their ROI or opportunity cost decisions now.
- lwansbrough 7y agoHoly cloud. Well I guess the silver lining is they're leaving lots of room for cost improvements...
- maz1b 7y agoI'm a little surprised as to how you guys afford to give such a quality service away for free. Really cool of you guys to share this data though. Wishing you guys continued success!
- manigandham 7y agoWhat's the business model here? We would never run with such expensive options for this functionality, especially if it was a free/donations powered service. This company raised $7.5M though: https://techcrunch.com/2018/02/15/unsplash-simple-token-series-a/ https://techcrunch.com/2018/02/15/unsplash-simple-token-seri...
- Jackim 7y agoAs far as I can tell they're selling data. Their API requires hotlinking so they can track everything. They know exactly how often each picture is viewed and downloaded. Unsplash knows which images are successful and which ones are duds. I'm not sure who they're selling to, but I'm sure lots of folks would be interested.
- tjwds 7y agoThey also have extensive partnerships with certain brands — their homepage currently serves ads disguised as content from a handful of companies. (I say this as a user of and contributor to Unsplash)
- sergiotapia 7y agoSo ads? I thought it would be something a little more inspired.
- sdoering 7y agoYes. Of the first 7 images 3 are adverts. Then you can scroll down a long way on mobile without any more ads.
- kirillzubovsky 7y agoThat's pretty clever, actually. If you're just browsing, you're not going to hover over every image to see that it's sponsored, but a subtle product placement within that image might not be enough for your brain to recognize it as an ad bait. Not bad...
- amanzi 7y agoSo they must have some form of business model, right? All I can see on their website is that everything is for free, even their API which can be used for free without any limits or restrictions. And given the already huge Imgix costs, abusing or over-using their API to request and resize images would directly increase their overall costs. According to Crunchbase (https://www.crunchbase.com/organization/unsplash#section-funding-rounds https://www.crunchbase.com/organization/unsplash#section-fun...) they raised $10m last year, but they must be churning through $3m+ per year on staffing and hosting costs. I use and love the service but I can't see how it can last for more than a few more years.
- busymom0 7y agoThis reminds me of Soundcloud a few years ago when they were giving away API access for free for unlimited plays but then were almost close to bankrupt and pretty much shut down all API plays. I ended up shutting down my 3rd party app then but not sure what's their API limits now. Twitter recently made huge changes to their API limits too which impacted a lot of 3rd party apps. I hope Unsplash doesn't end up similar to those.
- HelloFellowDevs 7y agoOn the note of Twitter's restriction of API usage, Jack talked about it recently on the podcast Tales from the Crypt [0]. He doesn't give any specifics but attributes Twitters early struggle with uptime to unrestricted API usage [1]. [0] https://podcasts.apple.com/us/podcast/tales-from-the-crypt/id1292381204 https://podcasts.apple.com/us/podcast/tales-from-the-crypt/i... [1] https://overcast.fm/+KiHpBJHGo/15:30 https://overcast.fm/+KiHpBJHGo/15:30
- busymom0 7y agoIs there any particular reason they are still using Heroku instead of something like DO or AWS+S3? They can probably save a lot of money that way. Unless Heroku is offering them some good deal?
- ron22 7y agoThey tell you in the blog post. "We continue to use Heroku as our main web platform. Despite its premium cost over AWS, Azure, and Google Cloud, Heroku’s built-in deployment and configuration tools allow our team to move faster, more confidently, and more reliably. As we’ve detailed previously, the alternatives would undoubtably be cheaper on paper. But in reality, the increased simplicity and freedom offered by Heroku for a small, product-focused team is a major cost savings advantage."
- scarface74 7y agoI’m far from a Devops guy. But deploying to AWS using their native tools are Azure using Microsoft’s tools is dead simple.
- paulddraper 7y agoYeah something doesn't add up. I'll believe Heroku is easier, but can it be that much easier? For people spending piles of cash on their webscale?
- sudhirj 7y agoI do both at work, small sites on Heroku and big / compliance required ones on AWS, and the Heroku ones don’t need a devops team, while the AWS deployments do. Given the cost of building the devops teams and the time they spend, it is a big factor. Prefer to default to a Heroku unless there’s a clear cost benefit analysis that includes human and opportunity costs.
- scarface74 7y ago
- dna_polymerase 7y ago> We continue to use Heroku as our main web platform. Despite its premium cost over AWS, Azure, and Google Cloud, Heroku’s built-in deployment and configuration tools allow our team to move faster, more confidently, and more reliably. This is just bullshit, really. I'd buy this excuse from a service like Snapchat/Facebook in their early years, but unsplash is nothing that changes dramatically all the time. Every single point in their analysis just shows how they like to waste money. Imgix as the main image CDN? Start serving images using thumbor and a cheap CDN like BunnyCDN or KeyCDN. They could easily save 60% on their monthly bill. That would enable them to hire 1 or 2 engineers capable of maintaining a few dozen servers on AWS, DO or GCP.
- forrestthewoods 7y ago> They could easily save 60% on their monthly bill. That would enable them to hire 1 or 2 engineers capable of maintaining a few dozen servers on AWS, DO or GCP. If they reduce their bill and then hire engineers to do the same job they've gained nothing other than an increased headcount for no discernible benefit.
- manigandham 7y agoIf they're not really adding features then its the same staff. Not every change needs a new developer.
- indigodaddy 7y agoI think what the repliers missed from OC was that using that extra money for devops people will then allow even more savings after having consolidated down from Heroku to AWS/GCP/DO etc.
- bigiain 7y ago1 or 2 engineers is not enough to fully resource a 24x7x365 on-call roster. Sure, $42k/month seems like a lot of money, but proposing to hire 2 engineers to replace that seems naive to me. If you need to build/run/maintain a service that doesn't stay dark all weekend if something breaks at 6pm on a Friday, you need more than two people who're prepared to answer their phone on weekends.
- mharroun 7y agoI'm surprised that at that scale you are still not building some core functions internally. Though it makes sense, buy over build is almost always the best choice early stage. It's hard to slow down product velocity to fix things that work fine when costs start to add up. (Also it's easy to say "do you want to save 3k a month or deliver features x,y, and z" and let costs slide)
- ogirginc 7y agoDoes anyone know their bussiness model?
- idlewords 7y ago$99K/month in hosting costs and no visible source of revenue... I think this baby is ready for an IPO!
- danieltillett 7y agoThe losses are not large enough for an IPO. They need to scale at least 1000x to be IPO ready.
- blunte 7y agoNot until they have a few more rounds of financing, with each round increasing their valuation until they unicorn (verb). Then the baby is ready for IPO.
- OrgNet 7y agoit would be nice if each category would line up in this graph: https://cdn-images-1.medium.com/max/2400/1*Bvw2zcdE146WaXB7io3tZQ.png https://cdn-images-1.medium.com/max/2400/1*Bvw2zcdE146WaXB7i... ... kinda hard to read as-is
- selterdurst 7y agoIt’s interesting to see how so many people here focus on specific line items on their costs. What it comes down to is management focus. It’s a lot easier to switch a CDN than to lay off two engineers as needs change. I Do wonder why they don’t charge for API at all as even currency APIs charge and that is effectively public knowledge. Very interesting though as this is some serious scale indeed...
- systemBuilder 7y ago10(+) billion images served monthly for $100k = 1/1000th of a cent per image. This is not bad but I think Netflix is 2c a gigabyte; you are 10c (I'm assuming 100k per image).
- sudhirj 7y agoNetflix sends their own cache machines to tens, if not hundreds, of ISPs and interchanges all over the world, each of which have multiple terabytes of storage and act as cache nodes. I don’t think Unsplash is going to do that.
- reggieband 7y agoI'm curious what their price-per-GB is from Fastly/Imgix ... the monthly total doesn't really tell me the full story. At the last place I worked I was in charge of a team that ran our image service so I have a very solid understanding of the infrastructure around it and what resources we had in place to develop/monitor/support/etc. It wouldn't be hard to do a calculation and see if they are using money inefficiently. I will say that most people don't really have any idea of the cost of simply pushing bits. The majority of what I did was video distribution which gets into eye-watering costs pretty quickly. Without knowing a bit more detail on the precise volumes they are storing/moving I don't think it is fair to criticize.
- marknadal 7y agoFirst time hearing about unsplash, thank you!!! This is awesome for us OSS devs.
- tschellenbach 7y agoI work with these guys since they are customers of Stream. Super solid team, really respect how they have such focus in their execution.
- mychael 7y agoIt's cool that they're sharing this information, but using Heroku at this scale is wasteful and fiscally irresponsible.
- Ayesh 7y agoWow, these are some serious expenses! I run a service that does ~30 million requests a month, and I only spend $15 on it. It's an apples to orange comparison, but looking at their costs breakdown, I'm sure a couple engineers with a couple weeks can dramatically reduce their costs. For me, the biggest cost savings were at browser caching. With `immutable` and far future cache expiration dates, webp images, etc, I could pull of the 30m and growing stats without the server load going more than 0.40.
- kiran-rao 7y agoIf you're able to, could you share what service you run?
- jeandejean 7y agoThat's very tempting to do so but not knowing the exact details of the processing involved, you probably make a mistake on how easy it would be to optimize their costs. It could be true, but would mean engineering time not dedicated at features and likely more risk and more maintenance. It's often hard to admit but costs are not that easy to assess and trade of are tricky.
- umen 7y agohow do they monetize ? i can't find the monetization ...
- amolo 7y agoMaybe API partners
- jeandejean 7y agoThat's a great blog post. Very transparent. I'm especially glad he's willingly giving up cheaper alternative for having more value from his services. I just would have liked some more details on what it means technically... :-)
- GordonS 7y agoWell, it's transparent about their costs, but I would have liked a little transparency about how they actually make money
- jeandejean 7y agoVery true but I suppose we can't have it all, plus maybe it's sensitive information.
- sytelus 7y agoTLDR; $98K/mo. Year ago they were 10X smaller and used to cost only $17K.
- tschellenbach 7y agoSounds like they were focused on that 10x growth :D
- oliwarner 7y agoThe numbers being thrown around on this post are staggering. Not the bandwidth, storage or or the number of images —they're all pretty average for a small image host— but the casual tone that they're willing to throw money at services rather than engineering some quite simple problems. Seriously, I know I'm not in their shoes but I know what they do, I know popular options in these areas and I know they are literally throwing cash away by not trying to in-house any of this. The worst thing is they seem proud about their decisions; that their current situation has justified $100K/month expenses. I'm not the only person in here that thinks they could be delivering the same —even better— service on a fraction of their current budget, even after hiring. I think they should be ashamed.
- MobileVet 7y agoI hear you, but everything is a trade off. You and I would optimize the expenses more diligently, but they have chosen to focus 110% on the product. Could they do both, yes. Do they want to, no. The goal, imho, of a startup is doing what you love on your own terms. They clearly love the product and have optimized accordingly.
- speedplane 7y ago> The numbers being thrown around on this post are staggering. Not the bandwidth, storage or or the number of images ... but the casual tone that they're willing to throw money at services rather than engineering ... What I found interesting, is that just a sliver of the budget is spent on software rather than hardware. I don't think this is uncommon, few teams are willing to pay for software they can get for free or make themselves. It's a bit surprising (but explainable), that commodity hardware that everyone has access to dominates the budget.
- sudhirj 7y agoDoing a cost benefit analysis and deciding that using a service instead of hiring and paying humans is a perfectly rational decision. I’ve downvoted because I see a lot of vitriol like this and it’s exhausting to argue each time that these decisions are carefully considered and work well for the people making them. This is no different from saying people should be ashamed because they pay so much for food, then it’s much cheaper (almost free!) to grow it yourself in a pot.
- jopsen 7y agoCurious, anyone who knows what the business model is? The service seems pretty cool :)
- flywithdolp 7y agoI'm using unsplash for a while, They have the best quality free to use pictures on the internet. As someone who appreciate creative marketing i like how they generate millions of backlinks by asking you to link back to their site and credit the author. In few years they became among the most popular sites in their niche, a lot thanks to their creative growth approach.
- jon-wood 7y agoEveryone here seems to be screaming about how high their Heroku bill is, but I’m more interested in how they keep their New Relic bill so low. I find that New Relic are fine early on, and then the costs explode as you scale up and there’s no way to justify spending huge amounts of money on their platform.
- anonyxyz 7y agoHow does unsplash make money?