11 ms·
How Canva saves Amazon S3 costs
- andrewstuart 3y agoSeems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about USD$20 million to export their 230 petabytes. Let me know if my calculation is wrong.
- yashkadakia 3y agoYup, we just moved large chunks of our SaaS platform from the cloud to dedicated servers with Kubernetes serving as an internal cloud. Ended up reducing our cost to 30% (70% saved!). We realized that instead of using scaling for peaks, by having additional dedicated servers available full time - it still worked out significantly cheaper. We also moved a lot of our internal processing to run in specific windows of time where the expected load was low to maximize server utilization in those periods.
- ssss11 3y agoSorry what’s an “internal cloud”? Please tell me we haven’t renamed on-prem.
- simonjgreen 3y agoThe term "private cloud" has existed for easily 15 years, pretty much straight after Google coined the term "cloud computing". The current deifnition of "on-prem" actually changed after cloud computing reached mass adoption. Prior to then, on-prem referred to "in your office" vs. "in a datacenter". These days "on-prem" means "not in public cloud".
- Marazan 3y agoIt will mean compute resources that can be transparently shared amongst different things (which is where the K8s comes in) as opposed to boxes dedicated to a single service.
- _flux 3y agoYes, it's basically the same as "on prem", but instead just a bunch of individual (virtual) servers they are managed by a system that provides typical cloud APIs, such as OpenStack.
- monkeywork 3y agoon-prem services but running with the tooling that is typically used for cloud computing rather than bare-metal on-prem of old.
- onion2k 3y agoThere's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1), the price of building your own services, the price of software you need to buy, and so on. You only save money if the differential cost based on all the factors is lower. AWS is expensive for some things so it often does save some money, but not always, and if you haven't done a proper analysis you can't know.
- username_my1 3y agousually you're right but the cost mentioned here is specific to BLOB data storage ... a very specific problem that can be isolated and run on external services and save the company lots of money.
- CogitoCogito 3y ago^^^ The cloud vs. on-prem argument often seems to ignore the (enormous) middle-ground. Just because one portion of your architecture would do well to be run outside the cloud, doesn't mean you take out all the other parts you don't want to deal with yourself. Furthmore, "on-prem" might mean in your building, in someone else's building co-located and you rent space and control it, or in someone else's building where they deal with most hardware, etc. That said, maybe Canva has considered a non-AWS solution and decided against it. Or maybe they've gotten certain better deals from AWS. We can't really know for sure on the outside.
- maccard 3y agoMy team uses AWS for _everything_. There's three reasons for this on our team (and I've researched it for our use cases. - Consistency. Instead of saying "Oh look in AWS for this app, Azure for this one, and hetzner for this app, except it's test env is in AWS", it all just lives in AWS. It massively simplifies docs, onboarding, and reduces the amount of one-person specialised knowledge. - Engineering Costs. Similar to above, but in terms of engineering, there's less to know and understand. Instead of needing to know how the AWS load balancer routes/connects to a VM somewhere else, and how that VM gets it's blob-storage-data from azure, we only need to understand AWS concepts. - Vendor Lock In. Yeah, it's there. If we have a service that uses data from S3, there's egress costs from S3 to <other provider>, but not with EC2. We've consciously accepted this lock in for the time being. Now, we're a 50 person company so YMMV, but the above tradeoffs plus an "opinionated" setup in AWS (everything on ECS, logging to Cloudwatch, RDS for DB) drastically reduced the "ops" overhead on our side after the initial setup. If I started over, I'd make the same decisions again.
- zokier 3y agoMany aws services are hit or miss but s3 is just a marvel. Running geo-distributed 100s PB Ceph cluster with tiered storage isn't exactly trivial.
- littlestymaar 3y agoMillions of dollars per year isn't exactly a trivial amount of money either… Nobody said it's some super easy thing one could do on their free time, but given the scale it makes sense to build the knowledge in-house instead of getting vendor-locked-in at the most expensive service provider on the market.
- notyourwork 3y agoId argue this isn’t true until your spend surpasses $100MM. Lower amounts don’t justify the extra headache of on-prem.
- pc86 3y agoI know we're just arguing about amounts at this point but that's pushing $2M a week. It seems on on-prem would make financial sense long before that.
- shawabawa3 3y agoDepends on your requirements. If you need 11 nines durability S3 is probably still cheaper
- littlestymaar 3y agoI really hope that Canva's executive dont believe their product need such garantees…
- yencabulator 3y agoThere's also the hybrids. Only storing "unlikely to be accessed" data on S3 would likely remove a huge part of the AWS bill, while significantly limiting the size of the on-prem data set. Of all the possible related problems, object storage is the easiest.
- coldtea 3y agoIf you still make a profit and grow with public Cloud, and you had to risk time-to-marker, reliability, man-years, to make something on your own, then it's not that crazy. More of a "good problem to have" (and you can solve it when you need to).
- ramraj07 3y agoDo you know how much it’ll cost to store 230 petabytes per month on premises at the same redundancy and availability level? I have done such calculations for academia at a smaller scale and unless you literally want it to last decades it’s definitely not worth it to do your own infra on this.
- littlestymaar 3y agoI keep seeing that argument, but you, like the others, don't provide the figures you claim you got from your calculation. Also, it seams that it's 230PB in total, not per month, which fits in an apartment-sized server room (you want to have several of such places, but it's not that big).
- jabart 3y agoWasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space. https://wasabi.com/blog/on-premises-vs-cloud-storage/ https://wasabi.com/blog/on-premises-vs-cloud-storage/
- hinkley 3y ago140 racks is a quite a lot of tonnage of AC as well. And probably hot and cold rows to make it function.
- jabart 3y agoHot/cold was factored in not the AC or power needs. I don't know many places you could just casually ask for 100+ racks. Maybe our midwest datacenters are just smaller than what is on the coast.
- 3y ago
- xarope 3y agolet me give some hypothetical; let's say Canva decides to cut cost by cutting the amount of storage users could have (e.g. quota, or lifetime): - if they had a sunk cost CAPEX of 100 PB of storage, all they could do is sell the storage servers for a fraction of the original cost (and folks like me snapping up pre-owned server equipment for said fraction of cost for our home labs). - on the other, since S3 is OPEX (not sure if they are locked in to 1 year billing cycles? but still better than 3-5 year CAPEX runs), they could cut the cost of that storage "immediately". Similarly, if they needed to expand, CAPEX means placing an order for the storage, waiting for delivery, racking, burning in etc, versus just paying almost "instantly" for more storage. So long as they are still earning money, why wouldn't they stick to this OPEX model? Not saying this is good or bad, it depends on your business model, ultimately.
- re-thc 3y ago> So long as they are still earning money, why wouldn't they stick to this OPEX model? The suggestion was to move to a different / cheaper provider. You can stick to OPEX.
- ToJans 3y agoI'd go for a hybrid approach: new data is stored on private cloud, and old data would be gradually migrated or phased out.
- re-thc 3y ago> Cloudflare R2 charges zero egress fees. The scarier part is Canva actually does use Cloudflare CDN, so it's not so much the cost of adding a new provider. It's just that the CDN team that manages the Cloudflare account is likely a different team :) Too much isolation, silos and politics...
- sofixa 3y agoThe much more likely answer is that R2 is too new compared to when they started (announced in 2021) and switching from S3 to it is probably not trivial at Canva's scale.
- Sai_ 3y agoThey could optimize this by only egressing they most recently used petabytes of data for some value of “recently”.
- littlestymaar 3y ago“Nobody ever got fired for choosing I̵B̶M̶ Amazon”
- whycombagator 3y agoWhat happens when cloudflare changes the price?
- ttymck 3y agoWhy do they need to get everything out? Some not-insignificant percentage of the content is surely archived/unused/unneeded assets. From customers who will never log in again or projects which are long over. Export the most recent 10% and put anything new on <new storage>
- notyourwork 3y agoAbsolutely agree. There are more nuanced strategies to migrate off a cloud provider. Could even enable a soft export for idle accounts, leave it where it lie and move it when customer re-engages.
- pc86 3y agoI would think you could set up a migration process where existing things are loaded off of S3, and as soon as you make any changes, you save it to $NEW_STORAGE and delete it off of S3. Eliminates the need to egress anything, lowers your S3 bill over time, and as a user it'd be pretty hard to log in after a year and be upset that your FB cover image is no longer there, especially on the free tier.
- avereveard 3y agowhat's the durability of R2?
- re-thc 3y agoeleven 9's apparently
- VHRanger 3y agoHow would you measure that realistically? How durable is Backblaze or Azure? Cloudflare has been reliable on storage adjacent uses (caching) for a long time. Presumably you're more likely to be the cause of errors than R2 is by a few orders of magnitude.
- avereveard 3y agowell you don't measure it, you take it from their sla
- VHRanger 3y agoWhat I mean is that if you have a claim for eleven 9's of durability for a product that's 3 years old you have to take their word for it. You can't really check the SLA as being realistic
- speed_spread 3y agoNitpick: It's "you can _check out_ anytime you want but you can never leave"; the dissociation between checking out and actually leaving is what makes the line. Still a good reference for the S3 situation!
- electroly 3y agoTip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of spend you'd be crazy to pay the full $0.09/GB. Edit: Note also that Snowball egress is $0.03/GB. Slightly higher egress, much lower setup cost. You'll have to do the math but they're both clearly attractive options vs. full price $0.09/GB egress.
- fakedang 3y agoPlot twist : Canva blogged this so someone could tell them some solution like this, without them having to hire the guy. :)
- tough 3y agoThere's this law on the internet or SO or whatever where if you put the wrong answer in a post. Someone else in the comments will correct you with the correct answer. This in effect.
- lbotos 3y agohttps://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham's_Law%22 https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham'...
- leon1717 3y agoCunningham is cunning!!!
- grogenaut 3y agoThis isn't just the internet. I've used this at work several times to break stalemates. I've come into many stalled projects where people are arguing over trivial details or blue skying on requirements. 1-2 hours and a 7 page design doc later I'll have everyone review what I know is an 80% answer. Pretty much every time they'll all start attacking my design, pointing out minor issues and then I have them. They've agreed to my overall design and are into details. Same if someone is calling out all of the general problems with solving a problem but not providing answers where you can't get them to bite.
- xwowsersx 3y agoIt's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. If they wanted to move off of S3, they'd rely on the same type of analysis they present in this post + perhaps creating a mechanism whereby they permanently move an asset off of S3 if/when accessed.
- ignoramous 3y ago> It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. See the Docker x Cloudflare case study where improving cache-hit ratio by 2% (by moving to R2) decreased S3 egress fee by 66%: https://www.cloudflare.com/en/case-studies/docker/ https://www.cloudflare.com/en/case-studies/docker/
- shadowtree 3y agoMost businesses treat cloud like a utility, not tech. Don't want to generate their own power, deal with sewage ... and compute. Which in 99.9% of the cases is the right decision. Sorry for all you infra techies, but that's the way tech matures.
- betaby 3y agoInfra techies are fine. Likewise 'grid is mature' and yet most of buildings with floor space over NN m2 have team of electricians, being them direct hire or contractors. Same for the 'cloud' - 90%+ of the world deployed 'compute' is not cloud.
- shadowtree 3y agoWork for a 30bn market cap company, all our stuff is built on AWS. Offices all around the globe. Not a single electrician on payroll. YMMV.
- betaby 3y agoYou may call your company arrangement `electricityless` even! Like `serverless` - we don't use servers.
- mschuster91 3y ago> Not a single electrician on payroll. Directly, yes maybe. Many companies don't have in-house experts (and get shafted by vendors as a result, as they're lacking the competence required to assess the quality of work). But indirectly, they all have. In Germany, at least, you're required to have electrical appliances inspected by a licensed professional (an electrician or otherwise qualified person) at least every two years - most companies opt for using dedicated external companies, but you can also train someone for that task. On top of that come all the electricians hired by the landlords - the ones dealing with complaints and remodels, or servicing lifts, escalators, HVAC, datacenters, ...
- shadowtree 3y ago
- dehrmann 3y agoThe real story isn't how expensive it is to leave, it's how easy AWS makes it to spend. It used to be that you'd see how many servers a project needs and have to beg someone for them, and then only get half of what you wanted. With AWS, it's much easier to launch something new, but there isn't the same amount of scrutiny on spend.
- choppaface 3y ago“we store more than 230 Petabytes of data in Amazon S3, with our single largest S3 bucket coming in at a whopping 45 Petabytes” How did they get so much data?! They say they only have 75 million stock photos. They must have tens of petabytes that are just dark.
- jedberg 3y ago> At list price of 9 cents per GB it would cost Canva about USD$20 million to export their 230 petabytes. Let me know if my calculation is wrong. If you're doing a one-time export you can use a Snowball, which would cost about $70K for 230PB, assuming you had somewhere to load it off to once you got the data so you could give the device back.
- alecco 3y agoAnd now it's VC-fashionable to have S3 as "object store" for "serverless" databases. "offload to S3!"
- vkaku 3y agoCloudflare could build a product that not only caches, but transfers S3 to R2 on usage. - Since non transactional data is likely to be on GitHub or somewhere, it should be easy to re-hydrate that on R2. - Once enough time passes, there could be a sunset on unused data, or a strategy planned to 1-time exit write-off for S3 with those assets not on R2. Standard migration procedure.
- rat9988 3y agoReal question, how to scale a business to petabytes of data without cloud? How is it done in practice?
- acchow 3y ago10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. There's tons of OSS to manage all layers of this stuff now.
- colechristensen 3y agoYou can have a few petabytes in a single rack. 50-ish drives in a single 4U server is done.
- icedchai 3y agoWhat about single points of failure? Replication? High availability? Disaster recovery? Backup / snapshots/ versioning? This is one of the benefits of the cloud.
- fh973 3y agoUse a distributed file system or object store with what you need, like Quobyte or Ceph.
- colechristensen 3y agoPlenty of tools for that on premises. Are you actually asking?
- icedchai 3y agoNo, I'm just mentioning these things to show that the decision is more complex than a "4U server" vs "S3." S3/AWS gives you a ton of stuff you'd have to implement yourself. Plus you need to hire more people to manage it.
- jokethrowaway 3y agoCanva is a case where AWS made sense for the founders. The founders anyway made billions and who cares if you give a few hundreds millions to Jeff and have to hire expensive AWS consultants if you can avoid hiring a few ops people that actually know how to operate a server /s More seriously, I think the sweet spot is to buy 2x o 3x what you expect to need but do it off the cloud. At Canva's scale you'll spend hundreds of thousands, save literal millions and have way more headroom and stability than with crappy auctioned instances on AWS.
- iLoveOncall 3y agoAt Canva's scale you'll spend hundreds of thousands a year, just hiring one of the engineers working on your home-grown solution. Multiple this by 50 or more just for the staff, before talking about hardware, cost of delivery speed, cost of bugs, etc.
- andrewstuart 3y agoInstead you can spend millions on paying top market dollar to AWS experts trying to work out your giant pile of spaghetti cloud infrastructure. Desperately trying to work out if they can delete this resource or that because who the hell knows which bit of critical code is using it. Or spending days trying to manage a chewing gum ball of IAM policies.
- weird-eye-issue 3y agoWouldn't you just be swapping those problems with similar problems if you switched to on prem? > Desperately trying to work out if they can delete this resource or that because who the hell knows which bit of critical code is using it. How is this unique to the cloud?
- iLoveOncall 3y ago> Instead you can spend millions on paying top market dollar to AWS experts trying to work out your giant pile of spaghetti cloud infrastructure. The point is that you will spend millions anyway. Might as well have something better than whatever home-grown dumpster fire your team will come up with. You also seem very ignorant of how AWS works. I've never had to wonder if I could delete a resource or not, because all our resources are created via CloudFormation. Same for IAM policies.
- orangepanda 3y agoOfftopic — is it some cardinal sin to include a link to the home page from a documentation? I dont know what canva does, if I stumble on a documentation page first, should I not be allowed to see the product's landing page at some point? This is so prevalent, it has to be a conscious choice.
- SerCe 3y agoThank you for raising this! I'll take a look at how we can make the discovery more straightforward.
- skilled 3y agoWhat? There are multiple homepage links on this page, including the Logo, Home from navigation menu, as well as the first word in the article is a link out to Canva's homepage.
- orangepanda 3y agoThe logo and "home" do not lead to the product page - they lead to developer docs home page. Did not think to try the first link though.
- Solvency 3y agoMy favorite is when you click on some landing page made by a company.... and there is zero way to get to the rest of the site. Marketing-centric landing pages often completely strip out their global navigation for no reason.
- motoxpro 3y agoA lot of people in this thread should start consultancies that transition companies off the cloud. Apparently, it would be easy for them to save tens of millions, in turn making millions for the consultant.
- vasco 3y agoCouldn't agree more. One only says this if they don't have a good understanding of the differences between capex & opex, the amount of money spent on tech labor & compliance projects, as well as the level of assurances AWS give you regarding durability of data. All of this in 2023 when we have almost 20 years of data showing more and more business moving to the cloud and staying in the cloud. I think anti-cloud sentiment aligns with hacker mentality on decentralization=good (which it is), anti big corporation feelings (boo amazon) and so it leads people in these threads to make emotional arguments over something that is clearly going in the other direction. It's an excellent example of confirmation bias, trying to look for any and all justification to say the cloud is worse, when the majority of businesses have decided otherwise.
- charcircuit 3y agoHave you even looked at S3 bandwidth pricing? I don't get how anyone using S3 for storing media or binaries can justify that outrageous price unless every download is connected to something making you money.
- Quarrel 3y agoWell, step 1, have petabytes of data in S3, then don't pay the sticker price. Step 2, write articles like this as part of your newly contracted "engagement" with Amazon to help justify the lower price.
- charcircuit 3y ago>step 1, have petabytes of data in S3, then don't pay the sticker price. So pay Amazon a bunch for the privilege to have lower prices. How about just starting off somewhere where you don't have to commit to paying a lot of money to get a reasonable price?
- jaza 3y agoI'm really surprised how much anti cloud opinion is being voiced here (but maybe I should have known better, after all these years on HN). Even if you don't use any other cloud tech in your stack, my advice is, use S3 (or equivalent)! That level of durability, availability, and scalability, is not trivial to pull off, and is best made somebody else's problem. Even when you're as big as Canva.
- withinboredom 3y agoI'm a big fan of Garage[1], which is a dead-simple S3 drop-in that you can host on your own drives. It's designed for consumer hardware with shitty internet in-between nodes. [1]:https://garagehq.deuxfleurs.fr/ https://garagehq.deuxfleurs.fr/
- paxys 3y agoAll the marketing pages in the world mean nothing until they can show who currently uses them and at what scale.
- withinboredom 3y agoWell, someone has to make that experiment and see if it’s right for them, right? It’s only a few years old, and most bigger companies are already sold on whatever they’re using. Smaller companies you’ve never heard of will be the first drivers, most likely. FWIW, I have an EU region cluster (3 nodes per inner-region in the North, West and South). It’s all one logical region to S3 libraries and holds nearly half a terabyte. It’s probably one of the most stable parts of my infrastructure. I don’t know (or need to know) what the replication lag is between regions, but IIRC, there’s some configuration of consistency.
- tansan 3y agoWhoa! this is the first time I have seen deuxfleurs. Do french people use this over github?
- 3y ago
- xnx 3y agoArticles like this are a great example of how you can be a hero for fixing a problem, but get no recognition for preventing one.
- soderfoo 3y agoSurely they negotiated a private pricing agreement given the volume of data/usage...one would hope. Having been a part of these negotiations for services other than S3, DTO discounts can be generous if you have a good forecast (3-5 years) on your data egress.
- StratusBen 3y agoThis is an interesting post...though I'm surprised to see this kind of optimization occur so late stage/as large as Canva. There's quite a few anti-cloud opinions in the comments but for anyone who is hosted on S3 and would like this same level of automated analysis for cost savings on S3 (and other AWS services), we basically profile for this out-of-the-box on Vantage and it's free to get an optimization check to see savings: https://www.vantage.sh/ https://www.vantage.sh/ Not meant to be as shameless of a plug, but seems very relevant to the topic at hand for onlookers.
- Solvency 3y ago"I'm surprised to see this kind of optimization occur so late" Every single day I read the trope on HN that "premature optimization is the root of all evil". So, these guys literally followed HN's creed.
- lowbloodsugar 3y agoCanva is ten years old. They grew to 1 million users in two years. Doing this work 8 years ago wouldn't have been "premature optimization". More likely, this is happening now simply because the financial environment has changed and companies are now giving increased weight to projects that save money over those that add features.
- jgalt212 3y ago> 100 million monthly active users Oh, my. That's a lot. What does Figma? 300?
- jgalt212 3y ago> Canva now has users across 190 countries, who design in over 100 languages thanks to some amazing localization work done by our Internationalisation team. How may of those countries are profitable? My sense is Internationlization is a relatively fixed cost, but how does customer support in Burma, for example, not trump any realizable revenues there?
- Dunedan 3y agoSomething which is often overlooked is that transitioning objects in S3 to another storage class using lifecycle policies is a one-way operation. If you later notice that the chosen storage class doesn't fit or your use case changes, you can only go "forward" using lifecycle policies and not "back" to e.g. "S3 Standard" [1]. Going "back" is therefore a quite costly and cumbersome operation, as you have to apply the change of the storage class to each object individually. [1]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy...
- MagicMoonlight 3y agoThere is no way they need 230 PB saved. That's javascript-tier design in practice. If you can put almost all of it into glacial storage then that's because you don't actually need to store it.
- encodedrose 3y agoIf you're at a point of spending $1.6M to transition objects from one class to another and spending millions per month in storage costs -- you need to have a real conversation about if storing your data in a vendor-locked cloud is the right path forward. S3 isn't the only option, MinIO + dense storage is one viable option if your spend is high enough to justify running MinIO. Backblaze is another. List price of Backblaze: $0.005/GB - Base Storage $0.01/GB - Egress (Use a data partner to bring this down to $0) $0 - PUT API Requests $0.004/10,000 - GET API Requests No delete penalties a/k/a minimum storage List price of Glacier Instant Retrieval: $0.004/GB - Base Storage $0.09/GB - Egress $0.02/1,000 - PUT API Requests $0.01/1,000 - GET API Requests Billed for 90 days of storage
- DarthNebo 3y agoYeah no, R2 is a way better choice than being vendor locked like this when burning millions