7 ms·

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about USD$20 million to export their 230 petabytes. Let me know if my calculation is wrong.

yashkadakia 3y ago
Yup, we just moved large chunks of our SaaS platform from the cloud to dedicated servers with Kubernetes serving as an internal cloud. Ended up reducing our cost to 30% (70% saved!). We realized that instead of using scaling for peaks, by having additional dedicated servers available full time - it still worked out significantly cheaper. We also moved a lot of our internal processing to run in specific windows of time where the expected load was low to maximize server utilization in those periods.
- ssss11 3y ago
  Sorry what’s an “internal cloud”? Please tell me we haven’t renamed on-prem.
  - simonjgreen 3y ago
    The term "private cloud" has existed for easily 15 years, pretty much straight after Google coined the term "cloud computing". The current deifnition of "on-prem" actually changed after cloud computing reached mass adoption. Prior to then, on-prem referred to "in your office" vs. "in a datacenter". These days "on-prem" means "not in public cloud".
  - Marazan 3y ago
    It will mean compute resources that can be transparently shared amongst different things (which is where the K8s comes in) as opposed to boxes dedicated to a single service.
  - _flux 3y ago
    Yes, it's basically the same as "on prem", but instead just a bunch of individual (virtual) servers they are managed by a system that provides typical cloud APIs, such as OpenStack.
  - monkeywork 3y ago
    on-prem services but running with the tooling that is typically used for cloud computing rather than bare-metal on-prem of old.
onion2k 3y ago
There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1), the price of building your own services, the price of software you need to buy, and so on. You only save money if the differential cost based on all the factors is lower. AWS is expensive for some things so it often does save some money, but not always, and if you haven't done a proper analysis you can't know.
- username_my1 3y ago
  usually you're right but the cost mentioned here is specific to BLOB data storage ... a very specific problem that can be isolated and run on external services and save the company lots of money.
  - CogitoCogito 3y ago
    ^^^ The cloud vs. on-prem argument often seems to ignore the (enormous) middle-ground. Just because one portion of your architecture would do well to be run outside the cloud, doesn't mean you take out all the other parts you don't want to deal with yourself. Furthmore, "on-prem" might mean in your building, in someone else's building co-located and you rent space and control it, or in someone else's building where they deal with most hardware, etc. That said, maybe Canva has considered a non-AWS solution and decided against it. Or maybe they've gotten certain better deals from AWS. We can't really know for sure on the outside.
    - maccard 3y ago
      My team uses AWS for _everything_. There's three reasons for this on our team (and I've researched it for our use cases. - Consistency. Instead of saying "Oh look in AWS for this app, Azure for this one, and hetzner for this app, except it's test env is in AWS", it all just lives in AWS. It massively simplifies docs, onboarding, and reduces the amount of one-person specialised knowledge. - Engineering Costs. Similar to above, but in terms of engineering, there's less to know and understand. Instead of needing to know how the AWS load balancer routes/connects to a VM somewhere else, and how that VM gets it's blob-storage-data from azure, we only need to understand AWS concepts. - Vendor Lock In. Yeah, it's there. If we have a service that uses data from S3, there's egress costs from S3 to <other provider>, but not with EC2. We've consciously accepted this lock in for the time being. Now, we're a 50 person company so YMMV, but the above tradeoffs plus an "opinionated" setup in AWS (everything on ECS, logging to Cloudwatch, RDS for DB) drastically reduced the "ops" overhead on our side after the initial setup. If I started over, I'd make the same decisions again.
zokier 3y ago
Many aws services are hit or miss but s3 is just a marvel. Running geo-distributed 100s PB Ceph cluster with tiered storage isn't exactly trivial.
- littlestymaar 3y ago
  Millions of dollars per year isn't exactly a trivial amount of money either… Nobody said it's some super easy thing one could do on their free time, but given the scale it makes sense to build the knowledge in-house instead of getting vendor-locked-in at the most expensive service provider on the market.
  - notyourwork 3y ago
    Id argue this isn’t true until your spend surpasses $100MM. Lower amounts don’t justify the extra headache of on-prem.
    - pc86 3y ago
      I know we're just arguing about amounts at this point but that's pushing $2M a week. It seems on on-prem would make financial sense long before that.
      - shawabawa3 3y ago
        Depends on your requirements. If you need 11 nines durability S3 is probably still cheaper
        littlestymaar 3y ago
        I really hope that Canva's executive dont believe their product need such garantees…
        yencabulator 3y ago
        There's also the hybrids. Only storing "unlikely to be accessed" data on S3 would likely remove a huge part of the AWS bill, while significantly limiting the size of the on-prem data set. Of all the possible related problems, object storage is the easiest.
coldtea 3y ago
If you still make a profit and grow with public Cloud, and you had to risk time-to-marker, reliability, man-years, to make something on your own, then it's not that crazy. More of a "good problem to have" (and you can solve it when you need to).
ramraj07 3y ago
Do you know how much it’ll cost to store 230 petabytes per month on premises at the same redundancy and availability level? I have done such calculations for academia at a smaller scale and unless you literally want it to last decades it’s definitely not worth it to do your own infra on this.
- littlestymaar 3y ago
  I keep seeing that argument, but you, like the others, don't provide the figures you claim you got from your calculation. Also, it seams that it's 230PB in total, not per month, which fits in an apartment-sized server room (you want to have several of such places, but it's not that big).
  - jabart 3y ago
    Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space. https://wasabi.com/blog/on-premises-vs-cloud-storage/ https://wasabi.com/blog/on-premises-vs-cloud-storage/
    - hinkley 3y ago
      140 racks is a quite a lot of tonnage of AC as well. And probably hot and cold rows to make it function.
      - jabart 3y ago
        Hot/cold was factored in not the AC or power needs. I don't know many places you could just casually ask for 100+ racks. Maybe our midwest datacenters are just smaller than what is on the coast.
        3y ago
xarope 3y ago
let me give some hypothetical; let's say Canva decides to cut cost by cutting the amount of storage users could have (e.g. quota, or lifetime): - if they had a sunk cost CAPEX of 100 PB of storage, all they could do is sell the storage servers for a fraction of the original cost (and folks like me snapping up pre-owned server equipment for said fraction of cost for our home labs). - on the other, since S3 is OPEX (not sure if they are locked in to 1 year billing cycles? but still better than 3-5 year CAPEX runs), they could cut the cost of that storage "immediately". Similarly, if they needed to expand, CAPEX means placing an order for the storage, waiting for delivery, racking, burning in etc, versus just paying almost "instantly" for more storage. So long as they are still earning money, why wouldn't they stick to this OPEX model? Not saying this is good or bad, it depends on your business model, ultimately.
- re-thc 3y ago
  > So long as they are still earning money, why wouldn't they stick to this OPEX model? The suggestion was to move to a different / cheaper provider. You can stick to OPEX.
ToJans 3y ago
I'd go for a hybrid approach: new data is stored on private cloud, and old data would be gradually migrated or phased out.
re-thc 3y ago
> Cloudflare R2 charges zero egress fees. The scarier part is Canva actually does use Cloudflare CDN, so it's not so much the cost of adding a new provider. It's just that the CDN team that manages the Cloudflare account is likely a different team :) Too much isolation, silos and politics...
- sofixa 3y ago
  The much more likely answer is that R2 is too new compared to when they started (announced in 2021) and switching from S3 to it is probably not trivial at Canva's scale.
Sai_ 3y ago
They could optimize this by only egressing they most recently used petabytes of data for some value of “recently”.
littlestymaar 3y ago
“Nobody ever got fired for choosing I̵B̶M̶ Amazon”
whycombagator 3y ago
What happens when cloudflare changes the price?
ttymck 3y ago
Why do they need to get everything out? Some not-insignificant percentage of the content is surely archived/unused/unneeded assets. From customers who will never log in again or projects which are long over. Export the most recent 10% and put anything new on <new storage>
- notyourwork 3y ago
  Absolutely agree. There are more nuanced strategies to migrate off a cloud provider. Could even enable a soft export for idle accounts, leave it where it lie and move it when customer re-engages.
- pc86 3y ago
  I would think you could set up a migration process where existing things are loaded off of S3, and as soon as you make any changes, you save it to $NEW_STORAGE and delete it off of S3. Eliminates the need to egress anything, lowers your S3 bill over time, and as a user it'd be pretty hard to log in after a year and be upset that your FB cover image is no longer there, especially on the free tier.
avereveard 3y ago
what's the durability of R2?
- re-thc 3y ago
  eleven 9's apparently
- VHRanger 3y ago
  How would you measure that realistically? How durable is Backblaze or Azure? Cloudflare has been reliable on storage adjacent uses (caching) for a long time. Presumably you're more likely to be the cause of errors than R2 is by a few orders of magnitude.
  - avereveard 3y ago
    well you don't measure it, you take it from their sla
    - VHRanger 3y ago
      What I mean is that if you have a claim for eleven 9's of durability for a product that's 3 years old you have to take their word for it. You can't really check the SLA as being realistic
speed_spread 3y ago
Nitpick: It's "you can _check out_ anytime you want but you can never leave"; the dissociation between checking out and actually leaving is what makes the line. Still a good reference for the S3 situation!
electroly 3y ago
Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of spend you'd be crazy to pay the full $0.09/GB. Edit: Note also that Snowball egress is $0.03/GB. Slightly higher egress, much lower setup cost. You'll have to do the math but they're both clearly attractive options vs. full price $0.09/GB egress.
- fakedang 3y ago
  Plot twist : Canva blogged this so someone could tell them some solution like this, without them having to hire the guy. :)
  - tough 3y ago
    There's this law on the internet or SO or whatever where if you put the wrong answer in a post. Someone else in the comments will correct you with the correct answer. This in effect.
    - lbotos 3y ago
      https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham's_Law%22 https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham'...
      - leon1717 3y ago
        Cunningham is cunning!!!
    - grogenaut 3y ago
      This isn't just the internet. I've used this at work several times to break stalemates. I've come into many stalled projects where people are arguing over trivial details or blue skying on requirements. 1-2 hours and a 7 page design doc later I'll have everyone review what I know is an 80% answer. Pretty much every time they'll all start attacking my design, pointing out minor issues and then I have them. They've agreed to my overall design and are into details. Same if someone is calling out all of the general problems with solving a problem but not providing answers where you can't get them to bite.
xwowsersx 3y ago
It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. If they wanted to move off of S3, they'd rely on the same type of analysis they present in this post + perhaps creating a mechanism whereby they permanently move an asset off of S3 if/when accessed.
- ignoramous 3y ago
  > It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. See the Docker x Cloudflare case study where improving cache-hit ratio by 2% (by moving to R2) decreased S3 egress fee by 66%: https://www.cloudflare.com/en/case-studies/docker/ https://www.cloudflare.com/en/case-studies/docker/
shadowtree 3y ago
Most businesses treat cloud like a utility, not tech. Don't want to generate their own power, deal with sewage ... and compute. Which in 99.9% of the cases is the right decision. Sorry for all you infra techies, but that's the way tech matures.
- betaby 3y ago
  Infra techies are fine. Likewise 'grid is mature' and yet most of buildings with floor space over NN m2 have team of electricians, being them direct hire or contractors. Same for the 'cloud' - 90%+ of the world deployed 'compute' is not cloud.
  - shadowtree 3y ago
    Work for a 30bn market cap company, all our stuff is built on AWS. Offices all around the globe. Not a single electrician on payroll. YMMV.
    - betaby 3y ago
      You may call your company arrangement `electricityless` even! Like `serverless` - we don't use servers.
    - mschuster91 3y ago
      > Not a single electrician on payroll. Directly, yes maybe. Many companies don't have in-house experts (and get shafted by vendors as a result, as they're lacking the competence required to assess the quality of work). But indirectly, they all have. In Germany, at least, you're required to have electrical appliances inspected by a licensed professional (an electrician or otherwise qualified person) at least every two years - most companies opt for using dedicated external companies, but you can also train someone for that task. On top of that come all the electricians hired by the landlords - the ones dealing with complaints and remodels, or servicing lifts, escalators, HVAC, datacenters, ...
      - shadowtree 3y ago
dehrmann 3y ago
The real story isn't how expensive it is to leave, it's how easy AWS makes it to spend. It used to be that you'd see how many servers a project needs and have to beg someone for them, and then only get half of what you wanted. With AWS, it's much easier to launch something new, but there isn't the same amount of scrutiny on spend.
choppaface 3y ago
“we store more than 230 Petabytes of data in Amazon S3, with our single largest S3 bucket coming in at a whopping 45 Petabytes” How did they get so much data?! They say they only have 75 million stock photos. They must have tens of petabytes that are just dark.
jedberg 3y ago
> At list price of 9 cents per GB it would cost Canva about USD$20 million to export their 230 petabytes. Let me know if my calculation is wrong. If you're doing a one-time export you can use a Snowball, which would cost about $70K for 230PB, assuming you had somewhere to load it off to once you got the data so you could give the device back.
alecco 3y ago
And now it's VC-fashionable to have S3 as "object store" for "serverless" databases. "offload to S3!"
vkaku 3y ago
Cloudflare could build a product that not only caches, but transfers S3 to R2 on usage. - Since non transactional data is likely to be on GitHub or somewhere, it should be easy to re-hydrate that on R2. - Once enough time passes, there could be a sunset on unused data, or a strategy planned to 1-time exit write-off for S3 with those assets not on R2. Standard migration procedure.

yashkadakia 3y ago

Yup, we just moved large chunks of our SaaS platform from the cloud to dedicated servers with Kubernetes serving as an internal cloud. Ended up reducing our cost to 30% (70% saved!). We realized that instead of using scaling for peaks, by having additional dedicated servers available full time - it still worked out significantly cheaper. We also moved a lot of our internal processing to run in specific windows of time where the expected load was low to maximize server utilization in those periods.

ssss11 3y ago
Sorry what’s an “internal cloud”? Please tell me we haven’t renamed on-prem.
- simonjgreen 3y ago
  The term "private cloud" has existed for easily 15 years, pretty much straight after Google coined the term "cloud computing". The current deifnition of "on-prem" actually changed after cloud computing reached mass adoption. Prior to then, on-prem referred to "in your office" vs. "in a datacenter". These days "on-prem" means "not in public cloud".
- Marazan 3y ago
  It will mean compute resources that can be transparently shared amongst different things (which is where the K8s comes in) as opposed to boxes dedicated to a single service.
- _flux 3y ago
  Yes, it's basically the same as "on prem", but instead just a bunch of individual (virtual) servers they are managed by a system that provides typical cloud APIs, such as OpenStack.
- monkeywork 3y ago
  on-prem services but running with the tooling that is typically used for cloud computing rather than bare-metal on-prem of old.

onion2k 3y ago

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then you have to factor in things like the cost of downtime (on both sides of the equation, especially if you use us-east-1), the price of building your own services, the price of software you need to buy, and so on. You only save money if the differential cost based on all the factors is lower. AWS is expensive for some things so it often does save some money, but not always, and if you haven't done a proper analysis you can't know.

username_my1 3y ago
usually you're right but the cost mentioned here is specific to BLOB data storage ... a very specific problem that can be isolated and run on external services and save the company lots of money.
- CogitoCogito 3y ago
  ^^^ The cloud vs. on-prem argument often seems to ignore the (enormous) middle-ground. Just because one portion of your architecture would do well to be run outside the cloud, doesn't mean you take out all the other parts you don't want to deal with yourself. Furthmore, "on-prem" might mean in your building, in someone else's building co-located and you rent space and control it, or in someone else's building where they deal with most hardware, etc. That said, maybe Canva has considered a non-AWS solution and decided against it. Or maybe they've gotten certain better deals from AWS. We can't really know for sure on the outside.
  - maccard 3y ago
    My team uses AWS for _everything_. There's three reasons for this on our team (and I've researched it for our use cases. - Consistency. Instead of saying "Oh look in AWS for this app, Azure for this one, and hetzner for this app, except it's test env is in AWS", it all just lives in AWS. It massively simplifies docs, onboarding, and reduces the amount of one-person specialised knowledge. - Engineering Costs. Similar to above, but in terms of engineering, there's less to know and understand. Instead of needing to know how the AWS load balancer routes/connects to a VM somewhere else, and how that VM gets it's blob-storage-data from azure, we only need to understand AWS concepts. - Vendor Lock In. Yeah, it's there. If we have a service that uses data from S3, there's egress costs from S3 to <other provider>, but not with EC2. We've consciously accepted this lock in for the time being. Now, we're a 50 person company so YMMV, but the above tradeoffs plus an "opinionated" setup in AWS (everything on ECS, logging to Cloudwatch, RDS for DB) drastically reduced the "ops" overhead on our side after the initial setup. If I started over, I'd make the same decisions again.

zokier 3y ago

Many aws services are hit or miss but s3 is just a marvel. Running geo-distributed 100s PB Ceph cluster with tiered storage isn't exactly trivial.

littlestymaar 3y ago
Millions of dollars per year isn't exactly a trivial amount of money either… Nobody said it's some super easy thing one could do on their free time, but given the scale it makes sense to build the knowledge in-house instead of getting vendor-locked-in at the most expensive service provider on the market.
- notyourwork 3y ago
  Id argue this isn’t true until your spend surpasses $100MM. Lower amounts don’t justify the extra headache of on-prem.
  - pc86 3y ago
    I know we're just arguing about amounts at this point but that's pushing $2M a week. It seems on on-prem would make financial sense long before that.
    - shawabawa3 3y ago
      Depends on your requirements. If you need 11 nines durability S3 is probably still cheaper
      - littlestymaar 3y ago
        I really hope that Canva's executive dont believe their product need such garantees…
      - yencabulator 3y ago
        There's also the hybrids. Only storing "unlikely to be accessed" data on S3 would likely remove a huge part of the AWS bill, while significantly limiting the size of the on-prem data set. Of all the possible related problems, object storage is the easiest.

coldtea 3y ago

If you still make a profit and grow with public Cloud, and you had to risk time-to-marker, reliability, man-years, to make something on your own, then it's not that crazy. More of a "good problem to have" (and you can solve it when you need to).

ramraj07 3y ago

Do you know how much it’ll cost to store 230 petabytes per month on premises at the same redundancy and availability level? I have done such calculations for academia at a smaller scale and unless you literally want it to last decades it’s definitely not worth it to do your own infra on this.

littlestymaar 3y ago
I keep seeing that argument, but you, like the others, don't provide the figures you claim you got from your calculation. Also, it seams that it's 230PB in total, not per month, which fits in an apartment-sized server room (you want to have several of such places, but it's not that big).
- jabart 3y ago
  Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space. https://wasabi.com/blog/on-premises-vs-cloud-storage/ https://wasabi.com/blog/on-premises-vs-cloud-storage/
  - hinkley 3y ago
    140 racks is a quite a lot of tonnage of AC as well. And probably hot and cold rows to make it function.
    - jabart 3y ago
      Hot/cold was factored in not the AC or power needs. I don't know many places you could just casually ask for 100+ racks. Maybe our midwest datacenters are just smaller than what is on the coast.
      - 3y ago

xarope 3y ago

let me give some hypothetical; let's say Canva decides to cut cost by cutting the amount of storage users could have (e.g. quota, or lifetime): - if they had a sunk cost CAPEX of 100 PB of storage, all they could do is sell the storage servers for a fraction of the original cost (and folks like me snapping up pre-owned server equipment for said fraction of cost for our home labs). - on the other, since S3 is OPEX (not sure if they are locked in to 1 year billing cycles? but still better than 3-5 year CAPEX runs), they could cut the cost of that storage "immediately". Similarly, if they needed to expand, CAPEX means placing an order for the storage, waiting for delivery, racking, burning in etc, versus just paying almost "instantly" for more storage. So long as they are still earning money, why wouldn't they stick to this OPEX model? Not saying this is good or bad, it depends on your business model, ultimately.

re-thc 3y ago
> So long as they are still earning money, why wouldn't they stick to this OPEX model? The suggestion was to move to a different / cheaper provider. You can stick to OPEX.

ToJans 3y ago

I'd go for a hybrid approach: new data is stored on private cloud, and old data would be gradually migrated or phased out.

re-thc 3y ago

> Cloudflare R2 charges zero egress fees. The scarier part is Canva actually does use Cloudflare CDN, so it's not so much the cost of adding a new provider. It's just that the CDN team that manages the Cloudflare account is likely a different team :) Too much isolation, silos and politics...

sofixa 3y ago
The much more likely answer is that R2 is too new compared to when they started (announced in 2021) and switching from S3 to it is probably not trivial at Canva's scale.

Sai_ 3y ago

They could optimize this by only egressing they most recently used petabytes of data for some value of “recently”.

littlestymaar 3y ago

“Nobody ever got fired for choosing I̵B̶M̶ Amazon”

whycombagator 3y ago

What happens when cloudflare changes the price?

ttymck 3y ago

Why do they need to get everything out? Some not-insignificant percentage of the content is surely archived/unused/unneeded assets. From customers who will never log in again or projects which are long over. Export the most recent 10% and put anything new on <new storage>

notyourwork 3y ago
Absolutely agree. There are more nuanced strategies to migrate off a cloud provider. Could even enable a soft export for idle accounts, leave it where it lie and move it when customer re-engages.
pc86 3y ago
I would think you could set up a migration process where existing things are loaded off of S3, and as soon as you make any changes, you save it to $NEW_STORAGE and delete it off of S3. Eliminates the need to egress anything, lowers your S3 bill over time, and as a user it'd be pretty hard to log in after a year and be upset that your FB cover image is no longer there, especially on the free tier.

avereveard 3y ago

what's the durability of R2?

re-thc 3y ago
eleven 9's apparently
VHRanger 3y ago
How would you measure that realistically? How durable is Backblaze or Azure? Cloudflare has been reliable on storage adjacent uses (caching) for a long time. Presumably you're more likely to be the cause of errors than R2 is by a few orders of magnitude.
- avereveard 3y ago
  well you don't measure it, you take it from their sla
  - VHRanger 3y ago
    What I mean is that if you have a claim for eleven 9's of durability for a product that's 3 years old you have to take their word for it. You can't really check the SLA as being realistic

speed_spread 3y ago

Nitpick: It's "you can _check out_ anytime you want but you can never leave"; the dissociation between checking out and actually leaving is what makes the line. Still a good reference for the S3 situation!

electroly 3y ago

Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of spend you'd be crazy to pay the full $0.09/GB. Edit: Note also that Snowball egress is $0.03/GB. Slightly higher egress, much lower setup cost. You'll have to do the math but they're both clearly attractive options vs. full price $0.09/GB egress.

fakedang 3y ago
Plot twist : Canva blogged this so someone could tell them some solution like this, without them having to hire the guy. :)
- tough 3y ago
  There's this law on the internet or SO or whatever where if you put the wrong answer in a post. Someone else in the comments will correct you with the correct answer. This in effect.
  - lbotos 3y ago
    https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham's_Law%22 https://en.wikipedia.org/wiki/Ward_Cunningham#%22Cunningham'...
    - leon1717 3y ago
      Cunningham is cunning!!!
  - grogenaut 3y ago
    This isn't just the internet. I've used this at work several times to break stalemates. I've come into many stalled projects where people are arguing over trivial details or blue skying on requirements. 1-2 hours and a 7 page design doc later I'll have everyone review what I know is an 80% answer. Pretty much every time they'll all start attacking my design, pointing out minor issues and then I have them. They've agreed to my overall design and are into details. Same if someone is calling out all of the general problems with solving a problem but not providing answers where you can't get them to bite.

xwowsersx 3y ago

It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. If they wanted to move off of S3, they'd rely on the same type of analysis they present in this post + perhaps creating a mechanism whereby they permanently move an asset off of S3 if/when accessed.

ignoramous 3y ago
> It's hard to imagine why they would ever need to get everything out. That would be extremely wasteful. See the Docker x Cloudflare case study where improving cache-hit ratio by 2% (by moving to R2) decreased S3 egress fee by 66%: https://www.cloudflare.com/en/case-studies/docker/ https://www.cloudflare.com/en/case-studies/docker/

shadowtree 3y ago

Most businesses treat cloud like a utility, not tech. Don't want to generate their own power, deal with sewage ... and compute. Which in 99.9% of the cases is the right decision. Sorry for all you infra techies, but that's the way tech matures.

betaby 3y ago
Infra techies are fine. Likewise 'grid is mature' and yet most of buildings with floor space over NN m2 have team of electricians, being them direct hire or contractors. Same for the 'cloud' - 90%+ of the world deployed 'compute' is not cloud.
- shadowtree 3y ago
  Work for a 30bn market cap company, all our stuff is built on AWS. Offices all around the globe. Not a single electrician on payroll. YMMV.
  - betaby 3y ago
    You may call your company arrangement `electricityless` even! Like `serverless` - we don't use servers.
  - mschuster91 3y ago
    > Not a single electrician on payroll. Directly, yes maybe. Many companies don't have in-house experts (and get shafted by vendors as a result, as they're lacking the competence required to assess the quality of work). But indirectly, they all have. In Germany, at least, you're required to have electrical appliances inspected by a licensed professional (an electrician or otherwise qualified person) at least every two years - most companies opt for using dedicated external companies, but you can also train someone for that task. On top of that come all the electricians hired by the landlords - the ones dealing with complaints and remodels, or servicing lifts, escalators, HVAC, datacenters, ...
    - shadowtree 3y ago

dehrmann 3y ago

The real story isn't how expensive it is to leave, it's how easy AWS makes it to spend. It used to be that you'd see how many servers a project needs and have to beg someone for them, and then only get half of what you wanted. With AWS, it's much easier to launch something new, but there isn't the same amount of scrutiny on spend.

choppaface 3y ago

“we store more than 230 Petabytes of data in Amazon S3, with our single largest S3 bucket coming in at a whopping 45 Petabytes” How did they get so much data?! They say they only have 75 million stock photos. They must have tens of petabytes that are just dark.

jedberg 3y ago

> At list price of 9 cents per GB it would cost Canva about USD$20 million to export their 230 petabytes. Let me know if my calculation is wrong. If you're doing a one-time export you can use a Snowball, which would cost about $70K for 230PB, assuming you had somewhere to load it off to once you got the data so you could give the device back.

alecco 3y ago

And now it's VC-fashionable to have S3 as "object store" for "serverless" databases. "offload to S3!"

vkaku 3y ago

Cloudflare could build a product that not only caches, but transfers S3 to R2 on usage. - Since non transactional data is likely to be on GitHub or somewhere, it should be easy to re-hydrate that on R2. - Once enough time passes, there could be a sunset on unused data, or a strategy planned to 1-time exit write-off for S3 with those assets not on R2. Standard migration procedure.