5 ms·

Yes, you’re paying for a ton of stuff there that you probably don’t use and then are susceptible to bugs that have nothing to do with your use case. At $5B it

Yes, you’re paying for a ton of stuff there that you probably don’t use and then are susceptible to bugs that have nothing to do with your use case. At $5B it would not cost anywhere near that much to replicate. Your infra would be better tailored to your workloads and your sw teams which would further drive costs down. The upsides are that you only drive features you need and keep things simple. The downside is that your need a big org to do this. It’s a long term play.

nemothekid 6y ago
Snap was spending 2B over 5 years - so 400M/yr. By comparison, Uber, who operates under managed colo, spends nearly 200M/yr alone on real estate for their datacenters. Who knows how much they are paying in engineering salaries so manage those datacenters. Personally I don't think there's a one size fits all solution, you will have to do the math (like I'm sure Snap, Netflix and others have done) to see if cloud is worth it. However, I agree, for most teams the default should be cloud.
- objclxt 6y ago
  400M/yr was just Snap’s Google Cloud spend, they also signed a 1B deal with AWS for redundancy.
- avifreedman 6y ago
  Unless that's capex for building, or includes server capex that they depreciate, that seems way high given Uber's scale. A pretty dense cabinet should only cost ~$1400/mo at wholesale (1MW+ rooms) rates, and $200M is 143,000 cabinets. And it's public that Uber uses multiple clouds as well. Disclaimer: I haven't reviewed public Uber filings, would be very interested if there's any data that indicates they're really spending $200M on opex for real estate (which would be equivalent to $400M/year on cloud, which is either opex or potentially mix if there is some reserved instance-type cloud spend).
  - user5994461 6y ago
    The 42U rack might be only $1400 per month but the servers to put inside are $5000 upfront per U.
    - wongarsu 6y ago
      42U times $5000 divided by 3 years is still only $5833 per month per rack. Add the $1400 and we have a total amortized rack cost of $7233 per month. (naively assuming you could fill every spot in the rack with servers). But servers can't be included in a "Uber spends nearly 200M/yr alone on real estate for their datacenters" figure anyway.
      - user5994461 6y ago
        The rack is $2800 per month, not $1400. The lowest advertised pricing does not include enough power to supply half of what's in the rack. Add $200 per server per month to have a gigabit uplink. Add $4000 per server per lifetime for VmWare licenses. These are reasonable estimates of course. Could be multiple of that depending on what hardware is used and what colo. Physical hardware easily gets as expensive as any cloud.
        dijit 6y ago
        Not easily; you’re making the argument that there can be high expenses for collocating but you’re talking about a -lot- of computational power. I know that hardware varies a lot for sure, but for context I put together 3 racks 50% density with an 800gbit backplane for around 18k eur/mo. I spared no expense, official juniper QSFPs (which are egregiously overpriced) and top of the line Dell servers with full out of band licenses. And once there: interconnected bandwidth and IOPs became “free” (or, no extra charge). We put the same application in cloud And it costs us 40k eur/mo with a heavy amount of optimisation, with half-sized instances and an aggressive optimisation in bandwidth/iops. Clouds “sticker price” to us is 4x that of physical. You can buy a lot of human time for that price.
        6y ago
- raverbashing 6y ago
  Well those numbers are... weird Also Uber seems to be much less computing/BW demanding than Snap
- twic 6y ago
  > Uber, who operates under managed colo, spends nearly 200M/yr alone on real estate for their datacenters 200 million a year, huh. The cost of commercial office space in the U.S. can range from $6 per square foot in low cost regions to over $12 per square foot in New York City. On average, a 50-cabinet data center will occupy about 1,700 square feet. At a median cost of $8 per square foot, the space alone would cost about $13,600 per month. [1] Are Uber renting on the order of two million square feet of data centre? Do they have sixty thousand cabinets of hardware? If they do, i would absolutely love to see a quote for how much it would cost them to run in the cloud. I think it's far more likely that number is bullshit. [1] https://npifinancial.com/blog/data-center-pricing/ https://npifinancial.com/blog/data-center-pricing/
  - nemothekid 6y ago
    >Office and data center rent expense was $194 million and $221 million for the years ended December 31, 2017 and 2018, respectively. https://s23.q4cdn.com/407969754/files/doc_financials/2019/ar/Uber-Technologies-Inc-2019-Annual-Report.pdf https://s23.q4cdn.com/407969754/files/doc_financials/2019/ar... Page 124
    - ptomato 6y ago
      So that number includes (and is almost certainly dominated by) the cost of their office space.
- StreamBright 6y ago
  Exactly. Calculating TCO for on-prem vs cloud is tricky. Any time we have done it cloud came out the winner. I also found exceptions: predictable static workloads requiring a huge amount of bandwidth, mostly outgoing. An average company that needs some CPU and storage for various unpredictable workloads benefits from cloud services greatly.
- junkilo 6y ago
  disclaimer: former uber engineer we ran the numbers continually and like others mentioned, it was a no-brainer to build on-prem. that said, there are a few use cases where aws/gcp are a great fit. people selling cloud without putting in real engineering work to enterprises generally make my life harder than it already is.
kevindong 6y ago
@nemothekid Can you provide a source for your claim on Uber's real estate costs for their data centers? I couldn't find anything on Google that corroborates your assertion.
- soniman 6y ago
  Uber 10-K has operating lease payments in 2019 of $196M and in 2020 of $216M. That's not all data centers but a lot of it is data centers.
  - kortilla 6y ago
    Why would you assume that? SF office space goes for way more than data center space.
    - deleted 6y ago
      [deleted]
    - nemothekid 6y ago
      You can look up the prices of Uber's current office space - they currently pay 16M/yr for their HQ.
mayank 6y ago
> Yes, you’re paying for a ton of stuff there that you probably don’t use This is the antithesis of any cloud. You only pay for what you use. > At $5B it would not cost anywhere near that much to replicate. If you can recreate GCP for $5B in capex, there are likely some VCs lurking here who would like a word with you.
- mmmBacon 6y ago
  Sorry but that’s not correct. Have you ever looked at the economics of cloud compute at that scale? You absolutely are paying for a lot of tooling that’s irrelevant to your use case. This is because GCP has to be a general provider. They have to offer the broadest possible solution to capture as much share as possible. This means there are money losing services (also low margin) that are offered to provide a complete end-to-end solution. However, the fact that those services exist means that the cost has to be made up for elsewhere. These economics are not some complicated rocket science and exist in many other types of businesses.
  - hinkley 6y ago
    Like cable TV.
    - xmprt 6y ago
      That's not a perfect analogy but it's easy enough for laymen to understand so I'm stealing it.
StreamBright 6y ago
If I use S3 + EMR what exactly falls into the category of "a ton of stuff there that you probably don’t use"? >> Your infra would be better tailored to your workloads How do you scale elastically with an on-prem infra? What part do you tune more to your own workload when an average company cannot even tune the GC for Java for their own workload? Your claims do not reflect on reality, it is rather your imagination. I have migrated countless companies to the cloud, and almost every single migration was driven by a single factor: cost. Everything else was an added bonus: inscrased security, availability and elasticity.

nemothekid 6y ago

Snap was spending 2B over 5 years - so 400M/yr. By comparison, Uber, who operates under managed colo, spends nearly 200M/yr alone on real estate for their datacenters. Who knows how much they are paying in engineering salaries so manage those datacenters. Personally I don't think there's a one size fits all solution, you will have to do the math (like I'm sure Snap, Netflix and others have done) to see if cloud is worth it. However, I agree, for most teams the default should be cloud.

objclxt 6y ago
400M/yr was just Snap’s Google Cloud spend, they also signed a 1B deal with AWS for redundancy.
avifreedman 6y ago
Unless that's capex for building, or includes server capex that they depreciate, that seems way high given Uber's scale. A pretty dense cabinet should only cost ~$1400/mo at wholesale (1MW+ rooms) rates, and $200M is 143,000 cabinets. And it's public that Uber uses multiple clouds as well. Disclaimer: I haven't reviewed public Uber filings, would be very interested if there's any data that indicates they're really spending $200M on opex for real estate (which would be equivalent to $400M/year on cloud, which is either opex or potentially mix if there is some reserved instance-type cloud spend).
- user5994461 6y ago
  The 42U rack might be only $1400 per month but the servers to put inside are $5000 upfront per U.
  - wongarsu 6y ago
    42U times $5000 divided by 3 years is still only $5833 per month per rack. Add the $1400 and we have a total amortized rack cost of $7233 per month. (naively assuming you could fill every spot in the rack with servers). But servers can't be included in a "Uber spends nearly 200M/yr alone on real estate for their datacenters" figure anyway.
    - user5994461 6y ago
      The rack is $2800 per month, not $1400. The lowest advertised pricing does not include enough power to supply half of what's in the rack. Add $200 per server per month to have a gigabit uplink. Add $4000 per server per lifetime for VmWare licenses. These are reasonable estimates of course. Could be multiple of that depending on what hardware is used and what colo. Physical hardware easily gets as expensive as any cloud.
      - dijit 6y ago
        Not easily; you’re making the argument that there can be high expenses for collocating but you’re talking about a -lot- of computational power. I know that hardware varies a lot for sure, but for context I put together 3 racks 50% density with an 800gbit backplane for around 18k eur/mo. I spared no expense, official juniper QSFPs (which are egregiously overpriced) and top of the line Dell servers with full out of band licenses. And once there: interconnected bandwidth and IOPs became “free” (or, no extra charge). We put the same application in cloud And it costs us 40k eur/mo with a heavy amount of optimisation, with half-sized instances and an aggressive optimisation in bandwidth/iops. Clouds “sticker price” to us is 4x that of physical. You can buy a lot of human time for that price.
        6y ago
raverbashing 6y ago
Well those numbers are... weird Also Uber seems to be much less computing/BW demanding than Snap
twic 6y ago
> Uber, who operates under managed colo, spends nearly 200M/yr alone on real estate for their datacenters 200 million a year, huh. The cost of commercial office space in the U.S. can range from $6 per square foot in low cost regions to over $12 per square foot in New York City. On average, a 50-cabinet data center will occupy about 1,700 square feet. At a median cost of $8 per square foot, the space alone would cost about $13,600 per month. [1] Are Uber renting on the order of two million square feet of data centre? Do they have sixty thousand cabinets of hardware? If they do, i would absolutely love to see a quote for how much it would cost them to run in the cloud. I think it's far more likely that number is bullshit. [1] https://npifinancial.com/blog/data-center-pricing/ https://npifinancial.com/blog/data-center-pricing/
- nemothekid 6y ago
  >Office and data center rent expense was $194 million and $221 million for the years ended December 31, 2017 and 2018, respectively. https://s23.q4cdn.com/407969754/files/doc_financials/2019/ar/Uber-Technologies-Inc-2019-Annual-Report.pdf https://s23.q4cdn.com/407969754/files/doc_financials/2019/ar... Page 124
  - ptomato 6y ago
    So that number includes (and is almost certainly dominated by) the cost of their office space.
StreamBright 6y ago
Exactly. Calculating TCO for on-prem vs cloud is tricky. Any time we have done it cloud came out the winner. I also found exceptions: predictable static workloads requiring a huge amount of bandwidth, mostly outgoing. An average company that needs some CPU and storage for various unpredictable workloads benefits from cloud services greatly.
junkilo 6y ago
disclaimer: former uber engineer we ran the numbers continually and like others mentioned, it was a no-brainer to build on-prem. that said, there are a few use cases where aws/gcp are a great fit. people selling cloud without putting in real engineering work to enterprises generally make my life harder than it already is.

kevindong 6y ago

@nemothekid Can you provide a source for your claim on Uber's real estate costs for their data centers? I couldn't find anything on Google that corroborates your assertion.

soniman 6y ago
Uber 10-K has operating lease payments in 2019 of $196M and in 2020 of $216M. That's not all data centers but a lot of it is data centers.
- kortilla 6y ago
  Why would you assume that? SF office space goes for way more than data center space.
  - deleted 6y ago
    [deleted]
  - nemothekid 6y ago
    You can look up the prices of Uber's current office space - they currently pay 16M/yr for their HQ.

mayank 6y ago

> Yes, you’re paying for a ton of stuff there that you probably don’t use This is the antithesis of any cloud. You only pay for what you use. > At $5B it would not cost anywhere near that much to replicate. If you can recreate GCP for $5B in capex, there are likely some VCs lurking here who would like a word with you.

mmmBacon 6y ago
Sorry but that’s not correct. Have you ever looked at the economics of cloud compute at that scale? You absolutely are paying for a lot of tooling that’s irrelevant to your use case. This is because GCP has to be a general provider. They have to offer the broadest possible solution to capture as much share as possible. This means there are money losing services (also low margin) that are offered to provide a complete end-to-end solution. However, the fact that those services exist means that the cost has to be made up for elsewhere. These economics are not some complicated rocket science and exist in many other types of businesses.
- hinkley 6y ago
  Like cable TV.
  - xmprt 6y ago
    That's not a perfect analogy but it's easy enough for laymen to understand so I'm stealing it.

StreamBright 6y ago

If I use S3 + EMR what exactly falls into the category of "a ton of stuff there that you probably don’t use"? >> Your infra would be better tailored to your workloads How do you scale elastically with an on-prem infra? What part do you tune more to your own workload when an average company cannot even tune the GC for Java for their own workload? Your claims do not reflect on reality, it is rather your imagination. I have migrated countless companies to the cloud, and almost every single migration was driven by a single factor: cost. Everything else was an added bonus: inscrased security, availability and elasticity.