4 ms·

I'd like to challenge your implicit assertion that if due to knowing customer behavior patterns, scaling is predictable, therefore (this is the part I'm assumin

I'd like to challenge your implicit assertion that if due to knowing customer behavior patterns, scaling is predictable, therefore (this is the part I'm assuming you're implying) scaling is doable (or even easy). The counter argument I have is that at different sizes of operations, completely new skills become important, so you and your staff are left behind. Example: my previous employer became large enough in terms of hardware footprint (~>1M cores) that it started getting difficult to find commercial colo space. How good are software and systems engineers at electrical engineering? :)

vidarh 9y ago
The alternatives aren't either AWS or host yourself. You can rent managed servers for a fraction of the price of AWS instances. Granted, if you need 1M+ cores, you're going to be dealing with humans most places (including AWS) to get the best deal possible, and that also means the cost differences can change fairly substantially (e.g. the instances I know of that are in "ask us" territory are not paying anywhere near published prices) That said 1M cores is not that much. Depending on your needs it's as little as "just" 500 racks. Plenty of managed providers will be happy to provide customized service to e.g. design and/or manage a setup for something that size.
- user5994461 9y ago
  I will take a guess that 1M core is a big compute grid. You go to google cloud and order 10k preemptible instances per datacenter. That solves half of your problem. Then the same onto SoftLayer and AWS and revise monthly who is the cheapest one. It's not very difficult. Contrary to you, I think that most places are not ready to accept that kind of contract. I have a friend who evaluated to move one of his compute grid to DigitalOcean and he was contacted to stop doing that, their servers are not meant to run at 100% CPU use all the time.
  - vidarh 9y ago
    I wasn't talking about cloud providers, as for that kind of scale you're paying 3-4 times easy vs. hiring someone to prepare a cage design for you and wire it up at a suitable colo provider. And finding colo providers that can take a sufficient numbers of racks to be worthwhile is easy.
    - smueller1234 9y ago
      We were already paying colo providers by the MW and had the people to build out the space. It's just that the upfront capex required for a colo provider to build capacity was getting to a point where that meant a significant bet on their part on our continued growth. Thus we started having to look at one options, one of which is to contract them to build bespoke infra. Regarding cloud being much higher cost: indeed. Anecdotal evidence; I know of a big reference customer who said they would consider the move to cloud a success if total cost didn't go up more than 2x. A few years in, that hadn't happened yet...
      - user5994461 9y ago
        Agreed. It's a hassle to negotiate a few racks and that's barely a few millions dollars. I can't imagine doing that for a hundred rack at a times. Twice as much would be a huge savings. In the cloud, I can give you 20k cores in one hour and I can give you 100k cores in two hours, after the initial investment to open multiple regions and networks backbones. The price is known in advance and the investment can be cancelled anytime. Obtaining the same 20k cores from a colo will be a hassle. Months of negotiations with nothing being done. I'd expect any sane provider to bail if trying to discuss 100k cores, it's too much upfront investment. It would take years to get half of it, the project is long abandoned before anything is delivered and everyone who worked on it has left.
  - smueller1234 9y ago
    Our use case was actually very mixed. Certainly throw in somewhere around 100PB of storage if you want to do a semi accurate napkin math.
- mrep 9y ago
  For a contract that big, AWS will build you your own private datacenter. US intelligence bought one themselves back in 2013 [0] and AWS just built a second one for them (different security clearance for each)[1]. [0]: https://www.theatlantic.com/technology/archive/2014/07/the-details-about-the-cias-deal-with-amazon/374632/ https://www.theatlantic.com/technology/archive/2014/07/the-d... [1]: https://aws.amazon.com/blogs/publicsector/announcing-the-new-aws-secret-region/ https://aws.amazon.com/blogs/publicsector/announcing-the-new...

vidarh 9y ago

The alternatives aren't either AWS or host yourself. You can rent managed servers for a fraction of the price of AWS instances. Granted, if you need 1M+ cores, you're going to be dealing with humans most places (including AWS) to get the best deal possible, and that also means the cost differences can change fairly substantially (e.g. the instances I know of that are in "ask us" territory are not paying anywhere near published prices) That said 1M cores is not that much. Depending on your needs it's as little as "just" 500 racks. Plenty of managed providers will be happy to provide customized service to e.g. design and/or manage a setup for something that size.

user5994461 9y ago
I will take a guess that 1M core is a big compute grid. You go to google cloud and order 10k preemptible instances per datacenter. That solves half of your problem. Then the same onto SoftLayer and AWS and revise monthly who is the cheapest one. It's not very difficult. Contrary to you, I think that most places are not ready to accept that kind of contract. I have a friend who evaluated to move one of his compute grid to DigitalOcean and he was contacted to stop doing that, their servers are not meant to run at 100% CPU use all the time.
- vidarh 9y ago
  I wasn't talking about cloud providers, as for that kind of scale you're paying 3-4 times easy vs. hiring someone to prepare a cage design for you and wire it up at a suitable colo provider. And finding colo providers that can take a sufficient numbers of racks to be worthwhile is easy.
  - smueller1234 9y ago
    We were already paying colo providers by the MW and had the people to build out the space. It's just that the upfront capex required for a colo provider to build capacity was getting to a point where that meant a significant bet on their part on our continued growth. Thus we started having to look at one options, one of which is to contract them to build bespoke infra. Regarding cloud being much higher cost: indeed. Anecdotal evidence; I know of a big reference customer who said they would consider the move to cloud a success if total cost didn't go up more than 2x. A few years in, that hadn't happened yet...
    - user5994461 9y ago
      Agreed. It's a hassle to negotiate a few racks and that's barely a few millions dollars. I can't imagine doing that for a hundred rack at a times. Twice as much would be a huge savings. In the cloud, I can give you 20k cores in one hour and I can give you 100k cores in two hours, after the initial investment to open multiple regions and networks backbones. The price is known in advance and the investment can be cancelled anytime. Obtaining the same 20k cores from a colo will be a hassle. Months of negotiations with nothing being done. I'd expect any sane provider to bail if trying to discuss 100k cores, it's too much upfront investment. It would take years to get half of it, the project is long abandoned before anything is delivered and everyone who worked on it has left.
- smueller1234 9y ago
  Our use case was actually very mixed. Certainly throw in somewhere around 100PB of storage if you want to do a semi accurate napkin math.
mrep 9y ago
For a contract that big, AWS will build you your own private datacenter. US intelligence bought one themselves back in 2013 [0] and AWS just built a second one for them (different security clearance for each)[1]. [0]: https://www.theatlantic.com/technology/archive/2014/07/the-details-about-the-cias-deal-with-amazon/374632/ https://www.theatlantic.com/technology/archive/2014/07/the-d... [1]: https://aws.amazon.com/blogs/publicsector/announcing-the-new-aws-secret-region/ https://aws.amazon.com/blogs/publicsector/announcing-the-new...