10 ms·

I thought that's what their DCs were for. It definitely seems a bit out of character for a company that designs its own silicon to outsource something so compar

I thought that's what their DCs were for. It definitely seems a bit out of character for a company that designs its own silicon to outsource something so comparatively trivial.

sethhochberg 9y ago
Outsource the easy things so you can focus on the hard things - custom silicon provides a lot more unique value for Apple customers than commodity cloud storage.
- radiorental 9y ago
  If the cost savings of rolling your own versus paying someone else marginally more are not there, why bother? It gives Apple flexibility and avoids secondary technical and real estate type debts.
  - fixermark 9y ago
    Emphasis on the real-estate. It makes sense for Amazon, Google, and Microsoft to build out datacenters because they're both using them for their core business and selling compute and storage resources to third-parties. Unless Apple intends to get into the cloud computing business, eating the cost of tooling up a datacenter puts a big construction and maintenance cost on the "Liabilities" side of the T-sheet that they can avoid via a smaller rental cost.
    - what_ever 9y ago
      Real-estate is probably the cheapest part about datacenters. You also seem to be forgetting how much cash Apple is sitting on.
      - selectodude 9y ago
        I'm know how to change the oil on my car, it would quickly become cheaper for me to get the ramps and the oil catch pan and such instead of pay somebody else to do it, but I'm still never going to do it. I don't want to keep that stuff in my house, I don't value the extra few dollars I pay for somebody else to handle it. The fixed costs to save some money in the long run are irrelevant because I don't want to be handling any of it to begin with. Now, if I had a fleet of cars, the calculus may change on that.
      - fixermark 9y ago
        Yeah, sorry; I said "real estate" when what I should probably have said was "Physical ownership and plant maintenance." Owning and operating a datacenter is a whole kettle of fish that they don't want to get into if they don't need to. The real estate taxes (wherever they set it up) will matter, but the larger costs are likely to be in day-to-day operation, upgrading and maintaining hardware (and the entire process for that), dealing with actual natural disasters like a flood (or a bird flying into a transformer house and blowing power to N% of the datacenter, which means they need a backup generator, which means they need to test and maintain the backup generator, etc., etc.). There's definitely a break-point where it's cheaper to pay someone else (in Apple's case, multiple someone-else's) to deal with that hassle.
SEJeff 9y ago
They run siri in their own datacenter ontop of Mesos: https://www.infoq.com/news/2015/05/mesos-powers-apple-siri https://www.infoq.com/news/2015/05/mesos-powers-apple-siri I was at mesoscon a few (several?) years ago when they announced it and gave a talk regarding the tech side of things.
falcolas 9y ago
Reliably storing a lot of data for a long time with low latency and high bandwidth worldwide requires fairly specialized hardware and talent. If you're not willing/able to invest in the proper hardware (which is very different from regular compute work), it's definitely better to hand it off to a 3rd party. Looking at BackBlaze's posts about designing rack-mounted dense storage units and their reliability tests is very informative on how specialized it really can be.
- Cthulhu_ 9y ago
  And yet you'd think they have the numbers to warrant investing in it; Dropbox did too, and I read it saved them 75 million since moving to their own datacenters.
  - ceejayoz 9y ago
    With the pile of cash they're sitting on, it may be worth their while to pay more for storage rather than having to try to staff up their own infrastructure. They've plenty of money - more than they can figure out what to do with, it seems - but hiring's harder.
  - falcolas 9y ago
    $75 million isn't a lot, at Apple's scale. Especially since the entry cost could easily scale up into the billions (world-wide, high bandwidth, lots of storage). This is a swag, but I imagine more content is stored on Dropbox than iCloud, potentially lowering the savings even more.
- scurvy 9y ago
  Nah it's all fairly off the shelf at this point. You can run Ceph on top of server designs from Quanta and already be at near parity with S3 under your own roof. You just have to want to do it; something I bet Apple doesn't care about yet
- scarface74 9y ago
  Backblaze only has one data center. The latency is high if you are too far away from it. Which is fine for a backup solution. If that data center gets hit by a meteor -- you're toast. Again that's fine for a backup, especially if you actually implement the 3-2-1 back up strategy that they are always recommending.
agoodthrowaway 9y ago
It takes about 3 years to build a DataCenter. Also depending on the type of campus you want to build, the availability of power and contiguous land is fairly limited. Additionally most web scale DCs are "passively" cooled and require the local environment to support this. We are in a construction boom nationally and it's currently difficult to get enough people to construct DCs. The ability to scale out is limited by construction. Apple is building DCs, they are late to the game, but they probably can't scale their infra fast enough and need to use cloud services as a way to keep scaling without impacting their customers.
- tssva 9y ago
  Couldn't tell this driving around Loudoun County, VA where I live. Multiple large scale data center developments on going and it seems a new one gets kicked off every couple of weeks. A new 750,000 sq ft data center complex was just approved at the end of January and initial delivery is expected this fall. Such a quick delivery seems to indicate there is plenty of construction resources available.
  - walrus01 9y ago
    Based on reports from people I know: Apple already has a great deal of stuff in the NoVA datacenter cluster near Dulles and will most likely not spend any money to put more there. They need to be geographically distributed for latency and reliability reasons.
- walrus01 9y ago
  There is a middle ground between the two extremes of building your own datacenter from bare land, and using GCP or Azure or similar to host everything. Using WA and Oregon states as an example there's plenty of datacenter companies that will rent you "powered shell" space with cooling to do whatever you want with. There are plenty of datacenter-owning real estate investment trusts that build huge facilities expressly for the purpose of turning around and immediately subleasing space to large third parties. With facilities that are already built and online this can reduce your 3-year timespan to a couple of months from contract execution to thousands of live servers, if the people doing the contracts and technical specs are sharp. For a company the size of Apple, if they do have intentions to 100% own their datacenters, this sort of thing would be a stopgap solution. But it's a possibility.
deleted 9y ago
[deleted]
tooltalk 9y ago
I don't think Apple has the kind of technical know-how's that Google has. Google develops their own tools to support their infrastructure; Apple is narrowly focused on a few areas in product/services, namely mobile and desktop, only and buys off the shelf-ware for everything else, from storage (EMC Isilon), to OS (Redhat Linux), to orchestration/mgmt (Mesos), etc, etc...
- yeukhon 9y ago
  Apple can hire the engineers to build an object storage system, but to build one that can scale takes at least a couple years. If Dropbox can move away from S3 with much less capital, so can Apple. Apple doesn't want to invest in that effort at all.

I thought that's what their DCs were for. It definitely seems a bit out of character for a company that designs its own silicon to outsource something so comparatively trivial.

sethhochberg 9y ago

Outsource the easy things so you can focus on the hard things - custom silicon provides a lot more unique value for Apple customers than commodity cloud storage.

radiorental 9y ago
If the cost savings of rolling your own versus paying someone else marginally more are not there, why bother? It gives Apple flexibility and avoids secondary technical and real estate type debts.
- fixermark 9y ago
  Emphasis on the real-estate. It makes sense for Amazon, Google, and Microsoft to build out datacenters because they're both using them for their core business and selling compute and storage resources to third-parties. Unless Apple intends to get into the cloud computing business, eating the cost of tooling up a datacenter puts a big construction and maintenance cost on the "Liabilities" side of the T-sheet that they can avoid via a smaller rental cost.
  - what_ever 9y ago
    Real-estate is probably the cheapest part about datacenters. You also seem to be forgetting how much cash Apple is sitting on.
    - selectodude 9y ago
      I'm know how to change the oil on my car, it would quickly become cheaper for me to get the ramps and the oil catch pan and such instead of pay somebody else to do it, but I'm still never going to do it. I don't want to keep that stuff in my house, I don't value the extra few dollars I pay for somebody else to handle it. The fixed costs to save some money in the long run are irrelevant because I don't want to be handling any of it to begin with. Now, if I had a fleet of cars, the calculus may change on that.
    - fixermark 9y ago
      Yeah, sorry; I said "real estate" when what I should probably have said was "Physical ownership and plant maintenance." Owning and operating a datacenter is a whole kettle of fish that they don't want to get into if they don't need to. The real estate taxes (wherever they set it up) will matter, but the larger costs are likely to be in day-to-day operation, upgrading and maintaining hardware (and the entire process for that), dealing with actual natural disasters like a flood (or a bird flying into a transformer house and blowing power to N% of the datacenter, which means they need a backup generator, which means they need to test and maintain the backup generator, etc., etc.). There's definitely a break-point where it's cheaper to pay someone else (in Apple's case, multiple someone-else's) to deal with that hassle.

SEJeff 9y ago

They run siri in their own datacenter ontop of Mesos: https://www.infoq.com/news/2015/05/mesos-powers-apple-siri https://www.infoq.com/news/2015/05/mesos-powers-apple-siri I was at mesoscon a few (several?) years ago when they announced it and gave a talk regarding the tech side of things.

falcolas 9y ago

Reliably storing a lot of data for a long time with low latency and high bandwidth worldwide requires fairly specialized hardware and talent. If you're not willing/able to invest in the proper hardware (which is very different from regular compute work), it's definitely better to hand it off to a 3rd party. Looking at BackBlaze's posts about designing rack-mounted dense storage units and their reliability tests is very informative on how specialized it really can be.

Cthulhu_ 9y ago
And yet you'd think they have the numbers to warrant investing in it; Dropbox did too, and I read it saved them 75 million since moving to their own datacenters.
- ceejayoz 9y ago
  With the pile of cash they're sitting on, it may be worth their while to pay more for storage rather than having to try to staff up their own infrastructure. They've plenty of money - more than they can figure out what to do with, it seems - but hiring's harder.
- falcolas 9y ago
  $75 million isn't a lot, at Apple's scale. Especially since the entry cost could easily scale up into the billions (world-wide, high bandwidth, lots of storage). This is a swag, but I imagine more content is stored on Dropbox than iCloud, potentially lowering the savings even more.
scurvy 9y ago
Nah it's all fairly off the shelf at this point. You can run Ceph on top of server designs from Quanta and already be at near parity with S3 under your own roof. You just have to want to do it; something I bet Apple doesn't care about yet
scarface74 9y ago
Backblaze only has one data center. The latency is high if you are too far away from it. Which is fine for a backup solution. If that data center gets hit by a meteor -- you're toast. Again that's fine for a backup, especially if you actually implement the 3-2-1 back up strategy that they are always recommending.

agoodthrowaway 9y ago

It takes about 3 years to build a DataCenter. Also depending on the type of campus you want to build, the availability of power and contiguous land is fairly limited. Additionally most web scale DCs are "passively" cooled and require the local environment to support this. We are in a construction boom nationally and it's currently difficult to get enough people to construct DCs. The ability to scale out is limited by construction. Apple is building DCs, they are late to the game, but they probably can't scale their infra fast enough and need to use cloud services as a way to keep scaling without impacting their customers.

tssva 9y ago
Couldn't tell this driving around Loudoun County, VA where I live. Multiple large scale data center developments on going and it seems a new one gets kicked off every couple of weeks. A new 750,000 sq ft data center complex was just approved at the end of January and initial delivery is expected this fall. Such a quick delivery seems to indicate there is plenty of construction resources available.
- walrus01 9y ago
  Based on reports from people I know: Apple already has a great deal of stuff in the NoVA datacenter cluster near Dulles and will most likely not spend any money to put more there. They need to be geographically distributed for latency and reliability reasons.
walrus01 9y ago
There is a middle ground between the two extremes of building your own datacenter from bare land, and using GCP or Azure or similar to host everything. Using WA and Oregon states as an example there's plenty of datacenter companies that will rent you "powered shell" space with cooling to do whatever you want with. There are plenty of datacenter-owning real estate investment trusts that build huge facilities expressly for the purpose of turning around and immediately subleasing space to large third parties. With facilities that are already built and online this can reduce your 3-year timespan to a couple of months from contract execution to thousands of live servers, if the people doing the contracts and technical specs are sharp. For a company the size of Apple, if they do have intentions to 100% own their datacenters, this sort of thing would be a stopgap solution. But it's a possibility.

deleted 9y ago

[deleted]

tooltalk 9y ago

I don't think Apple has the kind of technical know-how's that Google has. Google develops their own tools to support their infrastructure; Apple is narrowly focused on a few areas in product/services, namely mobile and desktop, only and buys off the shelf-ware for everything else, from storage (EMC Isilon), to OS (Redhat Linux), to orchestration/mgmt (Mesos), etc, etc...

yeukhon 9y ago
Apple can hire the engineers to build an object storage system, but to build one that can scale takes at least a couple years. If Dropbox can move away from S3 with much less capital, so can Apple. Apple doesn't want to invest in that effort at all.