8 ms·

You’ve described the historical value of cloud computing perfectly. That said, I think the days where all but the largest or most stubborn companies run their o

You’ve described the historical value of cloud computing perfectly. That said, I think the days where all but the largest or most stubborn companies run their own datacenters are coming to a close. The problem will be finding skilled labor. Short-haul networking, power configurations, thermal load, hardware maintenance; these and many more are specific skills that can’t be learned overnight. Data center work used to be a viable middle-class career, but the pay scale for it has gone down and down. Companies that do run their own DCs like Google and Facebook have a few centralized experts, a thin professional staff on-prem, and an army of minimum wage disk swappers who are told what to do by a ticket system, just like an Amazon warehouse worker. The knowledge of how to build and run these things is all at the top now. I’m not saying the jobs or talent pool are gone. Just that they’re shrinking, and will continue to shrink. Like the manufacturing industry, the fewer people there are who are comfortable working with real hardware, the harder it will be to start anew.

betaby 6y ago
There is a lot of spectrum in the middle between AWS micro and your own datacenter(s). One don't have to jump from the cloud to datacenter - just rent/buy a server or two, or rack of server or two.
- blackaspen 6y ago
  Sure, but the management overhead from 1 server to 20 racks scales, at best, linearly. I've seen a lot of places just past the point where a single sys-ad person with a thumb drive is viable, yet they choose to not invest here and instead spend 2-5x on going to the cloud.
PopeDotNinja 6y ago
Basically it makes sense to set up your own infrastructure when your business IS your infrastructure.
- mooreds 6y ago
  Same as lawyering or accounting, right? I don't do contracts regularly unless my business in contracts. Same with taxes. Just another form of specialization. MSPs and data center companies have been doing this since the 1990s at least, this is just the next evolution.
birdyrooster 6y ago
They aren’t shrinking at all, we have the same talent pool and are trying to thinly spread it over all of the problems we can solve with networked computers. The amount of problems you can solve with networked computers has scaled way faster than our industries capability to find talent and train employees.
- ericbarrett 6y ago
  Who is “we”? Always curious what remains of my field—I’ve worked in the storage and infrastructure industry for over 20 years. If you mean the industry as a whole, I simply disagree. The talent pipeline is not there anymore. You used to be able to get started without an Engineering degree by “getting in at the bottom.” Now somebody on this path will never get past power supply swaps because they’re just doing drudge work fed to them on a tablet (“replace disk 17 on row 23, rack 9, slot 4”—“oh, yup, there’s the yellow light”). If you do have the degree, there are a billion more profitable paths, and most take them.
  - birdyrooster 6y ago
    We is engineers who work in datacenters. Colocation datacenters have existed for several decades and remote hands working there would probably not see much difference between getting a call/fax/email/app telling them to do a break fix order. Having worked in dozens of datacenters over the past two decades is that there will always be newer semi-technical workers there doing break fix. A lot of those people would consider those jobs dead ends too and so this cloud datacenter thing is nothing new, its just on a much larger scale. The datacenter operators don't have to keep working there to level up their careers but working in a datacenter gives familiarity and the confidence to try more technical roles later.
Ericson2314 6y ago
But it's insane that running a for-higher data center is considered a high-margin business. (And testament that the customers are VC-gorged price-unconcious baby gremlims.) In a sane economy, data centers for higher would be a fully-commoditized barely-profitable common carrier with little natural monopoly.
- dodobirdlord 6y ago
  Reliability concerns make datacenters resilient to commoditization. A datacenter that’s available 90% of the time is worth vastly, vastly less than 90% as much as one that 99.95% of the time. Commodity businesses are largely built on presumptions of linearity. Produce 90% as much corn/iron/wood/widgets as you expected and you’ll probably make something like 90% of the money you expected. Produce a 90% available datacenter and you’ll have a hard time finding anyone willing to pay you anything. And that’s just availability, not to mention data durability, which is even more critically nonlinear.
  - echelon 6y ago
    Then you should be able to sue your provider for breaches in SLA. 0.5% downtime should void the bill. I think presently the providers are getting all the upside. High margin, perpetual lock-in, and no consequences.
    - freeone3000 6y ago
      You can sue your provider for breaches in SLA. The rebate you automatically get likely exceeds what you'd otherwise recover. (Which is why they grant one). SLA payouts are usually very generous - I've gotten credit for the entire month for a one-hour outage of a service. I don't know where this idea that cloud providers and other DCs don't pay out for SLA agreements is, but they absolutely do.
  - dnautics 6y ago
    > A datacenter that’s available 90% of the time is worth vastly, vastly less than 90% as much as one that 99.95% of the time. That depends on your workload and the nature of the loss of availability. Is the completed work still there but just unreachable for an hour every few months? That might be okay for some folks. Is your data center always available mon-fri but constantly has scheduled downtime on weekends? Might still be okay.
hengheng 6y ago
I'm not in the web or cloud business, but I've filled a rack with my stuff before. My impression is that hardware has become a lot more capable even relative to its tasks. With high iops memory, many cores and obscene amounts of RAM, I would expect companies of a much larger scale (in $, FTEs, or most other metrics) can be served by one 4HE machine, or by one rack, or by one room. Thus I would expect the knowledge of how to handle 5000 hard drives to become more obscure, naturally, but the skill to run a decently sized web application to remain almost constant. Does this math work out, or have the tasks become more demanding at the same speed that hardware has improved?
- spullara 6y ago
  There are now businesses that explicitly depend on the elasticity of the cloud and can never really be moved on premise without massive up-front investment in hardware that may only be used a few times a year for their biggest customers. Trying to hybridize these workloads hasn't been very successful as of yet. It is possible that K8S could relive this problem but I haven't seen it in practice, at scale.
  - thor24 6y ago
    Instant Elasticity in Cloud is a myth. If you think you are going to get 1k hosts just like that from AWS you will have an unpleasant experience. I work at one of the decent size tech company and we are split between cloud and on prem. From our experience you have to inform AWS/GCP in advance (sometime way early) if you are looking to meaningfully increase capacity in zone/region. Sure, auto scaling few hundreds of hosts may be possible but people who run a service which needs few hundreds of hosts run it directly on AWS, they will run it some kind of scheduler+resource manager which will have some kind of operational buffer anyway (as in you would already have those hosts so cloud elasticity is not a factor here).
    - echelon 6y ago
      Why not go fully on-prem then? You can run kubernetes locally. Are managed data stores that attractive? You can pay for on-prem management. What workloads are in the cloud versus on-prem?
waprin 6y ago
It's not just managing the complexities of managing the bare metal, although that's certainly a huge component of it. There's some other huge arguments against running your own datacenters. One is being able to properly provision resources. Being able to write just a function and have it consume just that tiny amount of resources rather than a whole VM is huge. Being able to spin instances up and down as you need them is huge. I think that's been obvious for a long time, but what I've seen less obvious to the business analysts is the impact of more advanced cloud services. The direction of cloud computing is managed services where they run your databases, container platforms, etc for you. Trying to run a huge Cassandra cluster or Kubernetes cluster takes up a ton of expensive labor's time and there's a good chance the cloud providers are a lot better at it than you. Sure, cloud services tend to be really expensive, and besides cost, there's also concern about things like vendor lock-in, IP protection/data privacy, and ability to tweak the small details of your platform. But cloud platforms in 2020 have a lot more features than in 2010 and in 2030 will have even more. The direction is pretty obvious. Running your own datacenters will be about as common as running your own power plant.
- rualca 6y ago
  > One is being able to properly provision resources. Being able to write just a function and have it consume just that tiny amount of resources rather than a whole VM is huge. Being able to spin instances up and down as you need them is huge. A 75 million dollar price tag is also huge. Bothering about the operational impact of a VM or a request sent to a function-as-a-service might be a significant operational issue if your whole team can be moved around with a small sports utility vehicle. Once you've grow over the point where your monthly cloud pricetag eclipses your company's paycheck budget, operating your own hardware is a no-brainer. > But cloud platforms in 2020 have a lot more features (...) That really doesn't matter at all, does it? I mean, cloud providers are already repackaging FLOSS services as their Serverless offering. And besides pursuing the latest fad, how many of those features are killed off and vanish from the face of the earth? It's always great if we can get others to do the work for us, but if we consider the absurd premium charged by cloud providers for their services... Well, those "others" doing the work can be employed by your company and you still save money.
Shorel 6y ago
You say "stubborn" but when the savings can be in the millions of dollars, it becomes logical to own a datacenter. I can't believe datacenter people suddenly all disappeared or are retired. If the pay scale is down, they will surely appreciate working for a big company with datacenter needs. Obviously, this only applies to huge companies like Dropbox. Everyone else is better served by AWS.
- ericbarrett 6y ago
  This is explicitly what I’m calling out at the end. They didn’t all disappear or retire; it’s an ongoing process. Think COBOL programmers.
  - Shorel 6y ago
    Ha! I was also thinking about COBOL programmers when writing my comment.
- jonfromsf 6y ago
  You can rent racks of servers from a dedicated hosting provider and spend 10X less than AWS. You don't need to do you own wiring and HVAC and shit.
ryanmarsh 6y ago
all but the largest or most stubborn companies run their own datacenters are coming to a close You're right about the stubborn part, but not necessarily largest. Many IT leaders at companies of all sizes have their political capital tied up in the data center. What you're talking about doing (with cloud adoption) is outsourcing 90% of what they control. For them it's existential, uptime and agility be damned. Serverless is even worse for them, as far as IT fiefdoms are concerned.
mgiampapa 6y ago
Some of it is becoming more democratized too. Facebook as an example open sources it's datacenter design, it's servers, switches etc. I know they are the big bad for privacy these days, but the OCP project has contributed state of the art datacenter tech that legacy providers should be jumping at.
pjmlp 6y ago
If you think you can manage cloud deployments without skilled labour with cloud expertise, you are in for a surprise.
rstuart4133 6y ago
> The problem will be finding skilled labor. I think that overstates the problem. It does not require a whole pile of skill to purchase a few rack mount servers from Dell or Supermicro with onsite 24 hour warranty, and plonk them in a co-lo. In the rare event the hardware does break ring Dell and ask them to fix it for you. When the onsite warranty end's in 7 years it's time to replace the servers. The expertise required it literally minimal - not much beyond the ability to use a screw driver to install it into the rack and know how to connect the Ethernet cables. Then you have to plug in a USB and install whatever OS you want, of course, but you don't have to be onsite to do that. They all come with iDRAC's or the equivalent. They will cost about $1000/yr with maybe 10TB raid disk, co-lo costs of around another $1000/yr for unlimited bandwidth. To rent the same dedicated metal is about $500/mo from OHV where I live, so over twice the cost. Obviously, this is all impossible if you aren't big enough to have dedicated IT staff. And obviously, if you are likely to go through rapid change (well, something more extreme than adding a new server every now and then), it isn't the best plan. But for a stable mature business that employs several hundred people, all you are really doing is cutting out the middle man.

betaby 6y ago

There is a lot of spectrum in the middle between AWS micro and your own datacenter(s). One don't have to jump from the cloud to datacenter - just rent/buy a server or two, or rack of server or two.

blackaspen 6y ago
Sure, but the management overhead from 1 server to 20 racks scales, at best, linearly. I've seen a lot of places just past the point where a single sys-ad person with a thumb drive is viable, yet they choose to not invest here and instead spend 2-5x on going to the cloud.

PopeDotNinja 6y ago

Basically it makes sense to set up your own infrastructure when your business IS your infrastructure.

mooreds 6y ago
Same as lawyering or accounting, right? I don't do contracts regularly unless my business in contracts. Same with taxes. Just another form of specialization. MSPs and data center companies have been doing this since the 1990s at least, this is just the next evolution.

birdyrooster 6y ago

They aren’t shrinking at all, we have the same talent pool and are trying to thinly spread it over all of the problems we can solve with networked computers. The amount of problems you can solve with networked computers has scaled way faster than our industries capability to find talent and train employees.

ericbarrett 6y ago
Who is “we”? Always curious what remains of my field—I’ve worked in the storage and infrastructure industry for over 20 years. If you mean the industry as a whole, I simply disagree. The talent pipeline is not there anymore. You used to be able to get started without an Engineering degree by “getting in at the bottom.” Now somebody on this path will never get past power supply swaps because they’re just doing drudge work fed to them on a tablet (“replace disk 17 on row 23, rack 9, slot 4”—“oh, yup, there’s the yellow light”). If you do have the degree, there are a billion more profitable paths, and most take them.
- birdyrooster 6y ago
  We is engineers who work in datacenters. Colocation datacenters have existed for several decades and remote hands working there would probably not see much difference between getting a call/fax/email/app telling them to do a break fix order. Having worked in dozens of datacenters over the past two decades is that there will always be newer semi-technical workers there doing break fix. A lot of those people would consider those jobs dead ends too and so this cloud datacenter thing is nothing new, its just on a much larger scale. The datacenter operators don't have to keep working there to level up their careers but working in a datacenter gives familiarity and the confidence to try more technical roles later.

Ericson2314 6y ago

But it's insane that running a for-higher data center is considered a high-margin business. (And testament that the customers are VC-gorged price-unconcious baby gremlims.) In a sane economy, data centers for higher would be a fully-commoditized barely-profitable common carrier with little natural monopoly.

dodobirdlord 6y ago
Reliability concerns make datacenters resilient to commoditization. A datacenter that’s available 90% of the time is worth vastly, vastly less than 90% as much as one that 99.95% of the time. Commodity businesses are largely built on presumptions of linearity. Produce 90% as much corn/iron/wood/widgets as you expected and you’ll probably make something like 90% of the money you expected. Produce a 90% available datacenter and you’ll have a hard time finding anyone willing to pay you anything. And that’s just availability, not to mention data durability, which is even more critically nonlinear.
- echelon 6y ago
  Then you should be able to sue your provider for breaches in SLA. 0.5% downtime should void the bill. I think presently the providers are getting all the upside. High margin, perpetual lock-in, and no consequences.
  - freeone3000 6y ago
    You can sue your provider for breaches in SLA. The rebate you automatically get likely exceeds what you'd otherwise recover. (Which is why they grant one). SLA payouts are usually very generous - I've gotten credit for the entire month for a one-hour outage of a service. I don't know where this idea that cloud providers and other DCs don't pay out for SLA agreements is, but they absolutely do.
- dnautics 6y ago
  > A datacenter that’s available 90% of the time is worth vastly, vastly less than 90% as much as one that 99.95% of the time. That depends on your workload and the nature of the loss of availability. Is the completed work still there but just unreachable for an hour every few months? That might be okay for some folks. Is your data center always available mon-fri but constantly has scheduled downtime on weekends? Might still be okay.

hengheng 6y ago

I'm not in the web or cloud business, but I've filled a rack with my stuff before. My impression is that hardware has become a lot more capable even relative to its tasks. With high iops memory, many cores and obscene amounts of RAM, I would expect companies of a much larger scale (in $, FTEs, or most other metrics) can be served by one 4HE machine, or by one rack, or by one room. Thus I would expect the knowledge of how to handle 5000 hard drives to become more obscure, naturally, but the skill to run a decently sized web application to remain almost constant. Does this math work out, or have the tasks become more demanding at the same speed that hardware has improved?

spullara 6y ago
There are now businesses that explicitly depend on the elasticity of the cloud and can never really be moved on premise without massive up-front investment in hardware that may only be used a few times a year for their biggest customers. Trying to hybridize these workloads hasn't been very successful as of yet. It is possible that K8S could relive this problem but I haven't seen it in practice, at scale.
- thor24 6y ago
  Instant Elasticity in Cloud is a myth. If you think you are going to get 1k hosts just like that from AWS you will have an unpleasant experience. I work at one of the decent size tech company and we are split between cloud and on prem. From our experience you have to inform AWS/GCP in advance (sometime way early) if you are looking to meaningfully increase capacity in zone/region. Sure, auto scaling few hundreds of hosts may be possible but people who run a service which needs few hundreds of hosts run it directly on AWS, they will run it some kind of scheduler+resource manager which will have some kind of operational buffer anyway (as in you would already have those hosts so cloud elasticity is not a factor here).
  - echelon 6y ago
    Why not go fully on-prem then? You can run kubernetes locally. Are managed data stores that attractive? You can pay for on-prem management. What workloads are in the cloud versus on-prem?

waprin 6y ago

It's not just managing the complexities of managing the bare metal, although that's certainly a huge component of it. There's some other huge arguments against running your own datacenters. One is being able to properly provision resources. Being able to write just a function and have it consume just that tiny amount of resources rather than a whole VM is huge. Being able to spin instances up and down as you need them is huge. I think that's been obvious for a long time, but what I've seen less obvious to the business analysts is the impact of more advanced cloud services. The direction of cloud computing is managed services where they run your databases, container platforms, etc for you. Trying to run a huge Cassandra cluster or Kubernetes cluster takes up a ton of expensive labor's time and there's a good chance the cloud providers are a lot better at it than you. Sure, cloud services tend to be really expensive, and besides cost, there's also concern about things like vendor lock-in, IP protection/data privacy, and ability to tweak the small details of your platform. But cloud platforms in 2020 have a lot more features than in 2010 and in 2030 will have even more. The direction is pretty obvious. Running your own datacenters will be about as common as running your own power plant.

rualca 6y ago
> One is being able to properly provision resources. Being able to write just a function and have it consume just that tiny amount of resources rather than a whole VM is huge. Being able to spin instances up and down as you need them is huge. A 75 million dollar price tag is also huge. Bothering about the operational impact of a VM or a request sent to a function-as-a-service might be a significant operational issue if your whole team can be moved around with a small sports utility vehicle. Once you've grow over the point where your monthly cloud pricetag eclipses your company's paycheck budget, operating your own hardware is a no-brainer. > But cloud platforms in 2020 have a lot more features (...) That really doesn't matter at all, does it? I mean, cloud providers are already repackaging FLOSS services as their Serverless offering. And besides pursuing the latest fad, how many of those features are killed off and vanish from the face of the earth? It's always great if we can get others to do the work for us, but if we consider the absurd premium charged by cloud providers for their services... Well, those "others" doing the work can be employed by your company and you still save money.

Shorel 6y ago

You say "stubborn" but when the savings can be in the millions of dollars, it becomes logical to own a datacenter. I can't believe datacenter people suddenly all disappeared or are retired. If the pay scale is down, they will surely appreciate working for a big company with datacenter needs. Obviously, this only applies to huge companies like Dropbox. Everyone else is better served by AWS.

ericbarrett 6y ago
This is explicitly what I’m calling out at the end. They didn’t all disappear or retire; it’s an ongoing process. Think COBOL programmers.
- Shorel 6y ago
  Ha! I was also thinking about COBOL programmers when writing my comment.
jonfromsf 6y ago
You can rent racks of servers from a dedicated hosting provider and spend 10X less than AWS. You don't need to do you own wiring and HVAC and shit.

ryanmarsh 6y ago

all but the largest or most stubborn companies run their own datacenters are coming to a close You're right about the stubborn part, but not necessarily largest. Many IT leaders at companies of all sizes have their political capital tied up in the data center. What you're talking about doing (with cloud adoption) is outsourcing 90% of what they control. For them it's existential, uptime and agility be damned. Serverless is even worse for them, as far as IT fiefdoms are concerned.

mgiampapa 6y ago

Some of it is becoming more democratized too. Facebook as an example open sources it's datacenter design, it's servers, switches etc. I know they are the big bad for privacy these days, but the OCP project has contributed state of the art datacenter tech that legacy providers should be jumping at.

pjmlp 6y ago

If you think you can manage cloud deployments without skilled labour with cloud expertise, you are in for a surprise.

rstuart4133 6y ago

> The problem will be finding skilled labor. I think that overstates the problem. It does not require a whole pile of skill to purchase a few rack mount servers from Dell or Supermicro with onsite 24 hour warranty, and plonk them in a co-lo. In the rare event the hardware does break ring Dell and ask them to fix it for you. When the onsite warranty end's in 7 years it's time to replace the servers. The expertise required it literally minimal - not much beyond the ability to use a screw driver to install it into the rack and know how to connect the Ethernet cables. Then you have to plug in a USB and install whatever OS you want, of course, but you don't have to be onsite to do that. They all come with iDRAC's or the equivalent. They will cost about $1000/yr with maybe 10TB raid disk, co-lo costs of around another $1000/yr for unlimited bandwidth. To rent the same dedicated metal is about $500/mo from OHV where I live, so over twice the cost. Obviously, this is all impossible if you aren't big enough to have dedicated IT staff. And obviously, if you are likely to go through rapid change (well, something more extreme than adding a new server every now and then), it isn't the best plan. But for a stable mature business that employs several hundred people, all you are really doing is cutting out the middle man.