4 ms·

> Google SREs by last count were 1 engineer to 1000 machines That number does not seem particularly impressive, if it is accurate. Even "traditional" well-run

> Google SREs by last count were 1 engineer to 1000 machines That number does not seem particularly impressive, if it is accurate. Even "traditional" well-run enterprise IT organizations are often in the 1 admin/SRE to 600-ish machines, so I have a hard time seeing that Google can only do ~2x as good at their scale and with their level of focus. 1 SRE to 5k machines, 10k machines, that makes more sense to me.

skj 12y ago
You'll never see an accurate ratio, because revealing it would reveal the number of machines in Google data centers (considered a trade secret).
ryanburk 12y ago
having a ratio where you scale linearly is really not common in mega-scale web services since you have infrastructure services like borg / mesos / omega / autopilot / etc. you can take a snapshot and say you are 1:n because today you have so many sre and so many machines, but it is very unlikely to be the same ratio down the road.
devonkim 12y ago
I don't know which kinds of enterprise orgs you're envisioning but when I think about "traditional" enterprise IT orgs, I'm picturing companies in healthcare, insurance, certain finance business units, non-profits, public sector, and defense, not any place ever mentioned a lot here on HN in a technical discussion. I've worked with a LOT of them and I'd be surprised if the ratio of ops engineers to servers was anything better than 1:10 as a ball-park number. Network guy, storage guy, DB guy, VM guy, automation guy, net-sec guy, system-sec guy... that's pretty typical for maybe a 20-server rollout and grows shy of sub-linearly with number of users (wild guess of O(n^1/2) ). If you mean specific to SRE-type of roles or auto-scaled groups in EC2 / GCE to count as servers perhaps that may be the case, but my idea of a traditional enterprise IT org is that it views automation with fear and hesitation preferring to add more people before trying to "disrupt" their business with automation tools and so will be stuck somewhere around 1:30 at best. Admittedly, I probably have a bad impression of enterprise from consulting because who would pay for automation consulting at $$$ / hr when you do it pretty well with existing resources in the first place?
- WorldWideWayne 12y ago
  I think part of the difference is due to the fact that the business of administering healthcare, insurance, financials and other businesses are infinitely more complex than say, serving up search results or 140 character micro blog posts. It seems much easier in my opinion to scale a single function (search or tweet) than the kinds of tasks that a healthcare company has to do like say...scanning faxes from doctors, applying OCR and properly placing them into a pharmacy order system.
  - justincormack 12y ago
    The idea is that those services get commoditized, so the healthcare company gets them off the shelf rather than hires engineers to build them.
florianleibert 12y ago
Twitter recently stated they had roughly 1 SRE / 10K nodes.
jldugger 12y ago
It doesn't matter how high scale google gets, hard drives still fail at the same rate.
- presspot 12y ago
  Rip and replace of a hard drive is not hard when you have a datacenter that is completely self-healing. An intern on roller skates can do it.
spacecowboy_lon 12y ago
And goggle dont have to woory about the hard stuf in the same way that full on IBM sysplex thats running 20% of a coutrys bank acounts does. RBS COUGH COUGH
presspot 12y ago
the 1 admin to 600 machines quoted as the high end for traditional IT datacenter is, in my experience, a murky number. It's usually a ratio of people/virtual machines, not physical machines. When you remove the VM smokescreen and count physical boxes it's more like 1 person/100 machines, which is abysmal. I've seen order-of-magnitude people efficiency increases with automation like we're discussing here.

skj 12y ago

You'll never see an accurate ratio, because revealing it would reveal the number of machines in Google data centers (considered a trade secret).

ryanburk 12y ago

having a ratio where you scale linearly is really not common in mega-scale web services since you have infrastructure services like borg / mesos / omega / autopilot / etc. you can take a snapshot and say you are 1:n because today you have so many sre and so many machines, but it is very unlikely to be the same ratio down the road.

devonkim 12y ago

I don't know which kinds of enterprise orgs you're envisioning but when I think about "traditional" enterprise IT orgs, I'm picturing companies in healthcare, insurance, certain finance business units, non-profits, public sector, and defense, not any place ever mentioned a lot here on HN in a technical discussion. I've worked with a LOT of them and I'd be surprised if the ratio of ops engineers to servers was anything better than 1:10 as a ball-park number. Network guy, storage guy, DB guy, VM guy, automation guy, net-sec guy, system-sec guy... that's pretty typical for maybe a 20-server rollout and grows shy of sub-linearly with number of users (wild guess of O(n^1/2) ). If you mean specific to SRE-type of roles or auto-scaled groups in EC2 / GCE to count as servers perhaps that may be the case, but my idea of a traditional enterprise IT org is that it views automation with fear and hesitation preferring to add more people before trying to "disrupt" their business with automation tools and so will be stuck somewhere around 1:30 at best. Admittedly, I probably have a bad impression of enterprise from consulting because who would pay for automation consulting at $$$ / hr when you do it pretty well with existing resources in the first place?

WorldWideWayne 12y ago
I think part of the difference is due to the fact that the business of administering healthcare, insurance, financials and other businesses are infinitely more complex than say, serving up search results or 140 character micro blog posts. It seems much easier in my opinion to scale a single function (search or tweet) than the kinds of tasks that a healthcare company has to do like say...scanning faxes from doctors, applying OCR and properly placing them into a pharmacy order system.
- justincormack 12y ago
  The idea is that those services get commoditized, so the healthcare company gets them off the shelf rather than hires engineers to build them.

florianleibert 12y ago

Twitter recently stated they had roughly 1 SRE / 10K nodes.

jldugger 12y ago

It doesn't matter how high scale google gets, hard drives still fail at the same rate.

presspot 12y ago
Rip and replace of a hard drive is not hard when you have a datacenter that is completely self-healing. An intern on roller skates can do it.

spacecowboy_lon 12y ago

And goggle dont have to woory about the hard stuf in the same way that full on IBM sysplex thats running 20% of a coutrys bank acounts does. RBS COUGH COUGH

presspot 12y ago

the 1 admin to 600 machines quoted as the high end for traditional IT datacenter is, in my experience, a murky number. It's usually a ratio of people/virtual machines, not physical machines. When you remove the VM smokescreen and count physical boxes it's more like 1 person/100 machines, which is abysmal. I've seen order-of-magnitude people efficiency increases with automation like we're discussing here.