4 ms·

It's interesting and great to see this stuff take center stage more and more. Those lucky enough to work at places like Twitter, Google, Facebook and other larg

by chuhnk 12y ago

It's interesting and great to see this stuff take center stage more and more. Those lucky enough to work at places like Twitter, Google, Facebook and other large tech companies will have already seen how this kind of thing dominates the datacenter's there and has been at the core of their systems for many years. People on the outside though are rarely exposed to this concept of datacenter scale computing aside from things like hadoop and so the idea of leveraging a datacenter level API is hard to fathom. I believe it's still going to be a while before this truly becomes mainstream and really we'll see the historical two level forward shift whereby on one side we've previously achieved compute as a service with EC2 and GCE, while developing platform as a service on the other side with Google App Engine and Heroku. These service different demographics and we'll continue to see the level of abstractions occur providing SREs/Devops engineers access to Cluster As A Service and microservice platforms for developers/programmers to give them more flexibility. Google SREs by last count were 1 engineer to 1000 machines. In 10 years the common devops engineer at a 200 person startup will leverage the same number of resources using layers of abstraction like Mesos and Kubernetes.

mattzito 12y ago
> Google SREs by last count were 1 engineer to 1000 machines That number does not seem particularly impressive, if it is accurate. Even "traditional" well-run enterprise IT organizations are often in the 1 admin/SRE to 600-ish machines, so I have a hard time seeing that Google can only do ~2x as good at their scale and with their level of focus. 1 SRE to 5k machines, 10k machines, that makes more sense to me.
- skj 12y ago
  You'll never see an accurate ratio, because revealing it would reveal the number of machines in Google data centers (considered a trade secret).
- ryanburk 12y ago
  having a ratio where you scale linearly is really not common in mega-scale web services since you have infrastructure services like borg / mesos / omega / autopilot / etc. you can take a snapshot and say you are 1:n because today you have so many sre and so many machines, but it is very unlikely to be the same ratio down the road.
- devonkim 12y ago
  I don't know which kinds of enterprise orgs you're envisioning but when I think about "traditional" enterprise IT orgs, I'm picturing companies in healthcare, insurance, certain finance business units, non-profits, public sector, and defense, not any place ever mentioned a lot here on HN in a technical discussion. I've worked with a LOT of them and I'd be surprised if the ratio of ops engineers to servers was anything better than 1:10 as a ball-park number. Network guy, storage guy, DB guy, VM guy, automation guy, net-sec guy, system-sec guy... that's pretty typical for maybe a 20-server rollout and grows shy of sub-linearly with number of users (wild guess of O(n^1/2) ). If you mean specific to SRE-type of roles or auto-scaled groups in EC2 / GCE to count as servers perhaps that may be the case, but my idea of a traditional enterprise IT org is that it views automation with fear and hesitation preferring to add more people before trying to "disrupt" their business with automation tools and so will be stuck somewhere around 1:30 at best. Admittedly, I probably have a bad impression of enterprise from consulting because who would pay for automation consulting at $$$ / hr when you do it pretty well with existing resources in the first place?
  - WorldWideWayne 12y ago
    I think part of the difference is due to the fact that the business of administering healthcare, insurance, financials and other businesses are infinitely more complex than say, serving up search results or 140 character micro blog posts. It seems much easier in my opinion to scale a single function (search or tweet) than the kinds of tasks that a healthcare company has to do like say...scanning faxes from doctors, applying OCR and properly placing them into a pharmacy order system.
    - justincormack 12y ago
      The idea is that those services get commoditized, so the healthcare company gets them off the shelf rather than hires engineers to build them.
- florianleibert 12y ago
  Twitter recently stated they had roughly 1 SRE / 10K nodes.
- jldugger 12y ago
  It doesn't matter how high scale google gets, hard drives still fail at the same rate.
  - presspot 12y ago
    Rip and replace of a hard drive is not hard when you have a datacenter that is completely self-healing. An intern on roller skates can do it.
- spacecowboy_lon 12y ago
  And goggle dont have to woory about the hard stuf in the same way that full on IBM sysplex thats running 20% of a coutrys bank acounts does. RBS COUGH COUGH
- presspot 12y ago
  the 1 admin to 600 machines quoted as the high end for traditional IT datacenter is, in my experience, a murky number. It's usually a ratio of people/virtual machines, not physical machines. When you remove the VM smokescreen and count physical boxes it's more like 1 person/100 machines, which is abysmal. I've seen order-of-magnitude people efficiency increases with automation like we're discussing here.
stevedomin 12y ago
I agree with you but I don't think we'll have to wait 10 years to see that happening. We're still a fairly small startup but we're transitioning our whole platform to be built around Mesos (and we're definitely not the only ones in that case). I don't know why many people think that they need to be at datacenter scale computing to benefit from abstractions like Mesos, it's completely wrong imo. It's quite a big shift in mindset but it makes the life of everyone (dev and ops) so much easier when you stop having to think about single machines.
- presspot 12y ago
  >>> I don't know why many people think that they need to be at datacenter scale computing to benefit from abstractions like Mesos, it's completely wrong imo My next startup will be built on Mesosphere. Faster time to MVP and no "go dark for 18 months" when I have to scale.

by chuhnk 12y ago

mattzito 12y ago

> Google SREs by last count were 1 engineer to 1000 machines That number does not seem particularly impressive, if it is accurate. Even "traditional" well-run enterprise IT organizations are often in the 1 admin/SRE to 600-ish machines, so I have a hard time seeing that Google can only do ~2x as good at their scale and with their level of focus. 1 SRE to 5k machines, 10k machines, that makes more sense to me.

skj 12y ago
You'll never see an accurate ratio, because revealing it would reveal the number of machines in Google data centers (considered a trade secret).
ryanburk 12y ago
having a ratio where you scale linearly is really not common in mega-scale web services since you have infrastructure services like borg / mesos / omega / autopilot / etc. you can take a snapshot and say you are 1:n because today you have so many sre and so many machines, but it is very unlikely to be the same ratio down the road.
devonkim 12y ago
I don't know which kinds of enterprise orgs you're envisioning but when I think about "traditional" enterprise IT orgs, I'm picturing companies in healthcare, insurance, certain finance business units, non-profits, public sector, and defense, not any place ever mentioned a lot here on HN in a technical discussion. I've worked with a LOT of them and I'd be surprised if the ratio of ops engineers to servers was anything better than 1:10 as a ball-park number. Network guy, storage guy, DB guy, VM guy, automation guy, net-sec guy, system-sec guy... that's pretty typical for maybe a 20-server rollout and grows shy of sub-linearly with number of users (wild guess of O(n^1/2) ). If you mean specific to SRE-type of roles or auto-scaled groups in EC2 / GCE to count as servers perhaps that may be the case, but my idea of a traditional enterprise IT org is that it views automation with fear and hesitation preferring to add more people before trying to "disrupt" their business with automation tools and so will be stuck somewhere around 1:30 at best. Admittedly, I probably have a bad impression of enterprise from consulting because who would pay for automation consulting at $$$ / hr when you do it pretty well with existing resources in the first place?
- WorldWideWayne 12y ago
  I think part of the difference is due to the fact that the business of administering healthcare, insurance, financials and other businesses are infinitely more complex than say, serving up search results or 140 character micro blog posts. It seems much easier in my opinion to scale a single function (search or tweet) than the kinds of tasks that a healthcare company has to do like say...scanning faxes from doctors, applying OCR and properly placing them into a pharmacy order system.
  - justincormack 12y ago
    The idea is that those services get commoditized, so the healthcare company gets them off the shelf rather than hires engineers to build them.
florianleibert 12y ago
Twitter recently stated they had roughly 1 SRE / 10K nodes.
jldugger 12y ago
It doesn't matter how high scale google gets, hard drives still fail at the same rate.
- presspot 12y ago
  Rip and replace of a hard drive is not hard when you have a datacenter that is completely self-healing. An intern on roller skates can do it.
spacecowboy_lon 12y ago
And goggle dont have to woory about the hard stuf in the same way that full on IBM sysplex thats running 20% of a coutrys bank acounts does. RBS COUGH COUGH
presspot 12y ago
the 1 admin to 600 machines quoted as the high end for traditional IT datacenter is, in my experience, a murky number. It's usually a ratio of people/virtual machines, not physical machines. When you remove the VM smokescreen and count physical boxes it's more like 1 person/100 machines, which is abysmal. I've seen order-of-magnitude people efficiency increases with automation like we're discussing here.

stevedomin 12y ago

I agree with you but I don't think we'll have to wait 10 years to see that happening. We're still a fairly small startup but we're transitioning our whole platform to be built around Mesos (and we're definitely not the only ones in that case). I don't know why many people think that they need to be at datacenter scale computing to benefit from abstractions like Mesos, it's completely wrong imo. It's quite a big shift in mindset but it makes the life of everyone (dev and ops) so much easier when you stop having to think about single machines.

presspot 12y ago
>>> I don't know why many people think that they need to be at datacenter scale computing to benefit from abstractions like Mesos, it's completely wrong imo My next startup will be built on Mesosphere. Faster time to MVP and no "go dark for 18 months" when I have to scale.