4 ms·
I really like 0xide's insight here. I don't know if Bryan has an MBA but this article reminds me of how I learned that good systems engineers make for good prod
by ChuckMcM 3y ago
I really like 0xide's insight here. I don't know if Bryan has an MBA but this article reminds me of how I learned that good systems engineers make for good product people.
A friend of mine from Sun went to Santa Clara university for their MBA and I was always interested in running a company and we talked about what core knowledge the school felt that a "Master" of business administration should know. Not too surprisingly there was a lot about how you could organize a bunch of unreliable elements (employees) into a structure that reliably delivered results (products). And in one sense that is exactly what cloud computing did as well, organizing unreliable "cheap" PC type servers, into a fabric of reliable service delivery. It is the system of organizing the parts that is the solution.
Greg Lindahl was one of the founders of Blekko and his experience of managing hundreds of machines for the super computer types with big machine budgets and low IT budgets, was essential to Blekko's ability to deliver its search engine. With a DevOps group of six (one manager and five SREs) Blekko managed 2000+ machines in two data centers. IBM was astonished when they bought us that we could manage with so few devops engineers.
We had also done the "What would this cost us on AWS" calculation many times with many different variables and the 'break even' point, where the cost to self host at a Colocation facility fell below the cost to host with AWS was 120 servers. And that included reducing the DevOps group size to 3. So about 10 racks worth of servers. There was a really funny experience when IBM insisted we move our infrastructure to SoftLayer (an IBM business) when IBM acquired us and we pointed out that the "soft money" cost of running our infrastructure on SoftLayer was about $9M a month versus $120K a month. Which their finance group shut down that talk and just renewed the lease in the Colo.
But to successfully run a big distributed thing like that, you needed to both put your servers in a colocation facility and have Greg's software which allowed you to manage them with a small team. The total cost of that infrastructure depended on it.
I got a chance to go visit the 0xide team and they absolutely "got" that requirement. If I were running Engineering and Operations in Blekko today we totally would have kicked Supermicro to the curb and replaced them with the 0xide solution. I don't know if they have teamed up with Antithesis for testing their stuff but man, that combo of management software that never breaks and integrated server cabinets with all the things? That's a pretty good integration.
- mixmastamyk 3y agoWhat was the secret sauce?
- ChuckMcM 3y agoperl
- mixmastamyk 3y agoHah, was imagining some kind of organization breakthrough… “we wrote scripts,” not what I expected.
- ChuckMcM 3y agoAh but it's really encapsulating a decade plus of running clusters on limited resources into scripts that automated nearly all of the detective work on fault analysis, resulting in very clearly signaling of problems. Combined with good process for managing "microfaults" (Greg's term not mine) led to a well instrumented (only kept the data that was important for diagnosing the faults) and robust in the presence of nearly any failure.
- gary_0 3y ago> I don't know if Bryan has an MBA An MBA from the Sun Microsystems School of Hard Knocks, maybe. Otherwise I don't think so.
- bcantrill 3y agoHa, you're right on both counts. I really love the case study approach for MBAs, though -- and I feel like one of the best things we read in preparing for Oxide was the Stanford GSB case study on Nebula[0]. [0] https://www.gsb.stanford.edu/faculty-research/case-studies/nebula-fall-company https://www.gsb.stanford.edu/faculty-research/case-studies/n...