5 ms·

Aside from the rhetoric, the political opinions and people firing at you now over the 5-10K thing, I do have an actual question of substance. My assumption whe

Aside from the rhetoric, the political opinions and people firing at you now over the 5-10K thing, I do have an actual question of substance. My assumption when things melted down so drastically, was that the key problem was integration with the various vendors. Yes we all could look at the Html itself and make assumptions about poor practices but given that you are on the inside - what was the biggest issue?

brandonb 13y ago
The biggest general problem is that people thought of shipping the site the same way they thought of shipping an aircraft carrier -- you write the code, hand it over, and you're done. It wasn't treated as a running service. So, for example, when the site went down, there wasn't a group of people responsible for bringing it back up. That's what Mikey and the other Site Reliability Engineers fixed. They set up a war room with an engineer from each and every subcontractor, and the war room had three rules: Rule 1: "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame." Rule 2: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it." Rule 3: "We need to stay focused on the most urgent issues, like things that will hurt us in the next 24-48 hours." Once you have that process working, it's the same as optimizing software: you find the current bottleneck, fix it, find the next, etc. The Time article mentions two -- the lack of DB caching and the bad ID generator. There were dozens of things like that. And still are!
- keithwarren 13y ago
  What is the biggest issue right now architecturally?
  - brandonb 13y ago
    Complexity. We have 10x more code, 10x more components, and 10x more layers than we need. If the initial architecture had been dirt-simple, I don't think the site would have had so much trouble scaling or staying up. But, of course, removing complexity without breaking things takes longer than adding it in the first place. The other big problem is operations — many steps are done manually which should be done with tools like chef or puppet. When you have a lot of manual steps in your deploy process, it makes the whole system harder to scale, test, modify, and keep running. "Devops" has become a buzzword but it's definitely needed here.
    - keithwarren 13y ago
      Was there consideration given to doing a wholesale rewrite and later swapping out for that? Seems that if things are overly complex now, it may be easier to hold it together with band-aids while a true long term build is put together...what is going on behind the scenes with respect to this kind of planning?
    - keyhole_downs 13y ago
      Better call Scott!
- mbesto 13y ago
  > The biggest general problem is that people thought of shipping the site the same way they thought of shipping an aircraft carrier -- you write the code, hand it over, and you're done. So, the same problem every vendor encounters when doing a software project. I cannot tell you how many "lay people" can't simply grok this.
  - Perdition 13y ago
    It's amusing because that isn't the way aircraft carriers are made. Something like a tank might just get handed over after only manufacturer testing, but big ticket items like aircraft carriers go through a year plus of acceptance trials and testing. They don't finish the last coat of paint, load an air wing, and send it out on deployment.
    - twistedpair 13y ago
      Amen. Just watch any documentary about building an aircraft carrier. The systems integration phase is the longest part!
      - liyanchang 13y ago
        The difference here is that they had a very public deadline.
        keyhole_downs 13y ago
        The difference here is that politicians treat technology the way voters treat elections.
- grecy 13y ago
  My god I wish my company would instigate Rule 2 !

brandonb 13y ago

The biggest general problem is that people thought of shipping the site the same way they thought of shipping an aircraft carrier -- you write the code, hand it over, and you're done. It wasn't treated as a running service. So, for example, when the site went down, there wasn't a group of people responsible for bringing it back up. That's what Mikey and the other Site Reliability Engineers fixed. They set up a war room with an engineer from each and every subcontractor, and the war room had three rules: Rule 1: "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame." Rule 2: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it." Rule 3: "We need to stay focused on the most urgent issues, like things that will hurt us in the next 24-48 hours." Once you have that process working, it's the same as optimizing software: you find the current bottleneck, fix it, find the next, etc. The Time article mentions two -- the lack of DB caching and the bad ID generator. There were dozens of things like that. And still are!

keithwarren 13y ago
What is the biggest issue right now architecturally?
- brandonb 13y ago
  Complexity. We have 10x more code, 10x more components, and 10x more layers than we need. If the initial architecture had been dirt-simple, I don't think the site would have had so much trouble scaling or staying up. But, of course, removing complexity without breaking things takes longer than adding it in the first place. The other big problem is operations — many steps are done manually which should be done with tools like chef or puppet. When you have a lot of manual steps in your deploy process, it makes the whole system harder to scale, test, modify, and keep running. "Devops" has become a buzzword but it's definitely needed here.
  - keithwarren 13y ago
    Was there consideration given to doing a wholesale rewrite and later swapping out for that? Seems that if things are overly complex now, it may be easier to hold it together with band-aids while a true long term build is put together...what is going on behind the scenes with respect to this kind of planning?
  - keyhole_downs 13y ago
    Better call Scott!
mbesto 13y ago
> The biggest general problem is that people thought of shipping the site the same way they thought of shipping an aircraft carrier -- you write the code, hand it over, and you're done. So, the same problem every vendor encounters when doing a software project. I cannot tell you how many "lay people" can't simply grok this.
- Perdition 13y ago
  It's amusing because that isn't the way aircraft carriers are made. Something like a tank might just get handed over after only manufacturer testing, but big ticket items like aircraft carriers go through a year plus of acceptance trials and testing. They don't finish the last coat of paint, load an air wing, and send it out on deployment.
  - twistedpair 13y ago
    Amen. Just watch any documentary about building an aircraft carrier. The systems integration phase is the longest part!
    - liyanchang 13y ago
      The difference here is that they had a very public deadline.
      - keyhole_downs 13y ago
        The difference here is that politicians treat technology the way voters treat elections.
grecy 13y ago
My god I wish my company would instigate Rule 2 !