5 ms·

Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issue

Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issues. Not mopping up after them. To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.

tempestn 4y ago
I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by the parent, not trying to make one myself.)
- enkid 4y ago
  So 1-3 people have a big impact, the other 7997 must not be doing anything? I don't think that logic follows.
  - twelve40 4y ago
    it doesn't follow. The article posits that "many people think twitter headcount was bloated" then proceeds to describe a (presumably) really efficient work of a small SRE team. These two parts seem completely disconnected from each other - neither one proves, disproves or follows from the other - so it's unclear why the former was mentioned at all.
    - enkid 4y ago
      His personal experience was zero bloat. He was one deep in a critical function for the company. He isn't saying this proves without a doubt that there is no bloat, but he didn't see any in his time there. It's seems like a reasonable addition to the conversation to me.
      - Xunjin 4y ago
        It's a shame that most the conversation going on here are extrapolated arguments based on this article and another anecdotes. The problem starts when the ones making their points let beliefs on "how things should be" stronger instead of "how they really were".
- pcthrowaway 4y ago
  The SRE team mentioned in the article is SRE for one component of a complex architecture. There are probably many such components; I'd imagine SRE alone would be 200+ people How many of the remaining staff have the knowledge required to keep all of those components running smoothly?
  - adql 4y ago
    Once you get the automation going the number itself doesn't matter that much. You might have 200 different apps (hell, we have close to that, only 3 people in ops) but competent team will make sure they deploy in same way and are monitored in same way. And once you go from "a server" to multiple servers, whether the end number ends up being 20 or 200 isn't that important till you start hitting say switching capacity, and if you're in cloud that's usually not your concern anyway. Our biggest site (about dozen million users, a bunch of services and caching underneath, few gbits of traffic) took zero actual maintenance for 2022, "it just works", any job was implementing new stuff. It took some time to get to that state but once you do aside from hardware failures it "runs itself"
    - darkwater 4y ago
      > Our biggest site (about dozen million users, a bunch of services and caching underneath, few gbits of traffic) took zero actual maintenance for 2022, "it just works", any job was implementing new stuff. It took some time to get to that state but once you do aside from hardware failures it "runs itself" Nobody is adding changes that blows out the DB? or add some inefficient code that burns CPU much faster?
      - chasd00 4y ago
        > Nobody is adding changes that blows out the DB? or add some inefficient code that burns CPU much faster? not to be flippant but you find that out like 3 environments below production.
- TheOtherHobbes 4y ago
  It's not 1-3 people. The entire SRE team globally - including the technicians and the engineers with server access - is easily going to be in the hundreds. The SRE manager is in charge of keeping it all running. He isn't running around the world swapping out servers. He also isn't sitting back with his feet up thinking "All done - now how are my Pokemon doing?" It's a dynamic process with quality monitoring, budgeting and reports, post-mortems, continual experiments to see if uptime can be improved, and redesigns as hardware and software change. It's part of the backend, but is only loosely coupled to the content management and delivery system, the ad machine, moderation, marketing, and so on, all of which are going to have similarly complex structures.
b3lvedere 4y ago
Whenever clients complain about those costs and efforts, i tell them it's just like their car. Your car is working perfectly fine so why should you pay for maintenance?
- john_the_writer 4y ago
  software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly.
  - hgsgm 4y ago
  - pmoriarty 4y ago
    "software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly." In an organization of any appreciable size, things change all the time.. and I'm not just talking about code (for which you could have a code freeze in an emergency situation like this), but the external systems you're connected to could change for reasons completely out of your control. Content changes can break stuff because of bugs in your code. Legacy systems could require all sorts of ongoing tweaking and maintenance. And, yes, heat can break your software if the server it's running on overheats. Twitter is not a palm_os app.
    - john_the_writer 4y ago
      Agreed.. but lets say you fire 99% of your engineers, and declare a code freeze (because there's no-one left to write code).. Then in theory.. if you own the hardware and you've locked down the libraries... That code could keep running for a long time. Agreed it's not a Palm app, but with everything locked down, I'd argue it's safe But now I can third party stuff changing. Payment processors and such. Those don't happen fast though, and 100% not so fast that a company the size of twitter can't work out a sunsetting. To the heat can break software if the server it's running on overheats. I have a feeling twitter's has a system in place to scale out the faulty server. My point was, comparing code to a car is silly. A car needs maintenance. Code in code freeze does not.
ryanfreeborn 4y ago
No, I think the article makes it very clear what the value and function of SRE is. The point of the comment you're responding to is that the author was the only one doing this—not a team of ten, not even a team of two. This is Twitter's whole cache system! Probably the most important part of their hardware stack, in terms of "is the site performing well for users". There are other SRE needs at Twitter, but not that that many. What were the other 9k people at the company? It begs the question.
foton1981 4y ago
To be honest I was very surprised to hear what a cache SRE was working on. It sounded like he had to build all of handling of hardware issues, rack awareness and other basic datacenter stuff himself. Does it mean that every specialized team also had to do it? Why would cache engineer need to know about hardware failures at all, its datacenter team's responsibility to detect and predict issues and shutdown servers gracefully if possible. It should be completely abstracted from cache SRE, like cloud abstracts you from it. Yet he and is team spends years on automation around this stuff using Mesos stack that they probably regret adopting by now. I feel like in this zoomed in case of twitter caches what they were working on is questionable, but the team size seems to be adequate to the task, so my takeaway is that like any older, larger company Twitter accumulated fair amount of tech debt and there is no one to take large scale initiative to eliminate it.

tempestn 4y ago

I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by the parent, not trying to make one myself.)

enkid 4y ago
So 1-3 people have a big impact, the other 7997 must not be doing anything? I don't think that logic follows.
- twelve40 4y ago
  it doesn't follow. The article posits that "many people think twitter headcount was bloated" then proceeds to describe a (presumably) really efficient work of a small SRE team. These two parts seem completely disconnected from each other - neither one proves, disproves or follows from the other - so it's unclear why the former was mentioned at all.
  - enkid 4y ago
    His personal experience was zero bloat. He was one deep in a critical function for the company. He isn't saying this proves without a doubt that there is no bloat, but he didn't see any in his time there. It's seems like a reasonable addition to the conversation to me.
    - Xunjin 4y ago
      It's a shame that most the conversation going on here are extrapolated arguments based on this article and another anecdotes. The problem starts when the ones making their points let beliefs on "how things should be" stronger instead of "how they really were".
pcthrowaway 4y ago
The SRE team mentioned in the article is SRE for one component of a complex architecture. There are probably many such components; I'd imagine SRE alone would be 200+ people How many of the remaining staff have the knowledge required to keep all of those components running smoothly?
- adql 4y ago
  Once you get the automation going the number itself doesn't matter that much. You might have 200 different apps (hell, we have close to that, only 3 people in ops) but competent team will make sure they deploy in same way and are monitored in same way. And once you go from "a server" to multiple servers, whether the end number ends up being 20 or 200 isn't that important till you start hitting say switching capacity, and if you're in cloud that's usually not your concern anyway. Our biggest site (about dozen million users, a bunch of services and caching underneath, few gbits of traffic) took zero actual maintenance for 2022, "it just works", any job was implementing new stuff. It took some time to get to that state but once you do aside from hardware failures it "runs itself"
  - darkwater 4y ago
    > Our biggest site (about dozen million users, a bunch of services and caching underneath, few gbits of traffic) took zero actual maintenance for 2022, "it just works", any job was implementing new stuff. It took some time to get to that state but once you do aside from hardware failures it "runs itself" Nobody is adding changes that blows out the DB? or add some inefficient code that burns CPU much faster?
    - chasd00 4y ago
      > Nobody is adding changes that blows out the DB? or add some inefficient code that burns CPU much faster? not to be flippant but you find that out like 3 environments below production.
TheOtherHobbes 4y ago
It's not 1-3 people. The entire SRE team globally - including the technicians and the engineers with server access - is easily going to be in the hundreds. The SRE manager is in charge of keeping it all running. He isn't running around the world swapping out servers. He also isn't sitting back with his feet up thinking "All done - now how are my Pokemon doing?" It's a dynamic process with quality monitoring, budgeting and reports, post-mortems, continual experiments to see if uptime can be improved, and redesigns as hardware and software change. It's part of the backend, but is only loosely coupled to the content management and delivery system, the ad machine, moderation, marketing, and so on, all of which are going to have similarly complex structures.

b3lvedere 4y ago

Whenever clients complain about those costs and efforts, i tell them it's just like their car. Your car is working perfectly fine so why should you pay for maintenance?

john_the_writer 4y ago
software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly.
- hgsgm 4y ago
- pmoriarty 4y ago
  "software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly." In an organization of any appreciable size, things change all the time.. and I'm not just talking about code (for which you could have a code freeze in an emergency situation like this), but the external systems you're connected to could change for reasons completely out of your control. Content changes can break stuff because of bugs in your code. Legacy systems could require all sorts of ongoing tweaking and maintenance. And, yes, heat can break your software if the server it's running on overheats. Twitter is not a palm_os app.
  - john_the_writer 4y ago
    Agreed.. but lets say you fire 99% of your engineers, and declare a code freeze (because there's no-one left to write code).. Then in theory.. if you own the hardware and you've locked down the libraries... That code could keep running for a long time. Agreed it's not a Palm app, but with everything locked down, I'd argue it's safe But now I can third party stuff changing. Payment processors and such. Those don't happen fast though, and 100% not so fast that a company the size of twitter can't work out a sunsetting. To the heat can break software if the server it's running on overheats. I have a feeling twitter's has a system in place to scale out the faulty server. My point was, comparing code to a car is silly. A car needs maintenance. Code in code freeze does not.

ryanfreeborn 4y ago

No, I think the article makes it very clear what the value and function of SRE is. The point of the comment you're responding to is that the author was the only one doing this—not a team of ten, not even a team of two. This is Twitter's whole cache system! Probably the most important part of their hardware stack, in terms of "is the site performing well for users". There are other SRE needs at Twitter, but not that that many. What were the other 9k people at the company? It begs the question.

foton1981 4y ago

To be honest I was very surprised to hear what a cache SRE was working on. It sounded like he had to build all of handling of hardware issues, rack awareness and other basic datacenter stuff himself. Does it mean that every specialized team also had to do it? Why would cache engineer need to know about hardware failures at all, its datacenter team's responsibility to detect and predict issues and shutdown servers gracefully if possible. It should be completely abstracted from cache SRE, like cloud abstracts you from it. Yet he and is team spends years on automation around this stuff using Mesos stack that they probably regret adopting by now. I feel like in this zoomed in case of twitter caches what they were working on is questionable, but the team size seems to be adequate to the task, so my takeaway is that like any older, larger company Twitter accumulated fair amount of tech debt and there is no one to take large scale initiative to eliminate it.