5 ms·

No sane infrastructure team will answer „easily“ to that question. Things break, all of the time. And infra will need developer input to figure out some of the

No sane infrastructure team will answer „easily“ to that question. Things break, all of the time. And infra will need developer input to figure out some of the breakage, let alone fix it. And that‘s not even considering deliberate attacks, fixes for critical security/stability issues etc.

erulabs 4y ago
I disagree strongly, and I currently run an infrastructure team. Things break because of changes to the system. Assuming the entire infrastructure team wasn't fired, it should be dramatically easier to keep the service online assuming all development has stopped.
- falcolas 4y ago
  That depends entirely on the status of the stack when the code freeze happened. If it was unstable, then I'd want several bottles of whisky on hand, because I'd have to work around the stinky pile of dung I'd been left with. Given reports of debug and dev builds in production just before everything was frozen, I'd probably want to start stocking up on whisky.
- albertopv 4y ago
  Changes to a system can come in many ways, e.g. we had a noticeable increase in our db size because of a new customer data, what was ok. before wasn't after. Of course it's different for twitter. And what about security? Update to libraries, dependencies...?
  - Xylakant 4y ago
    Anyone remember the leap second?
  - cloverich 4y ago
    Of course but this is also twitter, not a startup in its infancy. I'm not saying new stuff won't happen (etc), but they have a very mature infra org.
    - Kye 4y ago
      Do they? It's hard to tell where these cuts are happening and how much thought is going into them.
    - albertopv 4y ago
      And almost no one left to manage it.
Raz2 4y ago
Not during the code freeze. Using the same container image that worked before for a while doesn’t need dev input.
- Xylakant 4y ago
  I‘ve had stuff break because of required kernel updates, software that needed to be patched, hardware breaking and no exact substitute being available, database sizes crossing critical thresholds, … - it‘s definitely less risky to run something that is well understood and no longer getting substantial amounts of updates, but every complex system has known and unknown deficiencies that are just a butterfly’s wingbeat away from being triggered. As long as you‘re on the happy path, everything will look fine. But if something pushes slightly and moves you to the unhappy path, you can find yourself very quickly in a position where you need someone with a deep understanding of some obscure component of the system that is misbehaving - and that someone is unlikely to be the infrastructure team.

erulabs 4y ago

I disagree strongly, and I currently run an infrastructure team. Things break because of changes to the system. Assuming the entire infrastructure team wasn't fired, it should be dramatically easier to keep the service online assuming all development has stopped.

falcolas 4y ago
That depends entirely on the status of the stack when the code freeze happened. If it was unstable, then I'd want several bottles of whisky on hand, because I'd have to work around the stinky pile of dung I'd been left with. Given reports of debug and dev builds in production just before everything was frozen, I'd probably want to start stocking up on whisky.
albertopv 4y ago
Changes to a system can come in many ways, e.g. we had a noticeable increase in our db size because of a new customer data, what was ok. before wasn't after. Of course it's different for twitter. And what about security? Update to libraries, dependencies...?
- Xylakant 4y ago
  Anyone remember the leap second?
- cloverich 4y ago
  Of course but this is also twitter, not a startup in its infancy. I'm not saying new stuff won't happen (etc), but they have a very mature infra org.
  - Kye 4y ago
    Do they? It's hard to tell where these cuts are happening and how much thought is going into them.
  - albertopv 4y ago
    And almost no one left to manage it.

Raz2 4y ago

Not during the code freeze. Using the same container image that worked before for a while doesn’t need dev input.

Xylakant 4y ago
I‘ve had stuff break because of required kernel updates, software that needed to be patched, hardware breaking and no exact substitute being available, database sizes crossing critical thresholds, … - it‘s definitely less risky to run something that is well understood and no longer getting substantial amounts of updates, but every complex system has known and unknown deficiencies that are just a butterfly’s wingbeat away from being triggered. As long as you‘re on the happy path, everything will look fine. But if something pushes slightly and moves you to the unhappy path, you can find yourself very quickly in a position where you need someone with a deep understanding of some obscure component of the system that is misbehaving - and that someone is unlikely to be the infrastructure team.