4 ms·

Ever been on a team where a key person responsible for a piece of infra leaves suddenly, and next incident takes hours to resolve instead of minutes as everyone

by lovecg 4y ago

Ever been on a team where a key person responsible for a piece of infra leaves suddenly, and next incident takes hours to resolve instead of minutes as everyone is running around trying to understand what the code is doing? It will be like that but way worse.

recuter 4y ago
Yes, I have actually. Let it take hours to resolve, what of it? Way better than keeping a person around who hoards institutional knowledge and admin access for job security. In fact that would be the first person I'd zero in on and get rid of, and have in the past. Not only are people like that not indispensable, they should never have been hired, anywhere. A well engineered twitter has everything necessary to run it in properly documented internal wikis and can be handed off to a small new group of competent ops people at the drop of a hat. Ideally new hires should be able to fix and change things within a day of starting their jobs with little to no hand holding. I'm sure it is very far from that at the moment due to the unprofessionalism of everybody involved, but this is actually the best and fastest way to get it there purely from an ops perspective - usually takes years to convince management to yank off the bandaid because everybody is too scared to take responsibility. Twitter used to have fail whales all the goddamn time, sky doesn't fall, it isn't amazon they aren't losing money every second it is down. Whoever stayed on will be able to figure out eventually which things to restart in what order so long as they aren't locked out of the servers. Unless Twitter is still using Zookeeper. :P In the process of figuring it out they'll be putting in-writing what should have been documented all along and getting a feel for the brittle bits. Long term twitter is better off. Good riddance to bad rubbish. I'm sure the databases have recent known good backups at worst you'll get a bunch out of prolonged outages and some dataloss - since we are talking about tweets, nothing of value was lost.
- thebruce87m 4y ago
  > at worst you'll get a bunch out of prolonged outages and some dataloss I hadn’t thought about what will happen next, but this is probably it. It’s fascinating watching the D-K effect play out at every level here. It might actually benefit engineers in the long term too, since they can ultimately blame Elon ( after he fires them ) and hopefully learn from it.
- wildrhythms 4y ago
  Domain expertise does not imply 'hoarding institutional knowledge.'
  - recuter 4y ago
    The random spaghetti of deployment scripts, the right JVM parameters that got derived from trial and error over time so things don't OOM, and what format the git tag should be for the CI branch are problems of ones own creation. Operating a Kafka cluster doesn't make one an expert at anything. Zilch. The site is still up and in a month or two Elon should cut another 90% of them and it will still be up. Some people will have to live down the discovery that they were never essential workers but rather seat fillers. https://en.wikipedia.org/wiki/Seat_filler https://en.wikipedia.org/wiki/Seat_filler

by lovecg 4y ago

recuter 4y ago

Yes, I have actually. Let it take hours to resolve, what of it? Way better than keeping a person around who hoards institutional knowledge and admin access for job security. In fact that would be the first person I'd zero in on and get rid of, and have in the past. Not only are people like that not indispensable, they should never have been hired, anywhere. A well engineered twitter has everything necessary to run it in properly documented internal wikis and can be handed off to a small new group of competent ops people at the drop of a hat. Ideally new hires should be able to fix and change things within a day of starting their jobs with little to no hand holding. I'm sure it is very far from that at the moment due to the unprofessionalism of everybody involved, but this is actually the best and fastest way to get it there purely from an ops perspective - usually takes years to convince management to yank off the bandaid because everybody is too scared to take responsibility. Twitter used to have fail whales all the goddamn time, sky doesn't fall, it isn't amazon they aren't losing money every second it is down. Whoever stayed on will be able to figure out eventually which things to restart in what order so long as they aren't locked out of the servers. Unless Twitter is still using Zookeeper. :P In the process of figuring it out they'll be putting in-writing what should have been documented all along and getting a feel for the brittle bits. Long term twitter is better off. Good riddance to bad rubbish. I'm sure the databases have recent known good backups at worst you'll get a bunch out of prolonged outages and some dataloss - since we are talking about tweets, nothing of value was lost.

thebruce87m 4y ago
> at worst you'll get a bunch out of prolonged outages and some dataloss I hadn’t thought about what will happen next, but this is probably it. It’s fascinating watching the D-K effect play out at every level here. It might actually benefit engineers in the long term too, since they can ultimately blame Elon ( after he fires them ) and hopefully learn from it.
wildrhythms 4y ago
Domain expertise does not imply 'hoarding institutional knowledge.'
- recuter 4y ago
  The random spaghetti of deployment scripts, the right JVM parameters that got derived from trial and error over time so things don't OOM, and what format the git tag should be for the CI branch are problems of ones own creation. Operating a Kafka cluster doesn't make one an expert at anything. Zilch. The site is still up and in a month or two Elon should cut another 90% of them and it will still be up. Some people will have to live down the discovery that they were never essential workers but rather seat fillers. https://en.wikipedia.org/wiki/Seat_filler https://en.wikipedia.org/wiki/Seat_filler