22 ms·

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incre

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. If a product is mature enough, like Twitter seems to be, removing engineers won't instantly crash the product. It'll happen slowly. Bugs will creep in, because less time is spent on review and over all architecture. Security issues will creep in because of about the same issues and less oversight. Then, once this causes enough issues for the product to actually crash, the right people to fix it quickly might not be there anymore. That's when fixing the issues suddenly takes a lot more time. If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. Especially with Elon pushing for new features to be implemented quickly, inevitably by people who cannot fully understand the implications of said features, because 80% of knowledge is missing.

ALittleLight 4y ago
I think Twitter was stuck in a not-especially profitable niche. They shift into fast-mode to get out of it and find a better spot, then they can shift back into stable mode once they occupy a better equilibrium. That said, there are lots of bugs in Twitter now, today, when they presumably had the benefit of being in stable mode for a long time. For example, Twitter regularly refreshes and loads new tweets while I'm reading them, pushing the tweet I was in the middle of reading out of view. That seems like a pretty silly bug to exist in a mature product. I regularly reach a state where I have to kill the app and relaunch it because all of the "back" commands just minimize the app instead of taking me back to the timeline. I could go on.
- groffee 4y ago
  That's not a bug, it's a 'feature', tons of sites do it for whatever asinine reason.
  - ldng 4y ago
    Like ... displaying new / refreshing ads ?
  - rjh29 4y ago
    Keeping vertical scroll position stable while asynchronously loading lots of randomly-sized content is harder than it seems.
    - mschuster91 4y ago
      Yeah, but there's always the option of showing a "fallback" when loading and inserting content in the middle of a feed - one link at the top "go to last bottom of top" and one at the bottom "go to last top of bottom".
    - vagrantJin 4y ago
      A complicated feature I am 100% certain no user ever asked for.
      - valeness 4y ago
        Actually twitter does it better than anyone else and it's the reason I use it over other apps. The User Experience is unparalleled
- WA 4y ago
  They replaced their somewhat productive engineering workforce with completely unproductive interest payments. I’m not convinced that this will lead to a better spot. But regarding the bugs, I’m totally with you. Same here. I use Twitter only in the browser. Browse long enough and the page reloads as if it ran out of memory.
kneebonian 4y ago
Based on the recent disclosures it seems that any new security issues pales in comparison to the current ones.
resonious 4y ago
> how does it persuade you of that? By flowing from many people think it's bloat - I'll tell you what's really going on to tiny team of 1~3 built whole infra for critical component. I'm not really trying to make commentary on whether or not Twitter engineering was bloat, or whether or not I think it'll hit problems in the future. Just commenting on the fact that the article broke my expectations a little bit as a reader.
- btown 4y ago
  https://twitter.com/atax1a/status/1594880931042824192 https://twitter.com/atax1a/status/1594880931042824192 provides a bit more context and nuance (albeit with a sardonic tone, but it's hard to blame someone for being sardonic who saw their entire professional community fired). There's no doubt that OP built a great and stable automation layer on top of Mesos for caching workloads. But there are numerous other types of workloads on top of Mesos (including, I presume mission-critical database deployments that need well-disciplined draining protocols to shift between nodes), as well as administrative needs for the Mesos-to-infrastructure level, and things running on bare metal below the Mesos level. These things all needed dedicated SREs, and the absence of these SREs could result in a scenario like the one mentioned in the Twitter thread I linked - two obscure mutually-dependent components expire and cannot be re-provisioned using documented tools. I also think an important meta-point is that when Twitter was bringing in substantial revenue from advertising, every minute of downtime would have significant costs - costs that could make it easily worthwhile to "over-provision" SRE talent. With advertisers pausing engagement, perhaps Twitter loses less money from a day-long outage than it would save having the right talent to turn a day-long outage into a minutes-long outage. Twitter is only judged by its profitability (namely, Musk's ability to service debt without selling more Tesla stock than he already has), while most other tech companies (both public and private) are judged by both profitability and revenue growth. If you want both, larger SRE teams, to say nothing of feature development and regulatory compliance teams, start to make a lot more sense.
deleted 4y ago
[deleted]
fnord123 4y ago
Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?
- orwin 4y ago
  I worked for a bank with 28000 total engineers. I know because part of what i did was crunching github and gitlab data and detecting double accounts (and deleting account from people not there anymore). I'd say 3/4th of them were just cost of doing business. Including me (my role wasn't really necessary, the stuff i did was cool and everything, but a bit useless).
  - Xylakant 4y ago
    At this scale, having someone take care of deleting double/expired accounts is just good hygiene and I would not consider that useless. Forgotten accounts are a security risk.
    - papito 4y ago
      The bigger question is why such a large bank does not have solid off-boarding procedures.
      - Xylakant 4y ago
        Any off-boarding procedure fails - at that scale, if even one in one thousand fails, you can expect a couple of failures per year. ensuring these get caught is part of "solid offboarding".
      - orwin 4y ago
        They had that on the financial side. The eng/HR side was behind for sure.
      - anon400232 4y ago
bryanrasmussen 4y ago
>Bugs will creep in, https://twitter.com/IlluminatiGanga/status/1594609790432444417 https://twitter.com/IlluminatiGanga/status/15946097904324444... new members joining in 1970. hmmm.
holoduke 4y ago
How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.
- kakwa_ 4y ago
  That's likely optimistic. Maybe the user facing site, but that's just the tip of the iceberg. There are plenty of internal/backend/restricted systems to support and/or monetize this part. And that's not counting the huge number of support people & moderators needed.
  - RivieraKid 4y ago
    Frontend, Android app and iOS app should be doable by 1 person each. Then one graphic designer and a project manager for all of the 3 frontends.
    - benj111 4y ago
      Well graphics design isn't hard. I'm sure one of the Devs could do that. And I'm sure there are tools to convert an iOS app to/from Android, so one dev can do both. And a website is easy. You could do it with 1 person. But Elon is such a machine, he could keep it running by himself.
      - RivieraKid 4y ago
        Well that's what I did, I made both iOS and Android version of an app and some of the design. I mean, a mediocre complexity app doesn't need more than one developer for a "slow improvement mode".
        benj111 4y ago
        But Twitter needs more than 5 employees. How many bug reports do you think the Twitter site and apps get each day? How many employees do you think it takes just to triage them?
wikfwikf 4y ago
data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?
- _3u10 4y ago
  No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…
  - wikfwikf 4y ago
    There are tens of thousands of engineers maintaining and supporting postgres and clickhouse in all the organizations where they are used, the vast majority of which are not on Twitter's scale.
    - pmoriarty 4y ago
      Tens of thousands across all the organizations where postgres and clickhouse are used, maybe (though even then probably not!). Not tens of thousands at any one organization supporting postgres and clickhouse. No single organization needs even hundreds of people to support these apps. You just need a good architecture and a handful of dba's, developers, and sysadmins... maybe.. depending on your scale. At many smaller orgs you can probably get away with one.
      - rvnx 4y ago
        This 100%. You don't need 10'000 engineers to run billions of queries. It's not like the SREs are running the queries themselves. Your job as an SRE is to design and run computer systems, not to become the computing platform yourself.
        _3u10 4y ago
birdyrooster 4y ago
They will stagnate and have more failed launches than Space X.
- pclmulqdq 4y ago
  There's something poetic about NASA sending a rocket to the moon at the same time as Elon Musk is fucking around with Twitter instead of sending a SpaceX rocket there too.
  - chasd00 4y ago
    Even in all the complexity of spaceflight, Twitter is the single greatest risk to SpaceX's mission. I would laugh but i'm crying.
mihaaly 4y ago
> The article touches a really small [...] subject. That's how it couldn't prove the claim.
ivanhoe 4y ago
> If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. It's not like Twitter was bug free before. How many times it annoyingly refreshed the timeline while I was reading something, or when it shows notification that it failed to send the DM, and when you retry it says "you've already wrote this", or you open the reply dialog, but it freezes, has no send button at all, so you have to re-open it. All of this was happening to me pretty regularly long before Elon came along. As we all know, just hiring more people is not necessarily the solution to every problem, and to me it seems it was exactly what Twitter tried to do in the past. Now they deconstructed it to the bare bones, which will clearly show what are the core problems and requirements. They basically turned Twitter back into a startup. And from that new starting point they can hire again to cover the needs as they arise. If they succeed it will be a huge success as they'll end up with far more optimal team (and huge savings), and of course, if they fail to catch up with problems it will be a huge failure. We'll see how well Musk can manage it...
- TheOtherHobbes 4y ago
  They basically gave an established global corporation with around 200mn active users a startup tech, marketing, and support stack. Anyone thinking "huge success" is unrealistically optimistic, IMO. It's going to be another MySpace/AOL/Bebo, added to the list of dumbest purchases ever. And that's still going to be true if the point was to destroy the original community and replace it with a different political orientation.
  - abirch 4y ago
    You're right on this potentially being the worse purchase of all time. The purchase price of 44 Billion was overvalued. If the only thing Twitter is supposed to do is to move tweets around then there was a ton of bloat of developers with a massively bloated purchase price. If Twitter had quality R&D products that could compete with TikTok, etc., then there may not have been a bloated workforce nor bloated purchase price.
itsoktocry 4y ago
>removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.
- Aeolun 4y ago
  > thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.
  - hef19898 4y ago
    That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.
    - anon400232 4y ago
      Saying everyone is replaceable is a first-degree approximation. The next level of detail would talk about, at least: (a) how much knowledge gets lost; (b) how many person hours -and- "wall clock time" it takes to rebuild that knowledge; (c) the amortized cost (or benefit) of losing that person
    - forgetfulness 4y ago
      It depends on how well or poorly run the business is. I worked at a company where everything hinged solely on one guy working from another country. When he left, loss of institutional knowledge took about three days to show real effects as things also came down crashing. I worked hard to make _myself_ replaceable for when I left, it was a pretty good exercise, but me having that degree of freedom was symptomatic of the problems of the company.
    - blueprint 4y ago
      how can you possibly say everyone is replaceable while next giving an example of a case where it's not only not true but potentially so invisible in its consequences as to make it hard or impossible to satisfactorily replace someone? a "person" can be rebuilt but not replaced. companies and societies aren't actually bose-einstein condensates. everyone is replaceable is just what different types of people say as a coping mechanism. it's a disgusting thing to promulgate too. maybe if someone is equivalent to a cog they can be replaced but humans are not exactly standardized cogs. what you mean to say, and may get in the habit of repeating instead, is that we don't necessarily need a specific person even though we are accustomed to them. it still might take you 10 years to find someone who does 80% of the same things as the other person while the missing 20% is what made that person such a unique hire in the first place. i find it a disgusting phrase that is so typical of the view of a society like ours today that is losing view of the value of life.
throw0101c 4y ago
> In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. It also (a) increases the bus factor, [1] and (b) allows people to take vacations and time off without having to watch their phones like hawk. [1] https://en.wikipedia.org/wiki/Bus_factor https://en.wikipedia.org/wiki/Bus_factor
- donkeyd 4y ago
  Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.
  - celim307 4y ago
    Sounds like those people are in strong negotiation positions
    - polynomial 4y ago
      H-1B would seem to belie that.
      - Rallen89 4y ago
        how would that negotiation go? give us the information we desperately need or we will basically deport you and lose the information anyway!
        polynomial 4y ago
        pretty much, but whatevs.
    - donkeyd 4y ago
      One of them just got promoted quite nicely, so I guess you're right!
thinkmcfly 4y ago
We are entering the windows ME era of Twitter
blueprint 4y ago
if 80% of knowledge is missing due to 80% of people being gone then the team who built it failed to document or automate themselves out of a job, meaning even other people on the team in the good ol' days would still have faced the same issues and merely had to chase down the original authors. that doesn't sound like it would be completely true. maybe 5-20% of un-documented knowledge walked out the door? completely rough guess. but even 0.5% of knowledge can sometimes be critical.
water8 4y ago
lightbendover 4y ago
> and I can keep going You focused mostly on additive bloat, there's also multiplicative bloat in the form of multiple teams focused on building separate versions of the same product to increase likelihood of success and empire building where leaders don't actually have a remit large enough to support the team size they have, but they have woven a narrative that defends the necessity nonetheless. Put everything together and teams are very easily 6x+ larger than they absolutely need to be to get a product into market.
cogman10 4y ago
The whole point of the article is twitter was designed to be resilient. (and it shows, twitter has great uptime). And the whole point of resiliency, beyond not negatively impacting customer experience, is to buy engineers time to fix things when stuff breaks. What we are watching is a massive failure event right now and the question really is if there's enough time for twitter management to fill in the gaps before there's an outage.
bumby 4y ago
Your post made me think of this discussion about how mature organizations naturally have to pivot towards maintenance: https://freakonomics.com/podcast/in-praise-of-maintenance/ https://freakonomics.com/podcast/in-praise-of-maintenance/
neycoda 4y ago
I'd argue that growth relies on stability, and if you don't have stability, you'll randomly lose growth.

ALittleLight 4y ago

I think Twitter was stuck in a not-especially profitable niche. They shift into fast-mode to get out of it and find a better spot, then they can shift back into stable mode once they occupy a better equilibrium. That said, there are lots of bugs in Twitter now, today, when they presumably had the benefit of being in stable mode for a long time. For example, Twitter regularly refreshes and loads new tweets while I'm reading them, pushing the tweet I was in the middle of reading out of view. That seems like a pretty silly bug to exist in a mature product. I regularly reach a state where I have to kill the app and relaunch it because all of the "back" commands just minimize the app instead of taking me back to the timeline. I could go on.

groffee 4y ago
That's not a bug, it's a 'feature', tons of sites do it for whatever asinine reason.
- ldng 4y ago
  Like ... displaying new / refreshing ads ?
- rjh29 4y ago
  Keeping vertical scroll position stable while asynchronously loading lots of randomly-sized content is harder than it seems.
  - mschuster91 4y ago
    Yeah, but there's always the option of showing a "fallback" when loading and inserting content in the middle of a feed - one link at the top "go to last bottom of top" and one at the bottom "go to last top of bottom".
  - vagrantJin 4y ago
    A complicated feature I am 100% certain no user ever asked for.
    - valeness 4y ago
      Actually twitter does it better than anyone else and it's the reason I use it over other apps. The User Experience is unparalleled
WA 4y ago
They replaced their somewhat productive engineering workforce with completely unproductive interest payments. I’m not convinced that this will lead to a better spot. But regarding the bugs, I’m totally with you. Same here. I use Twitter only in the browser. Browse long enough and the page reloads as if it ran out of memory.

kneebonian 4y ago

Based on the recent disclosures it seems that any new security issues pales in comparison to the current ones.

resonious 4y ago

> how does it persuade you of that? By flowing from many people think it's bloat - I'll tell you what's really going on to tiny team of 1~3 built whole infra for critical component. I'm not really trying to make commentary on whether or not Twitter engineering was bloat, or whether or not I think it'll hit problems in the future. Just commenting on the fact that the article broke my expectations a little bit as a reader.

btown 4y ago
https://twitter.com/atax1a/status/1594880931042824192 https://twitter.com/atax1a/status/1594880931042824192 provides a bit more context and nuance (albeit with a sardonic tone, but it's hard to blame someone for being sardonic who saw their entire professional community fired). There's no doubt that OP built a great and stable automation layer on top of Mesos for caching workloads. But there are numerous other types of workloads on top of Mesos (including, I presume mission-critical database deployments that need well-disciplined draining protocols to shift between nodes), as well as administrative needs for the Mesos-to-infrastructure level, and things running on bare metal below the Mesos level. These things all needed dedicated SREs, and the absence of these SREs could result in a scenario like the one mentioned in the Twitter thread I linked - two obscure mutually-dependent components expire and cannot be re-provisioned using documented tools. I also think an important meta-point is that when Twitter was bringing in substantial revenue from advertising, every minute of downtime would have significant costs - costs that could make it easily worthwhile to "over-provision" SRE talent. With advertisers pausing engagement, perhaps Twitter loses less money from a day-long outage than it would save having the right talent to turn a day-long outage into a minutes-long outage. Twitter is only judged by its profitability (namely, Musk's ability to service debt without selling more Tesla stock than he already has), while most other tech companies (both public and private) are judged by both profitability and revenue growth. If you want both, larger SRE teams, to say nothing of feature development and regulatory compliance teams, start to make a lot more sense.

deleted 4y ago

[deleted]

fnord123 4y ago

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

orwin 4y ago
I worked for a bank with 28000 total engineers. I know because part of what i did was crunching github and gitlab data and detecting double accounts (and deleting account from people not there anymore). I'd say 3/4th of them were just cost of doing business. Including me (my role wasn't really necessary, the stuff i did was cool and everything, but a bit useless).
- Xylakant 4y ago
  At this scale, having someone take care of deleting double/expired accounts is just good hygiene and I would not consider that useless. Forgotten accounts are a security risk.
  - papito 4y ago
    The bigger question is why such a large bank does not have solid off-boarding procedures.
    - Xylakant 4y ago
      Any off-boarding procedure fails - at that scale, if even one in one thousand fails, you can expect a couple of failures per year. ensuring these get caught is part of "solid offboarding".
    - orwin 4y ago
      They had that on the financial side. The eng/HR side was behind for sure.
    - anon400232 4y ago

bryanrasmussen 4y ago

>Bugs will creep in, https://twitter.com/IlluminatiGanga/status/1594609790432444417 https://twitter.com/IlluminatiGanga/status/15946097904324444... new members joining in 1970. hmmm.

holoduke 4y ago

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

kakwa_ 4y ago
That's likely optimistic. Maybe the user facing site, but that's just the tip of the iceberg. There are plenty of internal/backend/restricted systems to support and/or monetize this part. And that's not counting the huge number of support people & moderators needed.
- RivieraKid 4y ago
  Frontend, Android app and iOS app should be doable by 1 person each. Then one graphic designer and a project manager for all of the 3 frontends.
  - benj111 4y ago
    Well graphics design isn't hard. I'm sure one of the Devs could do that. And I'm sure there are tools to convert an iOS app to/from Android, so one dev can do both. And a website is easy. You could do it with 1 person. But Elon is such a machine, he could keep it running by himself.
    - RivieraKid 4y ago
      Well that's what I did, I made both iOS and Android version of an app and some of the design. I mean, a mediocre complexity app doesn't need more than one developer for a "slow improvement mode".
      - benj111 4y ago
        But Twitter needs more than 5 employees. How many bug reports do you think the Twitter site and apps get each day? How many employees do you think it takes just to triage them?

wikfwikf 4y ago

data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?

_3u10 4y ago
No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…
- wikfwikf 4y ago
  There are tens of thousands of engineers maintaining and supporting postgres and clickhouse in all the organizations where they are used, the vast majority of which are not on Twitter's scale.
  - pmoriarty 4y ago
    Tens of thousands across all the organizations where postgres and clickhouse are used, maybe (though even then probably not!). Not tens of thousands at any one organization supporting postgres and clickhouse. No single organization needs even hundreds of people to support these apps. You just need a good architecture and a handful of dba's, developers, and sysadmins... maybe.. depending on your scale. At many smaller orgs you can probably get away with one.
    - rvnx 4y ago
      This 100%. You don't need 10'000 engineers to run billions of queries. It's not like the SREs are running the queries themselves. Your job as an SRE is to design and run computer systems, not to become the computing platform yourself.
      - _3u10 4y ago

birdyrooster 4y ago

They will stagnate and have more failed launches than Space X.

pclmulqdq 4y ago
There's something poetic about NASA sending a rocket to the moon at the same time as Elon Musk is fucking around with Twitter instead of sending a SpaceX rocket there too.
- chasd00 4y ago
  Even in all the complexity of spaceflight, Twitter is the single greatest risk to SpaceX's mission. I would laugh but i'm crying.

mihaaly 4y ago

> The article touches a really small [...] subject. That's how it couldn't prove the claim.

ivanhoe 4y ago

> If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. It's not like Twitter was bug free before. How many times it annoyingly refreshed the timeline while I was reading something, or when it shows notification that it failed to send the DM, and when you retry it says "you've already wrote this", or you open the reply dialog, but it freezes, has no send button at all, so you have to re-open it. All of this was happening to me pretty regularly long before Elon came along. As we all know, just hiring more people is not necessarily the solution to every problem, and to me it seems it was exactly what Twitter tried to do in the past. Now they deconstructed it to the bare bones, which will clearly show what are the core problems and requirements. They basically turned Twitter back into a startup. And from that new starting point they can hire again to cover the needs as they arise. If they succeed it will be a huge success as they'll end up with far more optimal team (and huge savings), and of course, if they fail to catch up with problems it will be a huge failure. We'll see how well Musk can manage it...

TheOtherHobbes 4y ago
They basically gave an established global corporation with around 200mn active users a startup tech, marketing, and support stack. Anyone thinking "huge success" is unrealistically optimistic, IMO. It's going to be another MySpace/AOL/Bebo, added to the list of dumbest purchases ever. And that's still going to be true if the point was to destroy the original community and replace it with a different political orientation.
- abirch 4y ago
  You're right on this potentially being the worse purchase of all time. The purchase price of 44 Billion was overvalued. If the only thing Twitter is supposed to do is to move tweets around then there was a ton of bloat of developers with a massively bloated purchase price. If Twitter had quality R&D products that could compete with TikTok, etc., then there may not have been a bloated workforce nor bloated purchase price.

itsoktocry 4y ago

>removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

Aeolun 4y ago
> thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.
- hef19898 4y ago
  That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.
  - anon400232 4y ago
    Saying everyone is replaceable is a first-degree approximation. The next level of detail would talk about, at least: (a) how much knowledge gets lost; (b) how many person hours -and- "wall clock time" it takes to rebuild that knowledge; (c) the amortized cost (or benefit) of losing that person
  - forgetfulness 4y ago
    It depends on how well or poorly run the business is. I worked at a company where everything hinged solely on one guy working from another country. When he left, loss of institutional knowledge took about three days to show real effects as things also came down crashing. I worked hard to make _myself_ replaceable for when I left, it was a pretty good exercise, but me having that degree of freedom was symptomatic of the problems of the company.
  - blueprint 4y ago
    how can you possibly say everyone is replaceable while next giving an example of a case where it's not only not true but potentially so invisible in its consequences as to make it hard or impossible to satisfactorily replace someone? a "person" can be rebuilt but not replaced. companies and societies aren't actually bose-einstein condensates. everyone is replaceable is just what different types of people say as a coping mechanism. it's a disgusting thing to promulgate too. maybe if someone is equivalent to a cog they can be replaced but humans are not exactly standardized cogs. what you mean to say, and may get in the habit of repeating instead, is that we don't necessarily need a specific person even though we are accustomed to them. it still might take you 10 years to find someone who does 80% of the same things as the other person while the missing 20% is what made that person such a unique hire in the first place. i find it a disgusting phrase that is so typical of the view of a society like ours today that is losing view of the value of life.

throw0101c 4y ago

> In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. It also (a) increases the bus factor, [1] and (b) allows people to take vacations and time off without having to watch their phones like hawk. [1] https://en.wikipedia.org/wiki/Bus_factor https://en.wikipedia.org/wiki/Bus_factor

donkeyd 4y ago
Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.
- celim307 4y ago
  Sounds like those people are in strong negotiation positions
  - polynomial 4y ago
    H-1B would seem to belie that.
    - Rallen89 4y ago
      how would that negotiation go? give us the information we desperately need or we will basically deport you and lose the information anyway!
      - polynomial 4y ago
        pretty much, but whatevs.
  - donkeyd 4y ago
    One of them just got promoted quite nicely, so I guess you're right!

thinkmcfly 4y ago

We are entering the windows ME era of Twitter

blueprint 4y ago

if 80% of knowledge is missing due to 80% of people being gone then the team who built it failed to document or automate themselves out of a job, meaning even other people on the team in the good ol' days would still have faced the same issues and merely had to chase down the original authors. that doesn't sound like it would be completely true. maybe 5-20% of un-documented knowledge walked out the door? completely rough guess. but even 0.5% of knowledge can sometimes be critical.

water8 4y ago

lightbendover 4y ago

> and I can keep going You focused mostly on additive bloat, there's also multiplicative bloat in the form of multiple teams focused on building separate versions of the same product to increase likelihood of success and empire building where leaders don't actually have a remit large enough to support the team size they have, but they have woven a narrative that defends the necessity nonetheless. Put everything together and teams are very easily 6x+ larger than they absolutely need to be to get a product into market.

cogman10 4y ago

The whole point of the article is twitter was designed to be resilient. (and it shows, twitter has great uptime). And the whole point of resiliency, beyond not negatively impacting customer experience, is to buy engineers time to fix things when stuff breaks. What we are watching is a massive failure event right now and the question really is if there's enough time for twitter management to fill in the gaps before there's an outage.

bumby 4y ago

Your post made me think of this discussion about how mature organizations naturally have to pivot towards maintenance: https://freakonomics.com/podcast/in-praise-of-maintenance/ https://freakonomics.com/podcast/in-praise-of-maintenance/

neycoda 4y ago

I'd argue that growth relies on stability, and if you don't have stability, you'll randomly lose growth.