53 ms·

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually persuade me that it was all bloat.

donkeyd 4y ago
> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. If a product is mature enough, like Twitter seems to be, removing engineers won't instantly crash the product. It'll happen slowly. Bugs will creep in, because less time is spent on review and over all architecture. Security issues will creep in because of about the same issues and less oversight. Then, once this causes enough issues for the product to actually crash, the right people to fix it quickly might not be there anymore. That's when fixing the issues suddenly takes a lot more time. If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. Especially with Elon pushing for new features to be implemented quickly, inevitably by people who cannot fully understand the implications of said features, because 80% of knowledge is missing.
- ALittleLight 4y ago
  I think Twitter was stuck in a not-especially profitable niche. They shift into fast-mode to get out of it and find a better spot, then they can shift back into stable mode once they occupy a better equilibrium. That said, there are lots of bugs in Twitter now, today, when they presumably had the benefit of being in stable mode for a long time. For example, Twitter regularly refreshes and loads new tweets while I'm reading them, pushing the tweet I was in the middle of reading out of view. That seems like a pretty silly bug to exist in a mature product. I regularly reach a state where I have to kill the app and relaunch it because all of the "back" commands just minimize the app instead of taking me back to the timeline. I could go on.
  - groffee 4y ago
    That's not a bug, it's a 'feature', tons of sites do it for whatever asinine reason.
    - ldng 4y ago
      Like ... displaying new / refreshing ads ?
    - rjh29 4y ago
      Keeping vertical scroll position stable while asynchronously loading lots of randomly-sized content is harder than it seems.
      - mschuster91 4y ago
        Yeah, but there's always the option of showing a "fallback" when loading and inserting content in the middle of a feed - one link at the top "go to last bottom of top" and one at the bottom "go to last top of bottom".
      - vagrantJin 4y ago
        A complicated feature I am 100% certain no user ever asked for.
        valeness 4y ago
        Actually twitter does it better than anyone else and it's the reason I use it over other apps. The User Experience is unparalleled
  - WA 4y ago
    They replaced their somewhat productive engineering workforce with completely unproductive interest payments. I’m not convinced that this will lead to a better spot. But regarding the bugs, I’m totally with you. Same here. I use Twitter only in the browser. Browse long enough and the page reloads as if it ran out of memory.
- kneebonian 4y ago
  Based on the recent disclosures it seems that any new security issues pales in comparison to the current ones.
- resonious 4y ago
  > how does it persuade you of that? By flowing from many people think it's bloat - I'll tell you what's really going on to tiny team of 1~3 built whole infra for critical component. I'm not really trying to make commentary on whether or not Twitter engineering was bloat, or whether or not I think it'll hit problems in the future. Just commenting on the fact that the article broke my expectations a little bit as a reader.
  - btown 4y ago
    https://twitter.com/atax1a/status/1594880931042824192 https://twitter.com/atax1a/status/1594880931042824192 provides a bit more context and nuance (albeit with a sardonic tone, but it's hard to blame someone for being sardonic who saw their entire professional community fired). There's no doubt that OP built a great and stable automation layer on top of Mesos for caching workloads. But there are numerous other types of workloads on top of Mesos (including, I presume mission-critical database deployments that need well-disciplined draining protocols to shift between nodes), as well as administrative needs for the Mesos-to-infrastructure level, and things running on bare metal below the Mesos level. These things all needed dedicated SREs, and the absence of these SREs could result in a scenario like the one mentioned in the Twitter thread I linked - two obscure mutually-dependent components expire and cannot be re-provisioned using documented tools. I also think an important meta-point is that when Twitter was bringing in substantial revenue from advertising, every minute of downtime would have significant costs - costs that could make it easily worthwhile to "over-provision" SRE talent. With advertisers pausing engagement, perhaps Twitter loses less money from a day-long outage than it would save having the right talent to turn a day-long outage into a minutes-long outage. Twitter is only judged by its profitability (namely, Musk's ability to service debt without selling more Tesla stock than he already has), while most other tech companies (both public and private) are judged by both profitability and revenue growth. If you want both, larger SRE teams, to say nothing of feature development and regulatory compliance teams, start to make a lot more sense.
- deleted 4y ago
  [deleted]
- fnord123 4y ago
  Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?
  - orwin 4y ago
    I worked for a bank with 28000 total engineers. I know because part of what i did was crunching github and gitlab data and detecting double accounts (and deleting account from people not there anymore). I'd say 3/4th of them were just cost of doing business. Including me (my role wasn't really necessary, the stuff i did was cool and everything, but a bit useless).
    - Xylakant 4y ago
      At this scale, having someone take care of deleting double/expired accounts is just good hygiene and I would not consider that useless. Forgotten accounts are a security risk.
      - papito 4y ago
        The bigger question is why such a large bank does not have solid off-boarding procedures.
        Xylakant 4y ago
        Any off-boarding procedure fails - at that scale, if even one in one thousand fails, you can expect a couple of failures per year. ensuring these get caught is part of "solid offboarding".
        orwin 4y ago
        They had that on the financial side. The eng/HR side was behind for sure.
        anon400232 4y ago
- bryanrasmussen 4y ago
  >Bugs will creep in, https://twitter.com/IlluminatiGanga/status/1594609790432444417 https://twitter.com/IlluminatiGanga/status/15946097904324444... new members joining in 1970. hmmm.
- holoduke 4y ago
  How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.
  - kakwa_ 4y ago
    That's likely optimistic. Maybe the user facing site, but that's just the tip of the iceberg. There are plenty of internal/backend/restricted systems to support and/or monetize this part. And that's not counting the huge number of support people & moderators needed.
    - RivieraKid 4y ago
      Frontend, Android app and iOS app should be doable by 1 person each. Then one graphic designer and a project manager for all of the 3 frontends.
      - benj111 4y ago
        Well graphics design isn't hard. I'm sure one of the Devs could do that. And I'm sure there are tools to convert an iOS app to/from Android, so one dev can do both. And a website is easy. You could do it with 1 person. But Elon is such a machine, he could keep it running by himself.
        RivieraKid 4y ago
        Well that's what I did, I made both iOS and Android version of an app and some of the design. I mean, a mediocre complexity app doesn't need more than one developer for a "slow improvement mode".
        benj111 4y ago
        But Twitter needs more than 5 employees. How many bug reports do you think the Twitter site and apps get each day? How many employees do you think it takes just to triage them?
- wikfwikf 4y ago
  data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?
  - _3u10 4y ago
    No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…
    - wikfwikf 4y ago
      There are tens of thousands of engineers maintaining and supporting postgres and clickhouse in all the organizations where they are used, the vast majority of which are not on Twitter's scale.
      - pmoriarty 4y ago
        Tens of thousands across all the organizations where postgres and clickhouse are used, maybe (though even then probably not!). Not tens of thousands at any one organization supporting postgres and clickhouse. No single organization needs even hundreds of people to support these apps. You just need a good architecture and a handful of dba's, developers, and sysadmins... maybe.. depending on your scale. At many smaller orgs you can probably get away with one.
        rvnx 4y ago
        This 100%. You don't need 10'000 engineers to run billions of queries. It's not like the SREs are running the queries themselves. Your job as an SRE is to design and run computer systems, not to become the computing platform yourself.
        _3u10 4y ago
- birdyrooster 4y ago
  They will stagnate and have more failed launches than Space X.
  - pclmulqdq 4y ago
    There's something poetic about NASA sending a rocket to the moon at the same time as Elon Musk is fucking around with Twitter instead of sending a SpaceX rocket there too.
    - chasd00 4y ago
      Even in all the complexity of spaceflight, Twitter is the single greatest risk to SpaceX's mission. I would laugh but i'm crying.
- mihaaly 4y ago
  > The article touches a really small [...] subject. That's how it couldn't prove the claim.
- ivanhoe 4y ago
  > If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. It's not like Twitter was bug free before. How many times it annoyingly refreshed the timeline while I was reading something, or when it shows notification that it failed to send the DM, and when you retry it says "you've already wrote this", or you open the reply dialog, but it freezes, has no send button at all, so you have to re-open it. All of this was happening to me pretty regularly long before Elon came along. As we all know, just hiring more people is not necessarily the solution to every problem, and to me it seems it was exactly what Twitter tried to do in the past. Now they deconstructed it to the bare bones, which will clearly show what are the core problems and requirements. They basically turned Twitter back into a startup. And from that new starting point they can hire again to cover the needs as they arise. If they succeed it will be a huge success as they'll end up with far more optimal team (and huge savings), and of course, if they fail to catch up with problems it will be a huge failure. We'll see how well Musk can manage it...
  - TheOtherHobbes 4y ago
    They basically gave an established global corporation with around 200mn active users a startup tech, marketing, and support stack. Anyone thinking "huge success" is unrealistically optimistic, IMO. It's going to be another MySpace/AOL/Bebo, added to the list of dumbest purchases ever. And that's still going to be true if the point was to destroy the original community and replace it with a different political orientation.
    - abirch 4y ago
      You're right on this potentially being the worse purchase of all time. The purchase price of 44 Billion was overvalued. If the only thing Twitter is supposed to do is to move tweets around then there was a ton of bloat of developers with a massively bloated purchase price. If Twitter had quality R&D products that could compete with TikTok, etc., then there may not have been a bloated workforce nor bloated purchase price.
- itsoktocry 4y ago
  >removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.
  - Aeolun 4y ago
    > thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.
    - hef19898 4y ago
      That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.
      - anon400232 4y ago
        Saying everyone is replaceable is a first-degree approximation. The next level of detail would talk about, at least: (a) how much knowledge gets lost; (b) how many person hours -and- "wall clock time" it takes to rebuild that knowledge; (c) the amortized cost (or benefit) of losing that person
      - forgetfulness 4y ago
        It depends on how well or poorly run the business is. I worked at a company where everything hinged solely on one guy working from another country. When he left, loss of institutional knowledge took about three days to show real effects as things also came down crashing. I worked hard to make _myself_ replaceable for when I left, it was a pretty good exercise, but me having that degree of freedom was symptomatic of the problems of the company.
      - blueprint 4y ago
        how can you possibly say everyone is replaceable while next giving an example of a case where it's not only not true but potentially so invisible in its consequences as to make it hard or impossible to satisfactorily replace someone? a "person" can be rebuilt but not replaced. companies and societies aren't actually bose-einstein condensates. everyone is replaceable is just what different types of people say as a coping mechanism. it's a disgusting thing to promulgate too. maybe if someone is equivalent to a cog they can be replaced but humans are not exactly standardized cogs. what you mean to say, and may get in the habit of repeating instead, is that we don't necessarily need a specific person even though we are accustomed to them. it still might take you 10 years to find someone who does 80% of the same things as the other person while the missing 20% is what made that person such a unique hire in the first place. i find it a disgusting phrase that is so typical of the view of a society like ours today that is losing view of the value of life.
- throw0101c 4y ago
  > In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. It also (a) increases the bus factor, [1] and (b) allows people to take vacations and time off without having to watch their phones like hawk. [1] https://en.wikipedia.org/wiki/Bus_factor https://en.wikipedia.org/wiki/Bus_factor
  - donkeyd 4y ago
    Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.
    - celim307 4y ago
      Sounds like those people are in strong negotiation positions
      - polynomial 4y ago
        H-1B would seem to belie that.
        Rallen89 4y ago
        how would that negotiation go? give us the information we desperately need or we will basically deport you and lose the information anyway!
        polynomial 4y ago
        pretty much, but whatevs.
      - donkeyd 4y ago
        One of them just got promoted quite nicely, so I guess you're right!
- thinkmcfly 4y ago
  We are entering the windows ME era of Twitter
- blueprint 4y ago
  if 80% of knowledge is missing due to 80% of people being gone then the team who built it failed to document or automate themselves out of a job, meaning even other people on the team in the good ol' days would still have faced the same issues and merely had to chase down the original authors. that doesn't sound like it would be completely true. maybe 5-20% of un-documented knowledge walked out the door? completely rough guess. but even 0.5% of knowledge can sometimes be critical.
- water8 4y ago
- lightbendover 4y ago
  > and I can keep going You focused mostly on additive bloat, there's also multiplicative bloat in the form of multiple teams focused on building separate versions of the same product to increase likelihood of success and empire building where leaders don't actually have a remit large enough to support the team size they have, but they have woven a narrative that defends the necessity nonetheless. Put everything together and teams are very easily 6x+ larger than they absolutely need to be to get a product into market.
- cogman10 4y ago
  The whole point of the article is twitter was designed to be resilient. (and it shows, twitter has great uptime). And the whole point of resiliency, beyond not negatively impacting customer experience, is to buy engineers time to fix things when stuff breaks. What we are watching is a massive failure event right now and the question really is if there's enough time for twitter management to fill in the gaps before there's an outage.
- bumby 4y ago
  Your post made me think of this discussion about how mature organizations naturally have to pivot towards maintenance: https://freakonomics.com/podcast/in-praise-of-maintenance/ https://freakonomics.com/podcast/in-praise-of-maintenance/
- neycoda 4y ago
  I'd argue that growth relies on stability, and if you don't have stability, you'll randomly lose growth.
twblalock 4y ago
It’s easy to think it’s bloat at a steady state. When something important goes down and nobody knows how to fix it, it looks different.
- cm2187 4y ago
  That being said it's not like twitter is a massively complex product with lots of different features. I can imagine you could keep it running with a skeletton team. Liasing with ads buyers excepted.
  - twelve40 4y ago
    a bunch of people I talk to say it is massively complex but typically fail to explain how, especially given the super-glacial pace at which they added new features for 15 years. And yes, this article kind of doesn't disprove the bloat at all, unless every single SRE quit? but that is also not what's stated in there.
  - bryanrasmussen 4y ago
    I'm pretty sure one of the most complicated things is preventing automation of content - bots. which would be an arms race type condition. Bots you want to prevent - people not using twitter's api to do their bot stuff. Why would people not use twitter's api? 1. who would trust twitter not to change API and make code worthless 2. people who want to do stuff Twitter doesn't want you to do in an automated fashion.
  - ehnto 4y ago
    I imagine the infrastructure implemented is fairly complex, this post does outline that it's not a super simple operation. It also alludes to there being more business units than just the core application.
  - twblalock 4y ago
    Even "simple" things become complex when they need to scale to billions of users around the world, handle a high rate of traffic, deal professionally and legally with all the different jurisdictions they operate in and all the advertisers they serve, etc.
28304283409234 4y ago
Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issues. Not mopping up after them. To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.
- tempestn 4y ago
  I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by the parent, not trying to make one myself.)
  - enkid 4y ago
    So 1-3 people have a big impact, the other 7997 must not be doing anything? I don't think that logic follows.
    - twelve40 4y ago
      it doesn't follow. The article posits that "many people think twitter headcount was bloated" then proceeds to describe a (presumably) really efficient work of a small SRE team. These two parts seem completely disconnected from each other - neither one proves, disproves or follows from the other - so it's unclear why the former was mentioned at all.
      - enkid 4y ago
        His personal experience was zero bloat. He was one deep in a critical function for the company. He isn't saying this proves without a doubt that there is no bloat, but he didn't see any in his time there. It's seems like a reasonable addition to the conversation to me.
oxfordmale 4y ago
Let's do a thought experiment and see what functions aren't needed to keep the light on for 30 days: 1) HR 2) Legal 3) Sales 4) Marketing 5) Payroll 6) Admin staff 7) Most of Engineering, other than the bare minimum of L1/L2/3 support. As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.
- jiggawatts 4y ago
  This is more like a car with just the bare essentials: an engine, steering wheel, brakes, and the like. No: radio, air-conditioning, seat padding, wipers, lights, radar, etc... Oh, and no maintenance. It'll drive... for now. But that's it.
  - sydd 4y ago
    More like a racecar without maintenance. Heres an example checklist what they need: https://www.teamstradale.com/frequency-of-maintenance-keeping-your-racecar-in-tip-top-shape/ https://www.teamstradale.com/frequency-of-maintenance-keepin...
- coldtea 4y ago
  >As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend. On the other hand, a car without a second and third steering wheel, 20 windscreen wipers, and an oven in the back, keeps running just fine, even after the first bend...
  - yywwbbn 4y ago
    True, you might be also forced to sell the windshield, doors and the seats to pay the $4 million in interest each day you didn’t have to before
  - nashashmi 4y ago
    Unless the first steering wheel is prone to breaking. Redundancy is the difference between a million dollar company and billion dollar company. Of course getting workers to come on on weekends and midnight hours accomplishes the same thing.
bjarneh 4y ago
> I was partly expecting the rest of the article to explain to my why exactly it wasn't just bloat Same here. I guess his header was on point in why Twitter is still up; but I was also interested in hearing about why Twitter actually needs all those people. If it can be run with 50-80% of the staff gone, that does sound like some bloat at least.
- depereo 4y ago
  Slack space leads to innovations, like developing infrastructure automation and improving capacity planning. SRE as a practice needs slack space for operations teams to work on improvements and fixes in addition to BAU fault fixing, deployments and patching.
  - bjarneh 4y ago
    > Slack space leads to innovations I'm certainly no supporter of "lean operations" with minimum staff etc, and fully agree that you need people that are well rested (for the lack of a better analogy) to do great stuff. But I do think that some of these internet giants do have to many people working there; wasn't LinkedIn 14_000 strong when Microsoft bought it? I've always felt that the American model of doing business is based on how we optimize network traffic, i.e. double the amount of data until failure; then turn in down a bit. Fire on all cylinders until people are truly worn out, then turn down the pace a bit. Haven't worked in the US so I'm pulling this info out of thin air...
    - pas 4y ago
      Well, half of the staff prepares for some conference, meetup, tech talk, or helps organize one, or does 20% time, or sits in unnecessary meetings, or sits in necessary but inefficient meetings, or is on PTO or on unpaid leave in some retreat. Plus above a certain headcount the communication overhead becomes seriously large, so just to compensate for the lost velocity you need to break out into smaller more agile more autonomous teams, which further increases the coordination requirements (thus the comms overhead), but allows overall throughput to scale. And the leading edge technologies commonly used require a large headcount to begin with. (I mean just to start running something twitter/linkedin sized requires at least 1 engineer/million users, so a few hundred folks is a given. You need someone who understands networking, from BGP to TLS to VPN to whatever, internal IT, CI/CD, ops/SRE ... at that scale if you use anything, you need an expert for it. You use Kafka with a hundred millions of users? You might need at least a few people who actually know what the fuck a partition means. Unless you want to just directly give all of your money to Jeff in the form of egress fees, you might need folks to setup CDNs, and whatnot. So without naming names, ina big password manager company (around a hundred million users?) a few years ago there was a certain rewrite project. 3+ people worked on it for 8-10 months, then it was put on hold temporarily. And then obviously nobody speak of it ever. It happens that there are inefficiencies that for months not one line goes into production from certain individuals. It was bad management, yes. But if good management was easy to find then we would be talking about different things :)
weinzierl 4y ago
No matter if it was or not and for better or worse: If Twitter survives this without any major harm it will have profound consequences for the whole software industry.
- ravloony 4y ago
  This is exactly why this thread is full of slightly insecure comments making vague predictions. I'd suggest to most of them to get off HN and back to work, now is the time to make yourself useful!
- pas 4y ago
  That's tiny bit of an exaggeration, maaaybe. Maybe completely prophetic, though! That said, Instagram was run by just a dozen people back then, while it had hundreds of millions of users, right? So it's not a new data point. Storing, retrieving, indexing, managing 280 char blobs (with links, threads, embeds) is not exactly the most hardcore of a problem domains. Microblogs are the typical tutorial topic, and twitter's only "innovation" was to increase the text limit.
  - gizmo 4y ago
    Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentation.
    - grishka 4y ago
      A lot of things in a typical social media service can be segmented trivially. The hardest part is the feed because it requires querying many database servers at once, but then it can be cached and served quicker.
      - wayfarer2s 4y ago
        Isn’t cache invalidation the hardest part of the feed? Once a user with millions of followers tweets, doesn’t that essentially invalidate millions of caches? That doesn’t sound like a trivial problem.
usgroup 4y ago
I expect it all to work better with Musk in charge. He knows how to make scalable software and he knows about performant teams. That he’s not going to realise these totally obvious first order consequences people are raising seems unlikely.
- torben-friis 4y ago
  I honestly can’t tell if you’re being sarcastic.
- FranzFerdiNaN 4y ago
  Everything that is happening with Twitter proves Musk is yet another wealthy idiot who doenst know shit about shit, except how to blow his own horn. Musk is simlpy lucky that Twitter used to have such excellent engineers like this SRE so that the site isnt yet on fire.
  - drcross 4y ago
    Musk lead teams at the only company who can land commercial rockets and the first company to profit from making electric cars. To say he is a wealthy idiot says more about you than it does about Musk.
    - yywwbbn 4y ago
      So Musk hired the right people. I mean he made some brilliant decisions over the years. Does that mean that everything he does is a guaranteed success? There are plenty examples of ultra successful people in history who were blinded by their own success and became a little bit unhinged over time.. At this point Musk is behaving just like a twelve year old’s idea of a mega billionaire. Hyperloop, flamethrowers, self driving cars now Twitter... It’s great that he’s having fun I don’t think there is much to it than that.
- GoOnThenDoTell 4y ago
  You do have to add the /s, else its confusing
carabiner 4y ago
Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes. MBA's will be studying this for years.
- tempestn 4y ago
  Musk might have been right about some things. There probably was some degree of bloat. But to say he's badly mishandled this whole saga is a gross understatement. It is very difficult to utterly kill a site like Twitter; the fact that we're even considering that as a realistic possibility shows just how badly. I think Musk is used to Tesla and SpaceX, which are both companies that a lot of people are (or at least were) excited to work for because they believe in the mission and what's being created. Plus there aren't many alternatives if you want to do that work. Twitter really isn't like that for most people; a Twitter developer has many other options to do similar work. Add to that the fact that he's both cranked up the intensity of the abuse and that it's more visible to everyone, and you can't expect a lot of good people to stick around. And despite the fact that it might coast for quite a while on the back of excellent work in the past, eventually you do need good people to keep a business going. (This is leaving aside the direct impacts of his actions on users and advertisers!)
  - ReGenGen 4y ago
    What is the Mission now at twitter? Kiss Lord Musk's boot?
  - coldtea 4y ago
    >the fact that we're even considering that as a realistic possibility shows just how badly. Do we have much to go with to "really consider it", or is it just sensenationalist headlines, as Musk went ahead the accepted orthodoxy on Twitter and content moderation and so "he must pay"? That would be the same news outlets that built him up as the real life Tony Stark and propped up his quite ordinary companies, one of which is building electric cars like everybody else, just doing it in style as well-off upper-middle class people's toys, and another is doing space tech that was already a thing we were pursuing 40 years ago with a little modern engineering thrown in...
H8crilA 4y ago
This is a PR article that tries to push the idea that Twitter is OK (to work at, and maybe also to buy ad space from). Damage control.
- Lutger 4y ago
  I read it as the exact opposite: the reason Twitter is still ok* is not because all these people were just browsing reddit at work. You can't just gut Twitter to run on only a couple of hundred people and still expect the same results in the longterm. Twitter was not a leftist commie welfare company as Musk and its fans want it to be. It was actually the fine work SRE (amongst others) put into it that makes it still tick along as it does...for now. * actually some things are already breaking, but it will take some time for the real damage to surface on a technical level
  - pas 4y ago
    It could have been both! :) And seriously, the gaming industry is infamous for its "year round crunch mode". It can be done. Many sectors are "hard work" (ie. public sector, fundamentally non-tech gigacorps with no real in-house IT competence, etc.) Twitter was an innovator. I remember getting into Scala party because of some open source code library from them back then, I don't know, probably around Scala 2.8. That was absolutely unnecessary for their core business. All of this can be done in bad C++, or plain old Java 7, or whatever, for low cost, built in Bangalore. Now that would of course likely not withstand this kind of change storm that Elon caused. But that's a different story. (Also what Elon probably doesn't understand on a technical level (but likely guessed from a business aspect), is how flexible Twitter is. In this regard he hit arrived at the right time. Twitter is fundamentally the same as it was, now it's time to innovate with it, and the architecture he found is exactly the right one for that.)
Lutger 4y ago
I'm puzzled by this statement. Do you think of resiliency as waste? That twitter would have been fine without it? The article makes a point that the reason Twitter is running ok on 20% of personnel at this moment is exactly because it was build to be resilient, not because the personnel was bloated. A large part of this so called bloat, the 80%, was responsible for Twitter to be running right now. Calling this bloat implies it is actually not important for Twitter to be available all the time (or at all).
- concordDance 4y ago
  How was twitter doing fine a few years back at half the number of employees and same number of users?
adql 4y ago
You just need to look at profit and revenue. Revenue grew nicely in last 2-3 years, profit not so much. Bloat is only reason.
- raspberry1337 4y ago
  This, in my reflection, is that one insightful comment that should be higher up even it came to late. Twitter userbase did not expand significantly in the last couple of years. Revenue increased. Why did cost increase so much?
nashashmi 4y ago
In addition to the many great comments here, remember that super star engineers don't exactly fix problems from day to day. They fix the problems before they become problems. The engineer was doing stability planning for 6 months out for the purpose of cost optimization. I guess we can assume that the costs of infrastructure is about to go up and reliability is about to go down in the coming months.
Aeolun 4y ago
And so is the cache setup. It’s permanently (and deliberately) running at less than 50% utilization to prevent an issue that comes up only once every 5 years (according to the author).
HeavyStorm 4y ago
Indeed!
ryanfreeborn 4y ago
I have been under the good faith assumption that most (though definitely not all) of the employees that have departed Twitter were probably necessary and valuable to the company. I left the article with the same impression as you. This single person did this very important job, seemingly well, and didn't appear to be drowning in the work. What were the other 8-9k doing?
jameshart 4y ago
That small team seems to have been running the caches for other teams, by using infrastructure provided by another team, in two massive datacenters operated by other teams, using monitoring tools managed by another team, and a ticketing system run by another, on hardware purchases by another team… All just to put caching in front of services that actually do anything.
- water8 4y ago
  Can’t forget about those people that ordered that hardware once. He probably had to go to the business website and everything.
adam_arthur 4y ago
Well, WhatsApp had ~50 employees and Instagram around ~15 when FB acquired them, and they were around the same order of magnitude of complexity as Twitter. The only concern Id have is that by having so many people, your design probably comes to rely on them whereas a smaller team would be forced to make the system easier to maintain. Personally, if I were Elon, I’d build an entirely new backend and point the clients to that rather than trying to incrementally improve what they have. Get 50-100 10x engineers that are loyal to Elon, with big equity stakes, and crush it
- pclmulqdq 4y ago
  A lot of people who don't understand why Twitter owns two datacenters point to "complexity" as an argument and completely disregard scale. It turns out that massive scale adds a lot of complexity to a system, particularly around many-to-many pubsub systems (like social media). It also means a lot of features, like compliance with government regulations around the world.
  - adam_arthur 4y ago
    WhatsApp had a Billion users and 50 employees. You can say that Twitter is incrementally more complex due to wider broadcasting, but it’s also 1/5 the user count. Everybody stating this number of employees is necessary to maintain a Twitter scale system is simply wrong. And as technology progresses, fewer and fewer people are needed to maintain the same size system Many just don’t want to believe that the leverage in employees favor in the tech sector is fading fast, and their labor is not going to warrant 500k comp with marginal effort anymore
    - pclmulqdq 4y ago
      Whatsapp, discord, telegram, and all other chat apps have a very easy time scaling. Using user count as a metric and comparing them to Twitter is pretty disingenuous. Instagram is a much better comparison, which had 1/50th the user count of Twitter at its acquisition. Chat apps are well known to rely on infrastructure that scales up exceedingly well. That is why there are so many of them, and why they all have 100 tech employees or less. Instagram at the time of the acquisition could have run on 20 servers plus S3. Today's Instagram, along with today's Twitter, does a lot of work that is super-linear in user count, and has something like 2000 engineers. Timeline building is reportedly O(n^2) in user count. The scale difference has a huge effect. Notice that chat apps have no equivalent to "timeline building." The worst scale factor a chat app has is linear.
pron 4y ago
> But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. ... for the Cache component. There are many others.
toss1 4y ago
>>If anything, the article might actually persuade me that it was all bloat. Not for me This is almost exactly like the new manager coming in, noticing that the floors and surfaces are all clean, all the systems work, the trash is emptied, etc., and so deciding that the entire maintenance staff is unnecessary and firing them. The place doesn't become a decrepit pigsty the next morning; it slowly degrades. Same for these systems. They were designed, built, tuned, and maintained over the course of years to go from requiring constant manual intervention to running largely unattended and with a good buffer of ready hardware and automatic failover for failures. That "largely" in "largely unattended" is doing some very heavy lifting. The system WILL require human intervention to keep running, and more than just a skeleton crew. The only question is whether it will happen before the new crew gets up to speed to handle the inevitable degradation. This does NOT mean that the SREs were bloat - it means that they were doing an excellent job and could safely take a break. We're now in just the two-week vacation zone - same as if the entire SRE team went on a holiday. We'd expect it to work. Now let's see what happens in two months.
w0m 4y ago
If you only care about 'it mostly didn't crash' as the end-goal of a company, that would be a reasonable take.
v0idzer0 4y ago
Of course it was bloat. This whole “twitter is going to crash and burn” thing is a weird fantasy. Most likely it will just be run more efficiently by far less people.
lazyant 4y ago
I guess different definitions for "bloat" but how is it bloat to have a tiny team taken care of a fundamental piece of infrastructure? if the team is now gone, an issue there would mean hours of downtime. If that's acceptable then yes, it was "bloat".
halffaday 4y ago
I’m suspicious that most of the value in these systems comes from a small fraction of the effort and many technology jobs boil down to knowing you’re a huge cost center and putting on a performance to hide that.
ouid 4y ago
Of course its all bloat. Software runs on computers, not engineers. The default assumption for software is that it will go on working. The state might devolve, but the software is exactly as reliable right now as it was before. I'm no friend of Elon, and I think its hilarious to think that he can be king of twitter, but all these people talking about "code entropy" are certifiably insane. Big tech maintains talent so that they won't use their knowledge of the system to produce an identical competitor without the technical debt or investor liability of the original.
memish 4y ago
There's an incredible amount of bloat in big tech. It's become an adult daycare, https://twitter.com/DavidSacks/status/1561096423243800576 https://twitter.com/DavidSacks/status/1561096423243800576 Twitter's layoffs followed by that 1AM photo of hackers at work is terrifying to lifestyle employees. It's the Return of the Nerds. Google, Meta, Netflix, Microsoft are all watching.
- mylons 4y ago
  what does working until 1AM have anything to do with being a nerd or a hacker?

donkeyd 4y ago

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. If a product is mature enough, like Twitter seems to be, removing engineers won't instantly crash the product. It'll happen slowly. Bugs will creep in, because less time is spent on review and over all architecture. Security issues will creep in because of about the same issues and less oversight. Then, once this causes enough issues for the product to actually crash, the right people to fix it quickly might not be there anymore. That's when fixing the issues suddenly takes a lot more time. If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. Especially with Elon pushing for new features to be implemented quickly, inevitably by people who cannot fully understand the implications of said features, because 80% of knowledge is missing.

ALittleLight 4y ago
I think Twitter was stuck in a not-especially profitable niche. They shift into fast-mode to get out of it and find a better spot, then they can shift back into stable mode once they occupy a better equilibrium. That said, there are lots of bugs in Twitter now, today, when they presumably had the benefit of being in stable mode for a long time. For example, Twitter regularly refreshes and loads new tweets while I'm reading them, pushing the tweet I was in the middle of reading out of view. That seems like a pretty silly bug to exist in a mature product. I regularly reach a state where I have to kill the app and relaunch it because all of the "back" commands just minimize the app instead of taking me back to the timeline. I could go on.
- groffee 4y ago
  That's not a bug, it's a 'feature', tons of sites do it for whatever asinine reason.
  - ldng 4y ago
    Like ... displaying new / refreshing ads ?
  - rjh29 4y ago
    Keeping vertical scroll position stable while asynchronously loading lots of randomly-sized content is harder than it seems.
    - mschuster91 4y ago
      Yeah, but there's always the option of showing a "fallback" when loading and inserting content in the middle of a feed - one link at the top "go to last bottom of top" and one at the bottom "go to last top of bottom".
    - vagrantJin 4y ago
      A complicated feature I am 100% certain no user ever asked for.
      - valeness 4y ago
        Actually twitter does it better than anyone else and it's the reason I use it over other apps. The User Experience is unparalleled
- WA 4y ago
  They replaced their somewhat productive engineering workforce with completely unproductive interest payments. I’m not convinced that this will lead to a better spot. But regarding the bugs, I’m totally with you. Same here. I use Twitter only in the browser. Browse long enough and the page reloads as if it ran out of memory.
kneebonian 4y ago
Based on the recent disclosures it seems that any new security issues pales in comparison to the current ones.
resonious 4y ago
> how does it persuade you of that? By flowing from many people think it's bloat - I'll tell you what's really going on to tiny team of 1~3 built whole infra for critical component. I'm not really trying to make commentary on whether or not Twitter engineering was bloat, or whether or not I think it'll hit problems in the future. Just commenting on the fact that the article broke my expectations a little bit as a reader.
- btown 4y ago
  https://twitter.com/atax1a/status/1594880931042824192 https://twitter.com/atax1a/status/1594880931042824192 provides a bit more context and nuance (albeit with a sardonic tone, but it's hard to blame someone for being sardonic who saw their entire professional community fired). There's no doubt that OP built a great and stable automation layer on top of Mesos for caching workloads. But there are numerous other types of workloads on top of Mesos (including, I presume mission-critical database deployments that need well-disciplined draining protocols to shift between nodes), as well as administrative needs for the Mesos-to-infrastructure level, and things running on bare metal below the Mesos level. These things all needed dedicated SREs, and the absence of these SREs could result in a scenario like the one mentioned in the Twitter thread I linked - two obscure mutually-dependent components expire and cannot be re-provisioned using documented tools. I also think an important meta-point is that when Twitter was bringing in substantial revenue from advertising, every minute of downtime would have significant costs - costs that could make it easily worthwhile to "over-provision" SRE talent. With advertisers pausing engagement, perhaps Twitter loses less money from a day-long outage than it would save having the right talent to turn a day-long outage into a minutes-long outage. Twitter is only judged by its profitability (namely, Musk's ability to service debt without selling more Tesla stock than he already has), while most other tech companies (both public and private) are judged by both profitability and revenue growth. If you want both, larger SRE teams, to say nothing of feature development and regulatory compliance teams, start to make a lot more sense.
deleted 4y ago
[deleted]
fnord123 4y ago
Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?
- orwin 4y ago
  I worked for a bank with 28000 total engineers. I know because part of what i did was crunching github and gitlab data and detecting double accounts (and deleting account from people not there anymore). I'd say 3/4th of them were just cost of doing business. Including me (my role wasn't really necessary, the stuff i did was cool and everything, but a bit useless).
  - Xylakant 4y ago
    At this scale, having someone take care of deleting double/expired accounts is just good hygiene and I would not consider that useless. Forgotten accounts are a security risk.
    - papito 4y ago
      The bigger question is why such a large bank does not have solid off-boarding procedures.
      - Xylakant 4y ago
        Any off-boarding procedure fails - at that scale, if even one in one thousand fails, you can expect a couple of failures per year. ensuring these get caught is part of "solid offboarding".
      - orwin 4y ago
        They had that on the financial side. The eng/HR side was behind for sure.
      - anon400232 4y ago
bryanrasmussen 4y ago
>Bugs will creep in, https://twitter.com/IlluminatiGanga/status/1594609790432444417 https://twitter.com/IlluminatiGanga/status/15946097904324444... new members joining in 1970. hmmm.
holoduke 4y ago
How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.
- kakwa_ 4y ago
  That's likely optimistic. Maybe the user facing site, but that's just the tip of the iceberg. There are plenty of internal/backend/restricted systems to support and/or monetize this part. And that's not counting the huge number of support people & moderators needed.
  - RivieraKid 4y ago
    Frontend, Android app and iOS app should be doable by 1 person each. Then one graphic designer and a project manager for all of the 3 frontends.
    - benj111 4y ago
      Well graphics design isn't hard. I'm sure one of the Devs could do that. And I'm sure there are tools to convert an iOS app to/from Android, so one dev can do both. And a website is easy. You could do it with 1 person. But Elon is such a machine, he could keep it running by himself.
      - RivieraKid 4y ago
        Well that's what I did, I made both iOS and Android version of an app and some of the design. I mean, a mediocre complexity app doesn't need more than one developer for a "slow improvement mode".
        benj111 4y ago
        But Twitter needs more than 5 employees. How many bug reports do you think the Twitter site and apps get each day? How many employees do you think it takes just to triage them?
wikfwikf 4y ago
data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?
- _3u10 4y ago
  No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…
  - wikfwikf 4y ago
    There are tens of thousands of engineers maintaining and supporting postgres and clickhouse in all the organizations where they are used, the vast majority of which are not on Twitter's scale.
    - pmoriarty 4y ago
      Tens of thousands across all the organizations where postgres and clickhouse are used, maybe (though even then probably not!). Not tens of thousands at any one organization supporting postgres and clickhouse. No single organization needs even hundreds of people to support these apps. You just need a good architecture and a handful of dba's, developers, and sysadmins... maybe.. depending on your scale. At many smaller orgs you can probably get away with one.
      - rvnx 4y ago
        This 100%. You don't need 10'000 engineers to run billions of queries. It's not like the SREs are running the queries themselves. Your job as an SRE is to design and run computer systems, not to become the computing platform yourself.
        _3u10 4y ago
birdyrooster 4y ago
They will stagnate and have more failed launches than Space X.
- pclmulqdq 4y ago
  There's something poetic about NASA sending a rocket to the moon at the same time as Elon Musk is fucking around with Twitter instead of sending a SpaceX rocket there too.
  - chasd00 4y ago
    Even in all the complexity of spaceflight, Twitter is the single greatest risk to SpaceX's mission. I would laugh but i'm crying.
mihaaly 4y ago
> The article touches a really small [...] subject. That's how it couldn't prove the claim.
ivanhoe 4y ago
> If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. It's not like Twitter was bug free before. How many times it annoyingly refreshed the timeline while I was reading something, or when it shows notification that it failed to send the DM, and when you retry it says "you've already wrote this", or you open the reply dialog, but it freezes, has no send button at all, so you have to re-open it. All of this was happening to me pretty regularly long before Elon came along. As we all know, just hiring more people is not necessarily the solution to every problem, and to me it seems it was exactly what Twitter tried to do in the past. Now they deconstructed it to the bare bones, which will clearly show what are the core problems and requirements. They basically turned Twitter back into a startup. And from that new starting point they can hire again to cover the needs as they arise. If they succeed it will be a huge success as they'll end up with far more optimal team (and huge savings), and of course, if they fail to catch up with problems it will be a huge failure. We'll see how well Musk can manage it...
- TheOtherHobbes 4y ago
  They basically gave an established global corporation with around 200mn active users a startup tech, marketing, and support stack. Anyone thinking "huge success" is unrealistically optimistic, IMO. It's going to be another MySpace/AOL/Bebo, added to the list of dumbest purchases ever. And that's still going to be true if the point was to destroy the original community and replace it with a different political orientation.
  - abirch 4y ago
    You're right on this potentially being the worse purchase of all time. The purchase price of 44 Billion was overvalued. If the only thing Twitter is supposed to do is to move tweets around then there was a ton of bloat of developers with a massively bloated purchase price. If Twitter had quality R&D products that could compete with TikTok, etc., then there may not have been a bloated workforce nor bloated purchase price.
itsoktocry 4y ago
>removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.
- Aeolun 4y ago
  > thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.
  - hef19898 4y ago
    That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.
    - anon400232 4y ago
      Saying everyone is replaceable is a first-degree approximation. The next level of detail would talk about, at least: (a) how much knowledge gets lost; (b) how many person hours -and- "wall clock time" it takes to rebuild that knowledge; (c) the amortized cost (or benefit) of losing that person
    - forgetfulness 4y ago
      It depends on how well or poorly run the business is. I worked at a company where everything hinged solely on one guy working from another country. When he left, loss of institutional knowledge took about three days to show real effects as things also came down crashing. I worked hard to make _myself_ replaceable for when I left, it was a pretty good exercise, but me having that degree of freedom was symptomatic of the problems of the company.
    - blueprint 4y ago
      how can you possibly say everyone is replaceable while next giving an example of a case where it's not only not true but potentially so invisible in its consequences as to make it hard or impossible to satisfactorily replace someone? a "person" can be rebuilt but not replaced. companies and societies aren't actually bose-einstein condensates. everyone is replaceable is just what different types of people say as a coping mechanism. it's a disgusting thing to promulgate too. maybe if someone is equivalent to a cog they can be replaced but humans are not exactly standardized cogs. what you mean to say, and may get in the habit of repeating instead, is that we don't necessarily need a specific person even though we are accustomed to them. it still might take you 10 years to find someone who does 80% of the same things as the other person while the missing 20% is what made that person such a unique hire in the first place. i find it a disgusting phrase that is so typical of the view of a society like ours today that is losing view of the value of life.
throw0101c 4y ago
> In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. It also (a) increases the bus factor, [1] and (b) allows people to take vacations and time off without having to watch their phones like hawk. [1] https://en.wikipedia.org/wiki/Bus_factor https://en.wikipedia.org/wiki/Bus_factor
- donkeyd 4y ago
  Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.
  - celim307 4y ago
    Sounds like those people are in strong negotiation positions
    - polynomial 4y ago
      H-1B would seem to belie that.
      - Rallen89 4y ago
        how would that negotiation go? give us the information we desperately need or we will basically deport you and lose the information anyway!
        polynomial 4y ago
        pretty much, but whatevs.
    - donkeyd 4y ago
      One of them just got promoted quite nicely, so I guess you're right!
thinkmcfly 4y ago
We are entering the windows ME era of Twitter
blueprint 4y ago
if 80% of knowledge is missing due to 80% of people being gone then the team who built it failed to document or automate themselves out of a job, meaning even other people on the team in the good ol' days would still have faced the same issues and merely had to chase down the original authors. that doesn't sound like it would be completely true. maybe 5-20% of un-documented knowledge walked out the door? completely rough guess. but even 0.5% of knowledge can sometimes be critical.
water8 4y ago
lightbendover 4y ago
> and I can keep going You focused mostly on additive bloat, there's also multiplicative bloat in the form of multiple teams focused on building separate versions of the same product to increase likelihood of success and empire building where leaders don't actually have a remit large enough to support the team size they have, but they have woven a narrative that defends the necessity nonetheless. Put everything together and teams are very easily 6x+ larger than they absolutely need to be to get a product into market.
cogman10 4y ago
The whole point of the article is twitter was designed to be resilient. (and it shows, twitter has great uptime). And the whole point of resiliency, beyond not negatively impacting customer experience, is to buy engineers time to fix things when stuff breaks. What we are watching is a massive failure event right now and the question really is if there's enough time for twitter management to fill in the gaps before there's an outage.
bumby 4y ago
Your post made me think of this discussion about how mature organizations naturally have to pivot towards maintenance: https://freakonomics.com/podcast/in-praise-of-maintenance/ https://freakonomics.com/podcast/in-praise-of-maintenance/
neycoda 4y ago
I'd argue that growth relies on stability, and if you don't have stability, you'll randomly lose growth.

twblalock 4y ago

It’s easy to think it’s bloat at a steady state. When something important goes down and nobody knows how to fix it, it looks different.

cm2187 4y ago
That being said it's not like twitter is a massively complex product with lots of different features. I can imagine you could keep it running with a skeletton team. Liasing with ads buyers excepted.
- twelve40 4y ago
  a bunch of people I talk to say it is massively complex but typically fail to explain how, especially given the super-glacial pace at which they added new features for 15 years. And yes, this article kind of doesn't disprove the bloat at all, unless every single SRE quit? but that is also not what's stated in there.
- bryanrasmussen 4y ago
  I'm pretty sure one of the most complicated things is preventing automation of content - bots. which would be an arms race type condition. Bots you want to prevent - people not using twitter's api to do their bot stuff. Why would people not use twitter's api? 1. who would trust twitter not to change API and make code worthless 2. people who want to do stuff Twitter doesn't want you to do in an automated fashion.
- ehnto 4y ago
  I imagine the infrastructure implemented is fairly complex, this post does outline that it's not a super simple operation. It also alludes to there being more business units than just the core application.
- twblalock 4y ago
  Even "simple" things become complex when they need to scale to billions of users around the world, handle a high rate of traffic, deal professionally and legally with all the different jurisdictions they operate in and all the advertisers they serve, etc.

28304283409234 4y ago

Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issues. Not mopping up after them. To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.

tempestn 4y ago
I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by the parent, not trying to make one myself.)
- enkid 4y ago
  So 1-3 people have a big impact, the other 7997 must not be doing anything? I don't think that logic follows.
  - twelve40 4y ago
    it doesn't follow. The article posits that "many people think twitter headcount was bloated" then proceeds to describe a (presumably) really efficient work of a small SRE team. These two parts seem completely disconnected from each other - neither one proves, disproves or follows from the other - so it's unclear why the former was mentioned at all.
    - enkid 4y ago
      His personal experience was zero bloat. He was one deep in a critical function for the company. He isn't saying this proves without a doubt that there is no bloat, but he didn't see any in his time there. It's seems like a reasonable addition to the conversation to me.

oxfordmale 4y ago

Let's do a thought experiment and see what functions aren't needed to keep the light on for 30 days: 1) HR 2) Legal 3) Sales 4) Marketing 5) Payroll 6) Admin staff 7) Most of Engineering, other than the bare minimum of L1/L2/3 support. As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.

jiggawatts 4y ago
This is more like a car with just the bare essentials: an engine, steering wheel, brakes, and the like. No: radio, air-conditioning, seat padding, wipers, lights, radar, etc... Oh, and no maintenance. It'll drive... for now. But that's it.
- sydd 4y ago
  More like a racecar without maintenance. Heres an example checklist what they need: https://www.teamstradale.com/frequency-of-maintenance-keeping-your-racecar-in-tip-top-shape/ https://www.teamstradale.com/frequency-of-maintenance-keepin...
coldtea 4y ago
>As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend. On the other hand, a car without a second and third steering wheel, 20 windscreen wipers, and an oven in the back, keeps running just fine, even after the first bend...
- yywwbbn 4y ago
  True, you might be also forced to sell the windshield, doors and the seats to pay the $4 million in interest each day you didn’t have to before
- nashashmi 4y ago
  Unless the first steering wheel is prone to breaking. Redundancy is the difference between a million dollar company and billion dollar company. Of course getting workers to come on on weekends and midnight hours accomplishes the same thing.

bjarneh 4y ago

> I was partly expecting the rest of the article to explain to my why exactly it wasn't just bloat Same here. I guess his header was on point in why Twitter is still up; but I was also interested in hearing about why Twitter actually needs all those people. If it can be run with 50-80% of the staff gone, that does sound like some bloat at least.

depereo 4y ago
Slack space leads to innovations, like developing infrastructure automation and improving capacity planning. SRE as a practice needs slack space for operations teams to work on improvements and fixes in addition to BAU fault fixing, deployments and patching.
- bjarneh 4y ago
  > Slack space leads to innovations I'm certainly no supporter of "lean operations" with minimum staff etc, and fully agree that you need people that are well rested (for the lack of a better analogy) to do great stuff. But I do think that some of these internet giants do have to many people working there; wasn't LinkedIn 14_000 strong when Microsoft bought it? I've always felt that the American model of doing business is based on how we optimize network traffic, i.e. double the amount of data until failure; then turn in down a bit. Fire on all cylinders until people are truly worn out, then turn down the pace a bit. Haven't worked in the US so I'm pulling this info out of thin air...
  - pas 4y ago
    Well, half of the staff prepares for some conference, meetup, tech talk, or helps organize one, or does 20% time, or sits in unnecessary meetings, or sits in necessary but inefficient meetings, or is on PTO or on unpaid leave in some retreat. Plus above a certain headcount the communication overhead becomes seriously large, so just to compensate for the lost velocity you need to break out into smaller more agile more autonomous teams, which further increases the coordination requirements (thus the comms overhead), but allows overall throughput to scale. And the leading edge technologies commonly used require a large headcount to begin with. (I mean just to start running something twitter/linkedin sized requires at least 1 engineer/million users, so a few hundred folks is a given. You need someone who understands networking, from BGP to TLS to VPN to whatever, internal IT, CI/CD, ops/SRE ... at that scale if you use anything, you need an expert for it. You use Kafka with a hundred millions of users? You might need at least a few people who actually know what the fuck a partition means. Unless you want to just directly give all of your money to Jeff in the form of egress fees, you might need folks to setup CDNs, and whatnot. So without naming names, ina big password manager company (around a hundred million users?) a few years ago there was a certain rewrite project. 3+ people worked on it for 8-10 months, then it was put on hold temporarily. And then obviously nobody speak of it ever. It happens that there are inefficiencies that for months not one line goes into production from certain individuals. It was bad management, yes. But if good management was easy to find then we would be talking about different things :)

weinzierl 4y ago

No matter if it was or not and for better or worse: If Twitter survives this without any major harm it will have profound consequences for the whole software industry.

ravloony 4y ago
This is exactly why this thread is full of slightly insecure comments making vague predictions. I'd suggest to most of them to get off HN and back to work, now is the time to make yourself useful!
pas 4y ago
That's tiny bit of an exaggeration, maaaybe. Maybe completely prophetic, though! That said, Instagram was run by just a dozen people back then, while it had hundreds of millions of users, right? So it's not a new data point. Storing, retrieving, indexing, managing 280 char blobs (with links, threads, embeds) is not exactly the most hardcore of a problem domains. Microblogs are the typical tutorial topic, and twitter's only "innovation" was to increase the text limit.
- gizmo 4y ago
  Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentation.
  - grishka 4y ago
    A lot of things in a typical social media service can be segmented trivially. The hardest part is the feed because it requires querying many database servers at once, but then it can be cached and served quicker.
    - wayfarer2s 4y ago
      Isn’t cache invalidation the hardest part of the feed? Once a user with millions of followers tweets, doesn’t that essentially invalidate millions of caches? That doesn’t sound like a trivial problem.

usgroup 4y ago

I expect it all to work better with Musk in charge. He knows how to make scalable software and he knows about performant teams. That he’s not going to realise these totally obvious first order consequences people are raising seems unlikely.

torben-friis 4y ago
I honestly can’t tell if you’re being sarcastic.
FranzFerdiNaN 4y ago
Everything that is happening with Twitter proves Musk is yet another wealthy idiot who doenst know shit about shit, except how to blow his own horn. Musk is simlpy lucky that Twitter used to have such excellent engineers like this SRE so that the site isnt yet on fire.
- drcross 4y ago
  Musk lead teams at the only company who can land commercial rockets and the first company to profit from making electric cars. To say he is a wealthy idiot says more about you than it does about Musk.
  - yywwbbn 4y ago
    So Musk hired the right people. I mean he made some brilliant decisions over the years. Does that mean that everything he does is a guaranteed success? There are plenty examples of ultra successful people in history who were blinded by their own success and became a little bit unhinged over time.. At this point Musk is behaving just like a twelve year old’s idea of a mega billionaire. Hyperloop, flamethrowers, self driving cars now Twitter... It’s great that he’s having fun I don’t think there is much to it than that.
GoOnThenDoTell 4y ago
You do have to add the /s, else its confusing

carabiner 4y ago

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes. MBA's will be studying this for years.

tempestn 4y ago
Musk might have been right about some things. There probably was some degree of bloat. But to say he's badly mishandled this whole saga is a gross understatement. It is very difficult to utterly kill a site like Twitter; the fact that we're even considering that as a realistic possibility shows just how badly. I think Musk is used to Tesla and SpaceX, which are both companies that a lot of people are (or at least were) excited to work for because they believe in the mission and what's being created. Plus there aren't many alternatives if you want to do that work. Twitter really isn't like that for most people; a Twitter developer has many other options to do similar work. Add to that the fact that he's both cranked up the intensity of the abuse and that it's more visible to everyone, and you can't expect a lot of good people to stick around. And despite the fact that it might coast for quite a while on the back of excellent work in the past, eventually you do need good people to keep a business going. (This is leaving aside the direct impacts of his actions on users and advertisers!)
- ReGenGen 4y ago
  What is the Mission now at twitter? Kiss Lord Musk's boot?
- coldtea 4y ago
  >the fact that we're even considering that as a realistic possibility shows just how badly. Do we have much to go with to "really consider it", or is it just sensenationalist headlines, as Musk went ahead the accepted orthodoxy on Twitter and content moderation and so "he must pay"? That would be the same news outlets that built him up as the real life Tony Stark and propped up his quite ordinary companies, one of which is building electric cars like everybody else, just doing it in style as well-off upper-middle class people's toys, and another is doing space tech that was already a thing we were pursuing 40 years ago with a little modern engineering thrown in...

H8crilA 4y ago

This is a PR article that tries to push the idea that Twitter is OK (to work at, and maybe also to buy ad space from). Damage control.

Lutger 4y ago
I read it as the exact opposite: the reason Twitter is still ok* is not because all these people were just browsing reddit at work. You can't just gut Twitter to run on only a couple of hundred people and still expect the same results in the longterm. Twitter was not a leftist commie welfare company as Musk and its fans want it to be. It was actually the fine work SRE (amongst others) put into it that makes it still tick along as it does...for now. * actually some things are already breaking, but it will take some time for the real damage to surface on a technical level
- pas 4y ago
  It could have been both! :) And seriously, the gaming industry is infamous for its "year round crunch mode". It can be done. Many sectors are "hard work" (ie. public sector, fundamentally non-tech gigacorps with no real in-house IT competence, etc.) Twitter was an innovator. I remember getting into Scala party because of some open source code library from them back then, I don't know, probably around Scala 2.8. That was absolutely unnecessary for their core business. All of this can be done in bad C++, or plain old Java 7, or whatever, for low cost, built in Bangalore. Now that would of course likely not withstand this kind of change storm that Elon caused. But that's a different story. (Also what Elon probably doesn't understand on a technical level (but likely guessed from a business aspect), is how flexible Twitter is. In this regard he hit arrived at the right time. Twitter is fundamentally the same as it was, now it's time to innovate with it, and the architecture he found is exactly the right one for that.)

Lutger 4y ago

I'm puzzled by this statement. Do you think of resiliency as waste? That twitter would have been fine without it? The article makes a point that the reason Twitter is running ok on 20% of personnel at this moment is exactly because it was build to be resilient, not because the personnel was bloated. A large part of this so called bloat, the 80%, was responsible for Twitter to be running right now. Calling this bloat implies it is actually not important for Twitter to be available all the time (or at all).

concordDance 4y ago
How was twitter doing fine a few years back at half the number of employees and same number of users?

adql 4y ago

You just need to look at profit and revenue. Revenue grew nicely in last 2-3 years, profit not so much. Bloat is only reason.

raspberry1337 4y ago
This, in my reflection, is that one insightful comment that should be higher up even it came to late. Twitter userbase did not expand significantly in the last couple of years. Revenue increased. Why did cost increase so much?

nashashmi 4y ago

In addition to the many great comments here, remember that super star engineers don't exactly fix problems from day to day. They fix the problems before they become problems. The engineer was doing stability planning for 6 months out for the purpose of cost optimization. I guess we can assume that the costs of infrastructure is about to go up and reliability is about to go down in the coming months.

Aeolun 4y ago

And so is the cache setup. It’s permanently (and deliberately) running at less than 50% utilization to prevent an issue that comes up only once every 5 years (according to the author).

HeavyStorm 4y ago

Indeed!

ryanfreeborn 4y ago

I have been under the good faith assumption that most (though definitely not all) of the employees that have departed Twitter were probably necessary and valuable to the company. I left the article with the same impression as you. This single person did this very important job, seemingly well, and didn't appear to be drowning in the work. What were the other 8-9k doing?

jameshart 4y ago

That small team seems to have been running the caches for other teams, by using infrastructure provided by another team, in two massive datacenters operated by other teams, using monitoring tools managed by another team, and a ticketing system run by another, on hardware purchases by another team… All just to put caching in front of services that actually do anything.

water8 4y ago
Can’t forget about those people that ordered that hardware once. He probably had to go to the business website and everything.

adam_arthur 4y ago

Well, WhatsApp had ~50 employees and Instagram around ~15 when FB acquired them, and they were around the same order of magnitude of complexity as Twitter. The only concern Id have is that by having so many people, your design probably comes to rely on them whereas a smaller team would be forced to make the system easier to maintain. Personally, if I were Elon, I’d build an entirely new backend and point the clients to that rather than trying to incrementally improve what they have. Get 50-100 10x engineers that are loyal to Elon, with big equity stakes, and crush it

pclmulqdq 4y ago
A lot of people who don't understand why Twitter owns two datacenters point to "complexity" as an argument and completely disregard scale. It turns out that massive scale adds a lot of complexity to a system, particularly around many-to-many pubsub systems (like social media). It also means a lot of features, like compliance with government regulations around the world.
- adam_arthur 4y ago
  WhatsApp had a Billion users and 50 employees. You can say that Twitter is incrementally more complex due to wider broadcasting, but it’s also 1/5 the user count. Everybody stating this number of employees is necessary to maintain a Twitter scale system is simply wrong. And as technology progresses, fewer and fewer people are needed to maintain the same size system Many just don’t want to believe that the leverage in employees favor in the tech sector is fading fast, and their labor is not going to warrant 500k comp with marginal effort anymore
  - pclmulqdq 4y ago
    Whatsapp, discord, telegram, and all other chat apps have a very easy time scaling. Using user count as a metric and comparing them to Twitter is pretty disingenuous. Instagram is a much better comparison, which had 1/50th the user count of Twitter at its acquisition. Chat apps are well known to rely on infrastructure that scales up exceedingly well. That is why there are so many of them, and why they all have 100 tech employees or less. Instagram at the time of the acquisition could have run on 20 servers plus S3. Today's Instagram, along with today's Twitter, does a lot of work that is super-linear in user count, and has something like 2000 engineers. Timeline building is reportedly O(n^2) in user count. The scale difference has a huge effect. Notice that chat apps have no equivalent to "timeline building." The worst scale factor a chat app has is linear.

pron 4y ago

> But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. ... for the Cache component. There are many others.

toss1 4y ago

>>If anything, the article might actually persuade me that it was all bloat. Not for me This is almost exactly like the new manager coming in, noticing that the floors and surfaces are all clean, all the systems work, the trash is emptied, etc., and so deciding that the entire maintenance staff is unnecessary and firing them. The place doesn't become a decrepit pigsty the next morning; it slowly degrades. Same for these systems. They were designed, built, tuned, and maintained over the course of years to go from requiring constant manual intervention to running largely unattended and with a good buffer of ready hardware and automatic failover for failures. That "largely" in "largely unattended" is doing some very heavy lifting. The system WILL require human intervention to keep running, and more than just a skeleton crew. The only question is whether it will happen before the new crew gets up to speed to handle the inevitable degradation. This does NOT mean that the SREs were bloat - it means that they were doing an excellent job and could safely take a break. We're now in just the two-week vacation zone - same as if the entire SRE team went on a holiday. We'd expect it to work. Now let's see what happens in two months.

w0m 4y ago

If you only care about 'it mostly didn't crash' as the end-goal of a company, that would be a reasonable take.

v0idzer0 4y ago

Of course it was bloat. This whole “twitter is going to crash and burn” thing is a weird fantasy. Most likely it will just be run more efficiently by far less people.

lazyant 4y ago

I guess different definitions for "bloat" but how is it bloat to have a tiny team taken care of a fundamental piece of infrastructure? if the team is now gone, an issue there would mean hours of downtime. If that's acceptable then yes, it was "bloat".

halffaday 4y ago

I’m suspicious that most of the value in these systems comes from a small fraction of the effort and many technology jobs boil down to knowing you’re a huge cost center and putting on a performance to hide that.

ouid 4y ago

Of course its all bloat. Software runs on computers, not engineers. The default assumption for software is that it will go on working. The state might devolve, but the software is exactly as reliable right now as it was before. I'm no friend of Elon, and I think its hilarious to think that he can be king of twitter, but all these people talking about "code entropy" are certifiably insane. Big tech maintains talent so that they won't use their knowledge of the system to produce an identical competitor without the technical debt or investor liability of the original.

memish 4y ago

There's an incredible amount of bloat in big tech. It's become an adult daycare, https://twitter.com/DavidSacks/status/1561096423243800576 https://twitter.com/DavidSacks/status/1561096423243800576 Twitter's layoffs followed by that 1AM photo of hackers at work is terrifying to lifestyle employees. It's the Return of the Nerds. Google, Meta, Netflix, Microsoft are all watching.

mylons 4y ago
what does working until 1AM have anything to do with being a nerd or a hacker?