6 ms·
Ask HN: How long could Twitter operate if the remaining staff quit?
In almost every organization there are roles that could be trimmed, but it seems extreme that Twitter is potentially laying off half the company today.
Can the company survive that?
If the remaining team is over worked or feels like they are not respected… what would happen if they all left en mass?
- LinuxBender 4y agoI've been through this a few times. It's very bumpy. Work flows will get interrupted and the remaining leadership will learn quickly to prioritize the bare minimum to keep the lights on. Some will fight the situation and they will also be leaving. The remaining individual contributors will feel stunned and dazed for a bit. Some people will be on edge and/or depressed for a while. I would expect in this case that Elon would bring in people that he trusts from his other businesses. Those people will feel awkward and uncomfortable. With time the remaining Twitter staff and the new employees will learn to get along, realizing they are all individual contributors and that this is not their fault. With time, things smooth out and the business evolves. The most bumpy parts are a result of the people that walked out the door with undocumented tribal knowledge. I greatly over-simplified the answer but this topic is worthy of a book.
- yrgulation 4y ago> I greatly over-simplified the answer but this topic is worthy of a book. Is it tho? In any company small and large once ownership of high level management changes hiccups can happen.
- LinuxBender 4y agoIs it tho? Only if one wanted a more thorough and in depth picture of what this typically looks like day-to-day, week-to-week. Hiccups happen but when a company has been around for a long time and processes are very entrenched, a take-over can disrupt business processes and work flows enough for customers/consumers to feel it. The business will learn with time what undocumented relationships it has/had with vendors and what contractual obligations must be met that may not have been accounted for. Someone will discover for example some message-queue server that nobody documented or knew about when it finally fails and turns out all of Twitter depends on it. This is not hypothetical, I found two very old workstation-class "servers" under a CTO's desk in a financial company [Edit] I should also add that since this is a take-over and not a chapter 11 reorg, there are no legal protections for contractual obligations so all of those will have to be discovered and addressed. Not doing so in a timely manor can be extra bumpy or costly. One can only hope they had good record keeping. Experienced leaders that have been through mergers and take-overs should not really face any problems they can not solve, but lesser experienced leaders will have opportunities to make some monumental and profound mistakes. This is the gap that leaders could benefit from reading books on the topic and even then some leaders will misinterpret the intentions of the books or will not get the lessons to be learned but it is still better than trying to apply the logic from ones past experiences alone. Leadership techniques and business logic that worked at one company may not translate appropriately to Twitter. Along that line I should also add that leaders should not literally use such books as step-by-step instruction manuals but rather they should apply some critical thinking how does this translate to the current situation.
- yrgulation 4y agoI see your point, makes sense.
- lolinder 4y agoWhich is exactly why a book on the topic would be well received!
- juancn 4y agoYou forget the key-retainers. This subset of the org, were set aside first, informed of their condition and promised something so they stay 1 or 2 years and plow through the upheaval.
- LinuxBender 4y agoIndeed, the golden hand-cuffs. I left out a myriad of events or my comment would evolve into a book. I'm trying to get better about not doing that.
- fragmede 4y agoOr, write the book and sell that. :)
- andyish 4y agoThey'll figure something out, but my money is on quick fixes will run rampant and tech debt will grow. When they need someone to fill a specific role, they will no doubt go to the nearest consultancy and spend 5x as much on temporarily filling the role. Or... rehire the same roles on a lower salary.
- viggity 4y agoregional grocery store chain did a huge IT layoff, it was terribly orchestrated. I knew a bunch of people there. They all left. IIRC their web store went from 50-ish devs to a single intern (this was 9 months ago-ish). Somehow it is still online, it boggles my mind.
- PM_me_your_math 4y agoThat's because it is likely the system became bloated. If you can still remain online after such a purge, you needed one.
- ebiester 4y agoOr it's well architected. Most of the time, developers are ideally working on new value additions rather than pure maintenance. The problem happens when something breaks - let's say your database changes the execution plan and it crawls to a halt. I've seen this happen out of the blue and it was a few hours of work to figure out a solution. If you don't have a developer who understands the code, you might be sunk.
- PM_me_your_math 4y agoFor sure, you're in a word of hurt if you don't have someone who can get it working.
- tfandango 4y agoOr the intern is pretty good :)
- twelve40 4y agoi hope they made him a CTO after this
- Jamie9912 4y agoIf every twitter employee collapsed and died right now how long would twitter actually still work for?
- nerpderp82 4y agoThis isn't twitter specific, but from what I have seen, things can fail for the following reasons. Something that autoupgrades would become out of spec, refuse to start due to a change in config files. A counter would roll over somewhere. A disk fills up. A cert expires. A domain name entry expires. Could be days/weeks/months.
- grahamlee 4y agoMy guess is not very long. I've been on a project with three people where if we all quit, it would run until AWS EKS switched off the version of Kubernetes we were using; I'd expect from experience at FB that there's way more done to keep plates spinning on the Twitter fleet and within a couple of days there would be a SEV that no-one was around to respond to. Of course they'd actually be pulled down by law enforcement much quicker than that if all their content mods left.
- seibelj 4y agoWhen things start getting really bad you make retention bonuses widespread. Was at a collapsing company and we all got a years salary paid quarterly for staying, but if you left before a year had to pay it back. Everyone was pleased.
- ebiester 4y agoIt depends on how Twitter is architected. Most teams need about 10% of staff for skeleton maintenance. You can last a little bit with one staff to restart things as they go, but you'll have a quick degradation of service. If certain microservices fail, it may be the advertizing not working right or something failing with billing or moderation or some other internal tool, but a user might just notice that something is slightly wonky. However, you won't be able to change much at that point. Changes become exceedingly difficult because of the maintenance overhead.
- sgc 4y agoTwitter must have to deal with a lot of hacking attempts. If they go into that mode, I could see them falling victim to semi-regular disruptions to service and higher profile account takeovers, that would push at least some users away from the service. I could see them failing in ways many companies in the same situation would not.
- itronitron 4y agoIf they all left en masse I would expect Twitter to stop working within 48 hours due to the scale at which Twitter operates.
- bpodgursky 4y agoI suspect you could get enough core devops to stay onboard by just quadrupling their salary, even if it's just for a couple quarters. It's messy but there are pretty straightforward levers to pull.
- ianai 4y agoI hate to take this side, but I do wonder how it took 7500 people to run twitter. Scott Galloway made this point yesterday in his Prof G podcast. It’s not like they had to pipe data ultimately efficiently like a Netflix, for example. It’s more like a message routing service. Seems do-able with 200 people.
- dagmx 4y agoEasy, you’re only considering engineers for the core technical product. Now consider sales, content moderation, HR, lawyers. Now multiply that by multiple countries that need servicing.
- socialismisok 4y agoAdd data center management, scaling, developer tools, reporting, advertiser portals, stats aggregation, ad bidding, etc. 7500 people is 750 teams of ten. It's really a rather moderate engineering organization for a problem of their scale.
- Onewildgamer 4y agoI agree, I'm assuming their engineering teams are the biggest in size like my company which is of a very comparable size with legal, hr, ops and business counterparts. But coming to Twitter engineering teams, what are they doing actually? their primary technical product is tweet timelines and trends, their mobile apps are kind of okay, the user experience hasn't changed much except getting slower and slower. If all they do is rewrites, they are simply wasting the talent. There could be SRE, Security, Datacenter admins running 24x7 but other than that, I see no useful idea out of them in a long long while. See instagram, snapchat and others for example, they've explored the unknown and brought us some quite good and plenty bad social media. Twitter at least should've been masters in content moderation by now, but they're far from it. For an outsider they seem to have lost their way and are being a couch potato munching on the junk food.
- shiftpgdn 4y agoI think you'd need a senior SRE specifically at twitter to answer this. I know of a company that fired all of their developers and operations people a year ago and everything still works mostly. They contract out as needed. I think the real issue is that without heavy content moderation it'll quickly get overrun with spam. A great example is to go find an abandoned subreddit and see what bad shape it's in.
- blackaspen 4y agoI was an SRE there (even senior SRE!) -- left in 2018 but have still been pretty in-the-loop for the past few years due to friends there. The question was, how long would thing stay up if everything stopped: the answer was <1 week for everything (including ads, things you don't see, etc). Maybe 2 or 3 for most core functionality. With some of their service issues in the past few weeks (DC stuff that's been publicized), it could be less. To the very best of my knowledge no substantial resiliency work has been done recently. I do know that a good number of folks on-call for core services were laid off while on-call, so, that bodes well. I feel bad for everyone left trying to keep things running.
- shiftpgdn 4y agoThat's very interesting that there isn't enough resiliency baked in to keep it going forever.
- rented_mule 4y agoIn something of this scale, "enough resiliency baked in to keep it going forever" is not possible. There are many reasons for this... One is that hardware fails and needs to be replaced. That requires people who know how to install the replacement hardware and deploy to it. That's assuming the new hardware is 100% compatible - that won't be the case for more than a handful of years. Another is currently unknown security vulnerabilities, whether in their own code or in external packages they use. Those vulnerabilities are there and they will be discovered. Once they are, things start being taken down from the outside until the system collapses. Yet another is bugs. Every system of this scale has a large number of bugs, many of them unknown. Some of those won't be discovered until the right conditions arise - the right combination of data, timing, etc. When they are finally triggered, some of those bugs will take down entire subsystems, some of which are critical to the product functioning. There are many more examples like this. There is no such thing as indefinite resiliency for anything near this scale.
- laweijfmvo 4y agoAny reasonable, mature company should have a Keep-the-lights-On playbook, especially after 2020…
- Macha 4y agoI honestly expect we'll see an increase in errors propogating all the way to users within the next week. If the other half quit, I'd expect a hard downtime within a month
- reacharavindh 4y agoI’d guess a long time! The way we engineers build systems is to have a solid core and then a lot of fancy cruft to handle the constant stream of inevitable changes that bring in new features or update stuff. Take the changes away by removing people who’d roll them out, and you have the core running as it was designed to do. I believe when the remaining engineers or the new ones(if they will be..) start rolling out changes is when shit will break and make the whole thing wildly unpredictable.
- yardie 4y agoThis might be true for internal applications. Once you plug anything into the Internet all bets are off. Who's auditing for vulnerabilities, patching, rollbacks. Changes aren't made on a whim. There is either a business, strategic, or security need for them. And it's that last one where everyone is worried. No system is perfect. It's how you respond to them that matters and if no human is in the loop what sort of response can you expect?
- Curzel 4y agoLayoff was probably 50% accounting for another chunk of people leaving soon after, so that staff reduction will be 60-70% in total. Also, I don’t think the majority of people even work on core products.. for example, did you know Twitter implemented their own OnlyFans competitor? It never launched afaik Step 1 is to trim as much as possible and consolidate core functionality… Step 2 is back to profitability. I understand that, let’s see if it works
- blindriver 4y agoI suspect a large percentage of "surviving" Twitter employees will enjoy working for Musk. I've worked in environments where there was a lot of unproductive, highly paid people and it was very frustrating, especially if you believed in the company. Twitter definitely seems very cushy from what I've heard from my friends that used to work there. I would be shocked if they actually had a 50% layoff, from the news reports and everything I've seen on social media, it's not that deep otherwise there would be far larger reports of it by now. If it turns out that 50% number is wrong, I think that would really open the eyes of Twitter employees as to how much the media is spreading lies about Musk. If it turns out the 50% number is true, I'm pretty sure the company won't survive. Having been through layoffs, the rest of the employees of such a huge cut will likely not work or not even care and immediately start looking for a new job because they know they are next. That's not a wise way to run a company, in my opinion.
- amriksohata 4y agoTwitter ain't going to collapse even if many left, it's a business. Profitable, and can hire quick They might slow down but not stop operating
- tschwimmer 4y agoIronically, I think some of the mudge leaks can cast light on this. One of the allegations he made was that "Twitter faced a non-negligible existential risk of even brief simultaneous, catastrophic data center failure, and had no workable disaster recovery plan."[0] Without more details, it's impossible to say but to me, this speaks to somewhat fragile infrastructure for an organization of this size and importance. I've worked at organizations with and without strong cold start plans, but the fact that Twitter didn't have one suggests it might difficult to do given the maturity of their infra. [0] https://s3.documentcloud.org/documents/22186683/twitter-whistleblower-disclosure.pdf https://s3.documentcloud.org/documents/22186683/twitter-whis... page 28
- bell-cot 4y ago> Can the company survive that? Yes, unless the folks in charge are incredibly incompetent. > what would happen if they all left en mass? For Twitter, that is an "...if a giant asteroid vaporized San Francisco" fantasy scenario. The remaining workers are not a clone army, and vary in all sorts of ways. Including what it would take to convince them to leave ~immediately. Elan has years of experience leading very large, technology-oriented companies - he is quite aware of the wide, grey optimization zones around "be very demanding of and difficult with your employees".
- freefaler 4y agoFrom my experience (I've build and sold 2 companies) there are 2 factors increasing headcount in larger organisations: 1. Layers of management and organizational dynamics push up the number of people. There is a correlation between the number a manager manages and his status in the company. The larger the team - the larger your budgets are - the more important you are. Internal goals (to rise in the organization) override external goals (sell and generate profits). When owners are leaving the company or have already cashed out, their motivation to keep financial discipline is lowered and management quality decreases. Owners differ from the hired managers because they prioritize external goals (sales, profit, long-term company health) not their own personal employee goals. 2. There is a strange fallacy that when you get your VC capital you need to increase your headcount to be perceived as a "serious" company. If you are 10 people working remotely you won't be able to ask for the same valuation in your second or third round as a "looks like a big company" startup with an office and 100 employees. Investors would push (sometimes inadvertenly) founders to grow personel at higher rate, because investors sell the company to other investors. They package the company to exit, they don't care that much about sales and profitability except as a baseline numbers for the next round valuation. I've seen this several times. The contrapoint is that Instagram had 13 people when it was sold to Facebook and Whatsapp had around 40 people. Twitter has 7500 employees. There's got to be a lot of layers of management, processes that can be removed from the company. For Elon, fixing cashflow now is more important that keeping everything working smoothly. It works smoothly to generate loss every quarter. It's easier to cut the losses than generate revenue. It makes perfect sense.
- pwinnski 4y agoExcept that the financing he used to buy Twitter require him to double net revenue immediately[0]. It's not clear he can cut expenses by more than 50% without negatively impacting revenue. And that's before a massive drop[1] coming from how badly he's handled the acquisition, as well as his personal reputation. 0. https://www.nytimes.com/2022/10/30/technology/elon-musk-twitter-debt.html https://www.nytimes.com/2022/10/30/technology/elon-musk-twit... 1. https://nypost.com/2022/11/04/elon-musk-says-twitter-has-seen-massive-drop-in-revenue-blames-activist-groups/ https://nypost.com/2022/11/04/elon-musk-says-twitter-has-see...