7 ms·

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drast

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people became really invested in the idea that a site cannot possibly stay up without dedicated SREs, as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations.

headsoup 4y ago
Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."
- emodendroket 4y ago
  I don't even doubt that they did a lot of work, but as this article suggests, that's part of the reason why we _wouldn't_ expect the site to just go down in a couple days. If all the servers needed constant manual massaging that would speak more poorly to their work than the other way around.
  - headsoup 4y ago
    I guess that highlights that for some time we just won't know the actual effects and everything right now is just posturing and speculating. If in six months (or suitable time period) we see stagnation in releases and technical issues increasing we'll have some data to infer from.
- AustinDev 4y ago
  I love this comment. It's so salient. It's probably hard for a lot of people on this board to digest but, I feel like most know where you're coming from.
- nemo44x 4y ago
  - kweingar 4y ago
    I will admit to this. I was so angry at how Musk treated his employees, and I took them at face value when they predicted outages. I still think that we will see outages in the coming weeks and months, but probably not in spectacular fashion. A lot of Musk’s fans get emotional about this too. I remember people gloating when they rolled out the new Twitter Blue, saying that Elon got more done in a couple weeks than Twitter had gotten done in years beforehand. And then they shut it off, and now they’re delaying the relaunch. People were praising Musk’s “moderation council” concept, and then he promptly abandoned it to make these decisions by fiat (and in one case, a poll).
- memish 4y ago
  It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees. It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail. Why? What's driving that response?
  - whatshisface 4y ago
    The only way a herd can move is by becoming confident that it is moving.
  - paganel 4y ago
    > put their reputation on the line saying Twitter wouldn't survive the weekend. Was that a real thing those people were saying? Genuinely asking, as I don't follow any of the former Twitter engineers on, well, Twitter, and if there were any such posts/articles on here I must have missed them.
  - JohnBooty 4y ago
    There are a lot of places where the systems would start to fall over if 70-80% of the team departed. Especially since a lot of folks left on bad terms and/or were suddenly terminated. It was the opposite of a smooth handover. So it wasn't unreasonable to think that Twitter would begin experiencing problems. I didn't think that Twitter was going to literally have an unrecoverable system crash and permanently shut its doors, but I thought we'd see some outages or partial breakage over time, which is basically what has happened albeit in an admirably mild way. It comes off as a kind of desperation, as though they need Elon to fail. You don't need to pathologize it, like it's some... deep weirdo psychological yearning. Some people think he's a jerk and wouldn't mind seeing him fall on his face! If a person was staking some significant part of their emotions on their feelings for Musk, yeah, that'd be unhealthy. But I think you are significantly overestimating the emotional weight behind 99.99% of the half-baked Twitter quick takes. It's okay to not like a guy!
    - emodendroket 4y ago
      Just not like him yes, but also probably feel some desire to defend their profession.
throwawaylinux 4y ago
It wasn't only news outlets. A majority of my... politically-noisy tech friends on Facebook went through a recent phase of intensely posting about Twitter on fire and collapsing. Now just because they're "in tech" doesn't mean they have any idea about Twitter, but they should at least know enough to know they don't know what's going to happen, but obviously they're not actually using their brains when posting comments like that. Point is a lot of people opposed to Musk have been participating in a spiraling echo chamber of fairy tales and wishful thinking, it's not just journalists (although clearly they're printing lies with ulterior motvies too, as usual).
- emodendroket 4y ago
  I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office. e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.
  - throwawaylinux 4y ago
    [flagged]
    - freshpots 4y ago
      - throwawaylinux 4y ago
        Yes, I've heard it all before. People who were drawn in to the baseless Russiagate conspiracy theory, and all the other ones have said exactly the same thing to me as you are now. Sure, I'm a secular-heretic for wanting to see evidence for extraordinary assertions from people with obvious ulterior motives, rather than taking them on faith. So far, the conspiracy theorists have had a poor track record as far as I can see. That said, I don't engage or try to debate my friends on it, they have always become extremely upset by any debate on such topics. And they're not hurting anybody so my philosophy is just to leave them to their faith. I don't go around challenging people to prove the existence of God whenever they make a religious Facebook post either, I'm not that kind of asshole.
porknubbins 4y ago
The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.
- sroussey 4y ago
  I had MySQL running on some bare metal for many years without a restart. I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully. Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.
  - kova12 4y ago
    Having single point of failure, and also not even knowing if it will even come up after reboot is a horrible way to run the service
  - deleted 4y ago
    [deleted]
  - kyriakos 4y ago
    Even if your mysql instance and hardware had run indefinitely, if a table is being written to it will eventually run out of disk space or key space and crash. How long it will take depends on the application but it will happen eventually and if no one is around to fix it...
- dcow 4y ago
  I have a piece of Rust software that has not gone down in its entire lifetime.
  - 4y ago
viraptor 4y ago
> as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations Not everyone has the same knowledge and skills. Not everything is documented. Not everything in the documentation is current and correct. (especially after recent changes) It's not even that they won't stay up, but depending on who left the company, the oncall response may look radically different and have different time to resolution.
- emodendroket 4y ago
  Yeah, but I think a judgment is being made that having the service be less reliable is tolerable.
  - kaba0 4y ago
    With that amount of staff change, I’m not sure that service would inevitably come back up at all, in case of a serious enough problem.
hn_throwaway_99 4y ago
I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now: 1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that have 0 people left. 2. More than the technical issues, betting that Twitter will go bankrupt is the easiest bet one can make. Musk saddled Twitter with a shit ton of debt - even if things worked as they did before he had to cut tons of people due to the debt burden. The issue I see is that #2 directly works against #1. Musk has said it will be lots of intense work adding new features to try to raise revenue. But making a ton of changes, probably with lots of shortcuts to get them out the door quickly, especially when so much institutional knowledge has walked out the door, will make keeping the site stable even that much harder.
- mikeweiss 4y ago
  Wait Musk saddled Twitter with debt? Please elaborate...
  - deleted 4y ago
    [deleted]
  - hn_throwaway_99 4y ago
    It was a leveraged buyout. Musk borrowed $13 billion to complete the deal, which Twitter needs to pay back.
    - kaba0 4y ago
      And if he doesn’t, it might effect tesla very negatively (he used his tesla shares as collateral)
      - minhazm 4y ago
        > he used his tesla shares as collateral No he did not. It was initially proposed but there's no margin loan against his TSLA holdings in the final deal. For the debt that Twitter took on in the buyout the collateral is the company itself. There's zero connection to Tesla in the actual financing. https://www.cnbctv18.com/business/who-is-funding-elon-musk-in-44-billion-twitter-acquisition-15036371.htm https://www.cnbctv18.com/business/who-is-funding-elon-musk-i...
fenomas 4y ago
To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...
- wardedVibe 4y ago
  Definitely stealing this as the right way to frame the issue. What would normally be a small kitchen mistake turns into no longer having an apartment complex.
- cratermoon 4y ago
  He also sold all the smoke detectors.
  - dcow 4y ago
    Credit where it’s due though: he did supply a spare sink.
    - gnicholas 4y ago
      You should generally ignore sunk costs.
- quickthrower2 4y ago
  My analogy. Someone purchased some land for $1. Built a house for say $100. And now spends $100,000 a year making it the perfect place to rent, receiving $100,000 a year in rent. Someone comes along and borrows $1m to buy that house. They feel ripped off but eventually are force to go ahead with the purchase. As a result they have to pay $100,000 a year in interest. They need this thing to be profitable! To do this they need to cut back on the $100,000 a year spent. They decide go go in quickly and so email all the services saying "go hard or go home". So the plumbers, tradie, cleaners etc that don't like it leave. As a side hustle also charge visitors to the house $9 to be allowed to wear their bowtie they used to wear for free. Some of the people do maintenance jobs and improvements. They keep the termites out, fix subsidence issues, and so on. And the house didn't fall down within 3 weeks of it being purchased.
lowbloodsugar 4y ago
Riiight. But what happens when there's nobody on that oncall rotation? I've worked at FAANGs, and I'm just thinking what would happen if there was a problem with an upstream team and nobody there. Maybe it's ours now. Maybe there's a principal engineer who worked on it back in the day that's still on the payroll? Maybe we're just going to try to bring it back up with no idea what is going on? What if you have a bug that silently loses data?
DSMan195276 4y ago
> but it was never going to happen overnight and I'm not sure why news outlets were saying that I don't think it's that obvious, it's one thing to just leave everything running but Elon was also talking about changes he was making (Ex. turning off a bunch of "microservices" because they don't do anything). If you turn off the wrong thing and don't have anybody left who knows how to properly turn it back on again, then you're in a pretty bad situation. It doesn't seem like that happened, but I don't also think we have enough information to say how close it was to happening.
edgyquant 4y ago
This whole thing has been quite telling for me. I don’t use Twitter and am not a Musk fan, but seeing someone try something to move away from ad driven seemed like a good possibility for me. Then I saw how bitter and nasty a lot of the online communities I belonged to were. I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue. I was met with people attacking me as an idiot with clickbait articles of “proof” that ad models were irrelevant to twitters problems etc, as if I was doing anything other than quoting musks official reasons for the purchase. Suddenly Twitter was this great beacon of graceful discussion and Musk has ruined it. People just love to be mad, no one had anything nice to say about Twitter and in a flick of a switch they’re holding completely opposite opinions. We’ve always been at war with Eurasia, it seems
- vba616 4y ago
  >I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue In the computer industry, there are some famous historical examples of companies that announced their new product before it was ready, people stopped buying the old one, and they went out of business before the new version was done. Not quite the same business, but a bit reminiscent.

headsoup 4y ago

Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."

emodendroket 4y ago
I don't even doubt that they did a lot of work, but as this article suggests, that's part of the reason why we _wouldn't_ expect the site to just go down in a couple days. If all the servers needed constant manual massaging that would speak more poorly to their work than the other way around.
- headsoup 4y ago
  I guess that highlights that for some time we just won't know the actual effects and everything right now is just posturing and speculating. If in six months (or suitable time period) we see stagnation in releases and technical issues increasing we'll have some data to infer from.
AustinDev 4y ago
I love this comment. It's so salient. It's probably hard for a lot of people on this board to digest but, I feel like most know where you're coming from.
nemo44x 4y ago
- kweingar 4y ago
  I will admit to this. I was so angry at how Musk treated his employees, and I took them at face value when they predicted outages. I still think that we will see outages in the coming weeks and months, but probably not in spectacular fashion. A lot of Musk’s fans get emotional about this too. I remember people gloating when they rolled out the new Twitter Blue, saying that Elon got more done in a couple weeks than Twitter had gotten done in years beforehand. And then they shut it off, and now they’re delaying the relaunch. People were praising Musk’s “moderation council” concept, and then he promptly abandoned it to make these decisions by fiat (and in one case, a poll).
memish 4y ago
It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees. It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail. Why? What's driving that response?
- whatshisface 4y ago
  The only way a herd can move is by becoming confident that it is moving.
- paganel 4y ago
  > put their reputation on the line saying Twitter wouldn't survive the weekend. Was that a real thing those people were saying? Genuinely asking, as I don't follow any of the former Twitter engineers on, well, Twitter, and if there were any such posts/articles on here I must have missed them.
- JohnBooty 4y ago
  There are a lot of places where the systems would start to fall over if 70-80% of the team departed. Especially since a lot of folks left on bad terms and/or were suddenly terminated. It was the opposite of a smooth handover. So it wasn't unreasonable to think that Twitter would begin experiencing problems. I didn't think that Twitter was going to literally have an unrecoverable system crash and permanently shut its doors, but I thought we'd see some outages or partial breakage over time, which is basically what has happened albeit in an admirably mild way. It comes off as a kind of desperation, as though they need Elon to fail. You don't need to pathologize it, like it's some... deep weirdo psychological yearning. Some people think he's a jerk and wouldn't mind seeing him fall on his face! If a person was staking some significant part of their emotions on their feelings for Musk, yeah, that'd be unhealthy. But I think you are significantly overestimating the emotional weight behind 99.99% of the half-baked Twitter quick takes. It's okay to not like a guy!
  - emodendroket 4y ago
    Just not like him yes, but also probably feel some desire to defend their profession.

throwawaylinux 4y ago

It wasn't only news outlets. A majority of my... politically-noisy tech friends on Facebook went through a recent phase of intensely posting about Twitter on fire and collapsing. Now just because they're "in tech" doesn't mean they have any idea about Twitter, but they should at least know enough to know they don't know what's going to happen, but obviously they're not actually using their brains when posting comments like that. Point is a lot of people opposed to Musk have been participating in a spiraling echo chamber of fairy tales and wishful thinking, it's not just journalists (although clearly they're printing lies with ulterior motvies too, as usual).

emodendroket 4y ago
I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office. e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.
- throwawaylinux 4y ago
  [flagged]
  - freshpots 4y ago
    - throwawaylinux 4y ago
      Yes, I've heard it all before. People who were drawn in to the baseless Russiagate conspiracy theory, and all the other ones have said exactly the same thing to me as you are now. Sure, I'm a secular-heretic for wanting to see evidence for extraordinary assertions from people with obvious ulterior motives, rather than taking them on faith. So far, the conspiracy theorists have had a poor track record as far as I can see. That said, I don't engage or try to debate my friends on it, they have always become extremely upset by any debate on such topics. And they're not hurting anybody so my philosophy is just to leave them to their faith. I don't go around challenging people to prove the existence of God whenever they make a religious Facebook post either, I'm not that kind of asshole.

porknubbins 4y ago

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.

sroussey 4y ago
I had MySQL running on some bare metal for many years without a restart. I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully. Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.
- kova12 4y ago
  Having single point of failure, and also not even knowing if it will even come up after reboot is a horrible way to run the service
- deleted 4y ago
  [deleted]
- kyriakos 4y ago
  Even if your mysql instance and hardware had run indefinitely, if a table is being written to it will eventually run out of disk space or key space and crash. How long it will take depends on the application but it will happen eventually and if no one is around to fix it...
dcow 4y ago
I have a piece of Rust software that has not gone down in its entire lifetime.
- 4y ago

viraptor 4y ago

> as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations Not everyone has the same knowledge and skills. Not everything is documented. Not everything in the documentation is current and correct. (especially after recent changes) It's not even that they won't stay up, but depending on who left the company, the oncall response may look radically different and have different time to resolution.

emodendroket 4y ago
Yeah, but I think a judgment is being made that having the service be less reliable is tolerable.
- kaba0 4y ago
  With that amount of staff change, I’m not sure that service would inevitably come back up at all, in case of a serious enough problem.

hn_throwaway_99 4y ago

I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now: 1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that have 0 people left. 2. More than the technical issues, betting that Twitter will go bankrupt is the easiest bet one can make. Musk saddled Twitter with a shit ton of debt - even if things worked as they did before he had to cut tons of people due to the debt burden. The issue I see is that #2 directly works against #1. Musk has said it will be lots of intense work adding new features to try to raise revenue. But making a ton of changes, probably with lots of shortcuts to get them out the door quickly, especially when so much institutional knowledge has walked out the door, will make keeping the site stable even that much harder.

mikeweiss 4y ago
Wait Musk saddled Twitter with debt? Please elaborate...
- deleted 4y ago
  [deleted]
- hn_throwaway_99 4y ago
  It was a leveraged buyout. Musk borrowed $13 billion to complete the deal, which Twitter needs to pay back.
  - kaba0 4y ago
    And if he doesn’t, it might effect tesla very negatively (he used his tesla shares as collateral)
    - minhazm 4y ago
      > he used his tesla shares as collateral No he did not. It was initially proposed but there's no margin loan against his TSLA holdings in the final deal. For the debt that Twitter took on in the buyout the collateral is the company itself. There's zero connection to Tesla in the actual financing. https://www.cnbctv18.com/business/who-is-funding-elon-musk-in-44-billion-twitter-acquisition-15036371.htm https://www.cnbctv18.com/business/who-is-funding-elon-musk-i...

fenomas 4y ago

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

wardedVibe 4y ago
Definitely stealing this as the right way to frame the issue. What would normally be a small kitchen mistake turns into no longer having an apartment complex.
cratermoon 4y ago
He also sold all the smoke detectors.
- dcow 4y ago
  Credit where it’s due though: he did supply a spare sink.
  - gnicholas 4y ago
    You should generally ignore sunk costs.
quickthrower2 4y ago
My analogy. Someone purchased some land for $1. Built a house for say $100. And now spends $100,000 a year making it the perfect place to rent, receiving $100,000 a year in rent. Someone comes along and borrows $1m to buy that house. They feel ripped off but eventually are force to go ahead with the purchase. As a result they have to pay $100,000 a year in interest. They need this thing to be profitable! To do this they need to cut back on the $100,000 a year spent. They decide go go in quickly and so email all the services saying "go hard or go home". So the plumbers, tradie, cleaners etc that don't like it leave. As a side hustle also charge visitors to the house $9 to be allowed to wear their bowtie they used to wear for free. Some of the people do maintenance jobs and improvements. They keep the termites out, fix subsidence issues, and so on. And the house didn't fall down within 3 weeks of it being purchased.

lowbloodsugar 4y ago

Riiight. But what happens when there's nobody on that oncall rotation? I've worked at FAANGs, and I'm just thinking what would happen if there was a problem with an upstream team and nobody there. Maybe it's ours now. Maybe there's a principal engineer who worked on it back in the day that's still on the payroll? Maybe we're just going to try to bring it back up with no idea what is going on? What if you have a bug that silently loses data?

DSMan195276 4y ago

> but it was never going to happen overnight and I'm not sure why news outlets were saying that I don't think it's that obvious, it's one thing to just leave everything running but Elon was also talking about changes he was making (Ex. turning off a bunch of "microservices" because they don't do anything). If you turn off the wrong thing and don't have anybody left who knows how to properly turn it back on again, then you're in a pretty bad situation. It doesn't seem like that happened, but I don't also think we have enough information to say how close it was to happening.

edgyquant 4y ago

This whole thing has been quite telling for me. I don’t use Twitter and am not a Musk fan, but seeing someone try something to move away from ad driven seemed like a good possibility for me. Then I saw how bitter and nasty a lot of the online communities I belonged to were. I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue. I was met with people attacking me as an idiot with clickbait articles of “proof” that ad models were irrelevant to twitters problems etc, as if I was doing anything other than quoting musks official reasons for the purchase. Suddenly Twitter was this great beacon of graceful discussion and Musk has ruined it. People just love to be mad, no one had anything nice to say about Twitter and in a flick of a switch they’re holding completely opposite opinions. We’ve always been at war with Eurasia, it seems

vba616 4y ago
>I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue In the computer industry, there are some famous historical examples of companies that announced their new product before it was ready, people stopped buying the old one, and they went out of business before the new version was done. Not quite the same business, but a bit reminiscent.