5 ms·

Those 99% of engineers should have warned him about it and didn't. You also can't expect Elon to know that coming in. If you're doing your job properly you don

Those 99% of engineers should have warned him about it and didn't. You also can't expect Elon to know that coming in. If you're doing your job properly you don't build things this way. What it does show is how much dead weight there was at Twitter and how little was lost by letting it go. Even that scale of the servers and monthly infrastructure cost is _absurd_ for the product Twitter is serving.

3seashells 3y ago
[flagged]
AlotOfReading 3y ago
Let he who has never written a permanent "temporary hack" cast the first stone. This is just tech debt. Every company has it, except apparently the ones you've worked for. I'll bet half of these issues are from the fail whale days when every day brought a new existential threat to the company's survival.
- busterarm 3y ago
  Hacks and tech debt happen here and there. 70,000 instances of the same hack speaks to a systemic failure of process and a company culture not focused on continuous improvement. This is a public company used to being analyzed for these types of risks. I also work at a public company. That's why I know that this shit is not okay.
  - Zigurd 3y ago
    Whatever you do don't apply for a job at Amazon. It will be a traumatic experience.
    - busterarm 3y ago
      Already aware. Amazon also gets away with this because of following the Bezos API mandate and these issues each being in isolation. As long as everything keeps functioning at the "api calls between services" level the code can be sick with rot without killing the organism. Some of the best people I've ever worked with come from Amazon though.
      - Zigurd 3y ago
        Amazon is one of the few top tier companies with a generally good reputation where I know engineers who have left voluntarily. The amount of baling wire and duct tape holding it together is only one reason.
        busterarm 3y ago
        On the other hand, here in the operations world we talk about designing for failure all the time. You could make the argument that Amazon has baked into their operating model the failings of human beings and the failings of the processes of teams. Being behind the curtain might be a nightmare but being in front of the curtain might just be the best ticket in town.
  - rsynnott 3y ago
    > and a company culture not focused on continuous improvement. You have to prioritise these things. At a certain point, you often do have to say “well, it’s not ideal, but we’ll know well in advance of it becoming a problem, deal with it then”. This approach is, of course, not compatible with the company being bought by an insane billionaire who thinks he’s an expert, but that’s a bit of a black swan event, and not something that _should_ form part of your prioritisation process.
fzeroracer 3y ago
> What it does show is how much dead weight there was at Twitter and how little was lost by letting it go. If he wasn't told something vitally important, then the odds are the engineers he let go were ones that knew about that vital knowledge. And if you think the scale and infrastructure cost is absurd for Twitter I'd like to see the services you've worked on. Right now.
- busterarm 3y ago
  156,000 individual physical servers (remember, it said 5200 racks * 30) in one building and not being the only building _to serve 140 character tweets_ IS fucking absurd. This is the same company that took a nuclear-fucking-orbit dump on Rails because they didn't understand that your standard relational databases cannot handle their architecture pattern of very high write rate and very high read rate. This is the same company where the same lead engineer's NIH-syndrome led to them writing their own bespoke message queue twice. And they threw it out twice. And I do run 10^5 scale infrastructure -- globally distributed and not with any single building at some eye-watering $100M/yr run rate. It may be a highly visited site, but if you tell me they actually need that much physical stuff to _host tweets_ then I'm calling bullshit. Their quarterly operating costs _not including staff_ were over half a billion dollars. Twice what they were paying in salaries. That's a completely inverted cost relationship to how companies should operate.
  - fzeroracer 3y ago
    First of all, their infrastructure is globally distributed. This was one data center of many. Second of all, this isn't just '140 character tweets'. It's images, video, data being sent at staggering speeds because people tweet and they tweet frequently. And people want instant response, they want to see tweets instantly and respond to them instantly. I highly doubt you do actually run infrastructure given that you're both misrepresenting what's actually going on here and taking the word of a person who in the article alone shows the various different ways of how he distorts reality. This isn't to say Twitter's infrastructure was great, because all services have heavy amounts of tech debt and cruft associated with years of fixes and updates. But rather that data being sent at high speeds with high reliability and at high frequency leads to high costs.
ImPostingOnHN 3y ago
New CEOs entering a company: Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, and doesn't take rash action without considering the consequences Goofus thinks he knows it all, doesn't heed good advice, fires all those people he employs, makes ill-advised, unplanned changes, and blames the resulting problems on someone else
- busterarm 3y ago
  If you buy a totally rotten building, what parts do you save? The foundation's fucked. The floors can be fallen through. There's black mold everywhere. It's probably just the location itself that's valuable. Knocking down walls here and there doesn't actually make a functional bit of difference. I'm not saying Musk isn't a stupid gorilla. I'm saying that anyone with a functioning brain could see from orbit that an organization at that size, with that product, with no customers, with that revenue and those expenses is deeply dysfunctional. The fact that Musk stepped in with real dollars to assume responsibility for that is somewhere between charitable and nonsensical.
  - ImPostingOnHN 3y ago
    emphasis on the first clause, because it seems to be something elmu struggles with Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, doesn't take rash action without considering the consequences, and honors the work put in by all the people who work there Goofus immediately says everything and everyone is rotten without taking time to learn about them, fires most people without taking time to figure out what they do, singlehandedly causes site-wide infrastructure fires pretending he was as capable as the people he fired, never admits he was wrong to do so, continues blaming everyone but himself these are personal qualities, or results of personal qualities. It doesn't matter if elmu's thoughts on the size of the company are wrong or right. He did all the things goofus did, and that's bad. Like, your entire first paragraph is goofus: elmu had no idea of the state of things, he just kept causing problems, blaming them on others, and crying "look at the state of things!” he fired the people who knew the state of things without asking them what it was, and he's not an engineer (hence all the fires he causes), so he certainly can't discern it on his own and the fact that elmu bought twitter after desperately trying to renege on his commitment to do so was purely a result of court action (which he fought against) making the purchase inevitable
saghm 3y ago
If you're doing your job properly and you can't be expected to know something coming in, you shouldn't just make extreme changes wholesale on a whim without any sort of check that it won't break things.
- UncleMeat 3y ago
  And especially not on time limits. In TFA, people tell Musk that this is hard to do and will take time. Musk ignores this, demands that it be done at rapid speed, and doesn't even consider that there is user data on the machines. He was warned.
  - busterarm 3y ago
    And yet nothing catastrophic happened as a result. The platform stumbled for a few months. Some people got bigmad. 80% of their most active users have stayed their most active users since the purchase. I'm so reminded of Carl Icahn's story of firing an entire 9 story building full of people and not hearing a single complaint about it from anyone. Everyone loves to tell you how critical they are. They even feel it to be true, but companies lay important people off all the time and the world goes on. People shouldn't be so invested in their jobs to define their value.
    - fzeroracer 3y ago
      Their advertisers have been leaving in droves, Twitter itself has been breaking itself in new and profound ways since his take over and the site itself was broken (per Musk's own admission) as a result of the move. Profitability is in the dumpster, they're dealing with multiple court cases for refusing to abide by their legal obligations in regards to severance AND rent. And this entire critical move was caused by their lease running out with zero plans in place due to the financial crashing of Twitter. I can't tell exactly where you lie, because you blast Twitter for having incompetent engineers, yet defend Twitter for also having incompetent engineers but this time under Musk. He 'supposedly' cleaned up Twitter (well, the employees, never mind the moderation policies when he personally unbans someone that posts CSAM), but runs it worse than anyone pre-acquisition did.
yongjik 3y ago
> She started to give a lot more details, but after a minute, Musk interrupted. > “This is making my brain hurt,” he said. > “I’m sorry, that was not my intention,” she replied in a measured monotone. > “Do you know the head-explosion emoji?” he asked her. “That’s what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It’s trivial to move servers one place to another.” Have we read the same article?
- rsynnott 3y ago
  > It’s trivial to move servers one place to another Ah, yes, the “confident insistence on something you know nothing about” style of management. This is The Office-esque.

3seashells 3y ago

[flagged]

AlotOfReading 3y ago

Let he who has never written a permanent "temporary hack" cast the first stone. This is just tech debt. Every company has it, except apparently the ones you've worked for. I'll bet half of these issues are from the fail whale days when every day brought a new existential threat to the company's survival.

busterarm 3y ago
Hacks and tech debt happen here and there. 70,000 instances of the same hack speaks to a systemic failure of process and a company culture not focused on continuous improvement. This is a public company used to being analyzed for these types of risks. I also work at a public company. That's why I know that this shit is not okay.
- Zigurd 3y ago
  Whatever you do don't apply for a job at Amazon. It will be a traumatic experience.
  - busterarm 3y ago
    Already aware. Amazon also gets away with this because of following the Bezos API mandate and these issues each being in isolation. As long as everything keeps functioning at the "api calls between services" level the code can be sick with rot without killing the organism. Some of the best people I've ever worked with come from Amazon though.
    - Zigurd 3y ago
      Amazon is one of the few top tier companies with a generally good reputation where I know engineers who have left voluntarily. The amount of baling wire and duct tape holding it together is only one reason.
      - busterarm 3y ago
        On the other hand, here in the operations world we talk about designing for failure all the time. You could make the argument that Amazon has baked into their operating model the failings of human beings and the failings of the processes of teams. Being behind the curtain might be a nightmare but being in front of the curtain might just be the best ticket in town.
- rsynnott 3y ago
  > and a company culture not focused on continuous improvement. You have to prioritise these things. At a certain point, you often do have to say “well, it’s not ideal, but we’ll know well in advance of it becoming a problem, deal with it then”. This approach is, of course, not compatible with the company being bought by an insane billionaire who thinks he’s an expert, but that’s a bit of a black swan event, and not something that _should_ form part of your prioritisation process.

fzeroracer 3y ago

> What it does show is how much dead weight there was at Twitter and how little was lost by letting it go. If he wasn't told something vitally important, then the odds are the engineers he let go were ones that knew about that vital knowledge. And if you think the scale and infrastructure cost is absurd for Twitter I'd like to see the services you've worked on. Right now.

busterarm 3y ago
156,000 individual physical servers (remember, it said 5200 racks * 30) in one building and not being the only building _to serve 140 character tweets_ IS fucking absurd. This is the same company that took a nuclear-fucking-orbit dump on Rails because they didn't understand that your standard relational databases cannot handle their architecture pattern of very high write rate and very high read rate. This is the same company where the same lead engineer's NIH-syndrome led to them writing their own bespoke message queue twice. And they threw it out twice. And I do run 10^5 scale infrastructure -- globally distributed and not with any single building at some eye-watering $100M/yr run rate. It may be a highly visited site, but if you tell me they actually need that much physical stuff to _host tweets_ then I'm calling bullshit. Their quarterly operating costs _not including staff_ were over half a billion dollars. Twice what they were paying in salaries. That's a completely inverted cost relationship to how companies should operate.
- fzeroracer 3y ago
  First of all, their infrastructure is globally distributed. This was one data center of many. Second of all, this isn't just '140 character tweets'. It's images, video, data being sent at staggering speeds because people tweet and they tweet frequently. And people want instant response, they want to see tweets instantly and respond to them instantly. I highly doubt you do actually run infrastructure given that you're both misrepresenting what's actually going on here and taking the word of a person who in the article alone shows the various different ways of how he distorts reality. This isn't to say Twitter's infrastructure was great, because all services have heavy amounts of tech debt and cruft associated with years of fixes and updates. But rather that data being sent at high speeds with high reliability and at high frequency leads to high costs.

ImPostingOnHN 3y ago

New CEOs entering a company: Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, and doesn't take rash action without considering the consequences Goofus thinks he knows it all, doesn't heed good advice, fires all those people he employs, makes ill-advised, unplanned changes, and blames the resulting problems on someone else

busterarm 3y ago
If you buy a totally rotten building, what parts do you save? The foundation's fucked. The floors can be fallen through. There's black mold everywhere. It's probably just the location itself that's valuable. Knocking down walls here and there doesn't actually make a functional bit of difference. I'm not saying Musk isn't a stupid gorilla. I'm saying that anyone with a functioning brain could see from orbit that an organization at that size, with that product, with no customers, with that revenue and those expenses is deeply dysfunctional. The fact that Musk stepped in with real dollars to assume responsibility for that is somewhere between charitable and nonsensical.
- ImPostingOnHN 3y ago
  emphasis on the first clause, because it seems to be something elmu struggles with Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, doesn't take rash action without considering the consequences, and honors the work put in by all the people who work there Goofus immediately says everything and everyone is rotten without taking time to learn about them, fires most people without taking time to figure out what they do, singlehandedly causes site-wide infrastructure fires pretending he was as capable as the people he fired, never admits he was wrong to do so, continues blaming everyone but himself these are personal qualities, or results of personal qualities. It doesn't matter if elmu's thoughts on the size of the company are wrong or right. He did all the things goofus did, and that's bad. Like, your entire first paragraph is goofus: elmu had no idea of the state of things, he just kept causing problems, blaming them on others, and crying "look at the state of things!” he fired the people who knew the state of things without asking them what it was, and he's not an engineer (hence all the fires he causes), so he certainly can't discern it on his own and the fact that elmu bought twitter after desperately trying to renege on his commitment to do so was purely a result of court action (which he fought against) making the purchase inevitable

saghm 3y ago

If you're doing your job properly and you can't be expected to know something coming in, you shouldn't just make extreme changes wholesale on a whim without any sort of check that it won't break things.

UncleMeat 3y ago
And especially not on time limits. In TFA, people tell Musk that this is hard to do and will take time. Musk ignores this, demands that it be done at rapid speed, and doesn't even consider that there is user data on the machines. He was warned.
- busterarm 3y ago
  And yet nothing catastrophic happened as a result. The platform stumbled for a few months. Some people got bigmad. 80% of their most active users have stayed their most active users since the purchase. I'm so reminded of Carl Icahn's story of firing an entire 9 story building full of people and not hearing a single complaint about it from anyone. Everyone loves to tell you how critical they are. They even feel it to be true, but companies lay important people off all the time and the world goes on. People shouldn't be so invested in their jobs to define their value.
  - fzeroracer 3y ago
    Their advertisers have been leaving in droves, Twitter itself has been breaking itself in new and profound ways since his take over and the site itself was broken (per Musk's own admission) as a result of the move. Profitability is in the dumpster, they're dealing with multiple court cases for refusing to abide by their legal obligations in regards to severance AND rent. And this entire critical move was caused by their lease running out with zero plans in place due to the financial crashing of Twitter. I can't tell exactly where you lie, because you blast Twitter for having incompetent engineers, yet defend Twitter for also having incompetent engineers but this time under Musk. He 'supposedly' cleaned up Twitter (well, the employees, never mind the moderation policies when he personally unbans someone that posts CSAM), but runs it worse than anyone pre-acquisition did.

yongjik 3y ago

> She started to give a lot more details, but after a minute, Musk interrupted. > “This is making my brain hurt,” he said. > “I’m sorry, that was not my intention,” she replied in a measured monotone. > “Do you know the head-explosion emoji?” he asked her. “That’s what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It’s trivial to move servers one place to another.” Have we read the same article?

rsynnott 3y ago
> It’s trivial to move servers one place to another Ah, yes, the “confident insistence on something you know nothing about” style of management. This is The Office-esque.