6 ms·

"What I wasn’t told was that we had 70,000 hard-coded references to Sacramento" You can't play the "I wasn't told" card, this is something that I would hope 99

"What I wasn’t told was that we had 70,000 hard-coded references to Sacramento" You can't play the "I wasn't told" card, this is something that I would hope 99% of engineers would have warned about, and if Elon was the "coder" he claims to be he should have known himself

busterarm 3y ago
Those 99% of engineers should have warned him about it and didn't. You also can't expect Elon to know that coming in. If you're doing your job properly you don't build things this way. What it does show is how much dead weight there was at Twitter and how little was lost by letting it go. Even that scale of the servers and monthly infrastructure cost is _absurd_ for the product Twitter is serving.
- 3seashells 3y ago
  [flagged]
- AlotOfReading 3y ago
  Let he who has never written a permanent "temporary hack" cast the first stone. This is just tech debt. Every company has it, except apparently the ones you've worked for. I'll bet half of these issues are from the fail whale days when every day brought a new existential threat to the company's survival.
  - busterarm 3y ago
    Hacks and tech debt happen here and there. 70,000 instances of the same hack speaks to a systemic failure of process and a company culture not focused on continuous improvement. This is a public company used to being analyzed for these types of risks. I also work at a public company. That's why I know that this shit is not okay.
    - Zigurd 3y ago
      Whatever you do don't apply for a job at Amazon. It will be a traumatic experience.
      - busterarm 3y ago
        Already aware. Amazon also gets away with this because of following the Bezos API mandate and these issues each being in isolation. As long as everything keeps functioning at the "api calls between services" level the code can be sick with rot without killing the organism. Some of the best people I've ever worked with come from Amazon though.
        Zigurd 3y ago
        Amazon is one of the few top tier companies with a generally good reputation where I know engineers who have left voluntarily. The amount of baling wire and duct tape holding it together is only one reason.
        busterarm 3y ago
        On the other hand, here in the operations world we talk about designing for failure all the time. You could make the argument that Amazon has baked into their operating model the failings of human beings and the failings of the processes of teams. Being behind the curtain might be a nightmare but being in front of the curtain might just be the best ticket in town.
    - rsynnott 3y ago
      > and a company culture not focused on continuous improvement. You have to prioritise these things. At a certain point, you often do have to say “well, it’s not ideal, but we’ll know well in advance of it becoming a problem, deal with it then”. This approach is, of course, not compatible with the company being bought by an insane billionaire who thinks he’s an expert, but that’s a bit of a black swan event, and not something that _should_ form part of your prioritisation process.
- fzeroracer 3y ago
  > What it does show is how much dead weight there was at Twitter and how little was lost by letting it go. If he wasn't told something vitally important, then the odds are the engineers he let go were ones that knew about that vital knowledge. And if you think the scale and infrastructure cost is absurd for Twitter I'd like to see the services you've worked on. Right now.
  - busterarm 3y ago
    156,000 individual physical servers (remember, it said 5200 racks * 30) in one building and not being the only building _to serve 140 character tweets_ IS fucking absurd. This is the same company that took a nuclear-fucking-orbit dump on Rails because they didn't understand that your standard relational databases cannot handle their architecture pattern of very high write rate and very high read rate. This is the same company where the same lead engineer's NIH-syndrome led to them writing their own bespoke message queue twice. And they threw it out twice. And I do run 10^5 scale infrastructure -- globally distributed and not with any single building at some eye-watering $100M/yr run rate. It may be a highly visited site, but if you tell me they actually need that much physical stuff to _host tweets_ then I'm calling bullshit. Their quarterly operating costs _not including staff_ were over half a billion dollars. Twice what they were paying in salaries. That's a completely inverted cost relationship to how companies should operate.
    - fzeroracer 3y ago
      First of all, their infrastructure is globally distributed. This was one data center of many. Second of all, this isn't just '140 character tweets'. It's images, video, data being sent at staggering speeds because people tweet and they tweet frequently. And people want instant response, they want to see tweets instantly and respond to them instantly. I highly doubt you do actually run infrastructure given that you're both misrepresenting what's actually going on here and taking the word of a person who in the article alone shows the various different ways of how he distorts reality. This isn't to say Twitter's infrastructure was great, because all services have heavy amounts of tech debt and cruft associated with years of fixes and updates. But rather that data being sent at high speeds with high reliability and at high frequency leads to high costs.
- ImPostingOnHN 3y ago
  New CEOs entering a company: Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, and doesn't take rash action without considering the consequences Goofus thinks he knows it all, doesn't heed good advice, fires all those people he employs, makes ill-advised, unplanned changes, and blames the resulting problems on someone else
  - busterarm 3y ago
    If you buy a totally rotten building, what parts do you save? The foundation's fucked. The floors can be fallen through. There's black mold everywhere. It's probably just the location itself that's valuable. Knocking down walls here and there doesn't actually make a functional bit of difference. I'm not saying Musk isn't a stupid gorilla. I'm saying that anyone with a functioning brain could see from orbit that an organization at that size, with that product, with no customers, with that revenue and those expenses is deeply dysfunctional. The fact that Musk stepped in with real dollars to assume responsibility for that is somewhere between charitable and nonsensical.
    - ImPostingOnHN 3y ago
      emphasis on the first clause, because it seems to be something elmu struggles with Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, doesn't take rash action without considering the consequences, and honors the work put in by all the people who work there Goofus immediately says everything and everyone is rotten without taking time to learn about them, fires most people without taking time to figure out what they do, singlehandedly causes site-wide infrastructure fires pretending he was as capable as the people he fired, never admits he was wrong to do so, continues blaming everyone but himself these are personal qualities, or results of personal qualities. It doesn't matter if elmu's thoughts on the size of the company are wrong or right. He did all the things goofus did, and that's bad. Like, your entire first paragraph is goofus: elmu had no idea of the state of things, he just kept causing problems, blaming them on others, and crying "look at the state of things!” he fired the people who knew the state of things without asking them what it was, and he's not an engineer (hence all the fires he causes), so he certainly can't discern it on his own and the fact that elmu bought twitter after desperately trying to renege on his commitment to do so was purely a result of court action (which he fought against) making the purchase inevitable
- saghm 3y ago
  If you're doing your job properly and you can't be expected to know something coming in, you shouldn't just make extreme changes wholesale on a whim without any sort of check that it won't break things.
  - UncleMeat 3y ago
    And especially not on time limits. In TFA, people tell Musk that this is hard to do and will take time. Musk ignores this, demands that it be done at rapid speed, and doesn't even consider that there is user data on the machines. He was warned.
    - busterarm 3y ago
      And yet nothing catastrophic happened as a result. The platform stumbled for a few months. Some people got bigmad. 80% of their most active users have stayed their most active users since the purchase. I'm so reminded of Carl Icahn's story of firing an entire 9 story building full of people and not hearing a single complaint about it from anyone. Everyone loves to tell you how critical they are. They even feel it to be true, but companies lay important people off all the time and the world goes on. People shouldn't be so invested in their jobs to define their value.
      - fzeroracer 3y ago
        Their advertisers have been leaving in droves, Twitter itself has been breaking itself in new and profound ways since his take over and the site itself was broken (per Musk's own admission) as a result of the move. Profitability is in the dumpster, they're dealing with multiple court cases for refusing to abide by their legal obligations in regards to severance AND rent. And this entire critical move was caused by their lease running out with zero plans in place due to the financial crashing of Twitter. I can't tell exactly where you lie, because you blast Twitter for having incompetent engineers, yet defend Twitter for also having incompetent engineers but this time under Musk. He 'supposedly' cleaned up Twitter (well, the employees, never mind the moderation policies when he personally unbans someone that posts CSAM), but runs it worse than anyone pre-acquisition did.
- yongjik 3y ago
  > She started to give a lot more details, but after a minute, Musk interrupted. > “This is making my brain hurt,” he said. > “I’m sorry, that was not my intention,” she replied in a measured monotone. > “Do you know the head-explosion emoji?” he asked her. “That’s what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It’s trivial to move servers one place to another.” Have we read the same article?
  - rsynnott 3y ago
    > It’s trivial to move servers one place to another Ah, yes, the “confident insistence on something you know nothing about” style of management. This is The Office-esque.
3seashells 3y ago
Reminds me of my first start up. Every time we ran out of money or good plans, we had to rearrange the office. Movement is activity is purpose is not dead&gone I guess.
EA-3167 3y ago
I have a sneaking sense that "I wasn't told" is Muskese for "I told people to shut up and didn't listen to them."
- busterarm 3y ago
  It's not an autobiography. Whatever is written to have been said here comes from multiple sources. That also means that whatever isn't written as having been said we know was not said.
  - EA-3167 3y ago
    I have a lot of skepticism when it comes to biographers of living people, compromises are necessarily going to be made for the sake of access.
    - busterarm 3y ago
      What purpose is served by grossly misrepresenting the number of servers in a datacenter and who said what about their status with the very real risk that they might sue you? Elon is the kind of deep-pocketed person that lawyers take cases on pro-bono against.
  - saghm 3y ago
    This is a ludicrous assertion. What if he said something to only one person in privacy, and there literally aren't multiple sources other than him (and the other party to the conversation might have incentive not to divulge anything he didn't approve of)? Not being an autobiography doesn't somehow imply that we know every single thing he's ever said.
    - busterarm 3y ago
      You're talking about a hypothetical scenario that doesn't even match the conversation described. The recounted conversation involved multiple individuals, many of whom have an adversarial relationship with the subject and would be inclined to sue. Misrepresentations of facts could have meaningful reputational damage. If you're the author you do not risk going to publication without verification that the gist of the conversation is accurate. Your editor literally will not let you. The publisher's reputation is on the line too.

busterarm 3y ago

Those 99% of engineers should have warned him about it and didn't. You also can't expect Elon to know that coming in. If you're doing your job properly you don't build things this way. What it does show is how much dead weight there was at Twitter and how little was lost by letting it go. Even that scale of the servers and monthly infrastructure cost is _absurd_ for the product Twitter is serving.

3seashells 3y ago
[flagged]
AlotOfReading 3y ago
Let he who has never written a permanent "temporary hack" cast the first stone. This is just tech debt. Every company has it, except apparently the ones you've worked for. I'll bet half of these issues are from the fail whale days when every day brought a new existential threat to the company's survival.
- busterarm 3y ago
  Hacks and tech debt happen here and there. 70,000 instances of the same hack speaks to a systemic failure of process and a company culture not focused on continuous improvement. This is a public company used to being analyzed for these types of risks. I also work at a public company. That's why I know that this shit is not okay.
  - Zigurd 3y ago
    Whatever you do don't apply for a job at Amazon. It will be a traumatic experience.
    - busterarm 3y ago
      Already aware. Amazon also gets away with this because of following the Bezos API mandate and these issues each being in isolation. As long as everything keeps functioning at the "api calls between services" level the code can be sick with rot without killing the organism. Some of the best people I've ever worked with come from Amazon though.
      - Zigurd 3y ago
        Amazon is one of the few top tier companies with a generally good reputation where I know engineers who have left voluntarily. The amount of baling wire and duct tape holding it together is only one reason.
        busterarm 3y ago
        On the other hand, here in the operations world we talk about designing for failure all the time. You could make the argument that Amazon has baked into their operating model the failings of human beings and the failings of the processes of teams. Being behind the curtain might be a nightmare but being in front of the curtain might just be the best ticket in town.
  - rsynnott 3y ago
    > and a company culture not focused on continuous improvement. You have to prioritise these things. At a certain point, you often do have to say “well, it’s not ideal, but we’ll know well in advance of it becoming a problem, deal with it then”. This approach is, of course, not compatible with the company being bought by an insane billionaire who thinks he’s an expert, but that’s a bit of a black swan event, and not something that _should_ form part of your prioritisation process.
fzeroracer 3y ago
> What it does show is how much dead weight there was at Twitter and how little was lost by letting it go. If he wasn't told something vitally important, then the odds are the engineers he let go were ones that knew about that vital knowledge. And if you think the scale and infrastructure cost is absurd for Twitter I'd like to see the services you've worked on. Right now.
- busterarm 3y ago
  156,000 individual physical servers (remember, it said 5200 racks * 30) in one building and not being the only building _to serve 140 character tweets_ IS fucking absurd. This is the same company that took a nuclear-fucking-orbit dump on Rails because they didn't understand that your standard relational databases cannot handle their architecture pattern of very high write rate and very high read rate. This is the same company where the same lead engineer's NIH-syndrome led to them writing their own bespoke message queue twice. And they threw it out twice. And I do run 10^5 scale infrastructure -- globally distributed and not with any single building at some eye-watering $100M/yr run rate. It may be a highly visited site, but if you tell me they actually need that much physical stuff to _host tweets_ then I'm calling bullshit. Their quarterly operating costs _not including staff_ were over half a billion dollars. Twice what they were paying in salaries. That's a completely inverted cost relationship to how companies should operate.
  - fzeroracer 3y ago
    First of all, their infrastructure is globally distributed. This was one data center of many. Second of all, this isn't just '140 character tweets'. It's images, video, data being sent at staggering speeds because people tweet and they tweet frequently. And people want instant response, they want to see tweets instantly and respond to them instantly. I highly doubt you do actually run infrastructure given that you're both misrepresenting what's actually going on here and taking the word of a person who in the article alone shows the various different ways of how he distorts reality. This isn't to say Twitter's infrastructure was great, because all services have heavy amounts of tech debt and cruft associated with years of fixes and updates. But rather that data being sent at high speeds with high reliability and at high frequency leads to high costs.
ImPostingOnHN 3y ago
New CEOs entering a company: Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, and doesn't take rash action without considering the consequences Goofus thinks he knows it all, doesn't heed good advice, fires all those people he employs, makes ill-advised, unplanned changes, and blames the resulting problems on someone else
- busterarm 3y ago
  If you buy a totally rotten building, what parts do you save? The foundation's fucked. The floors can be fallen through. There's black mold everywhere. It's probably just the location itself that's valuable. Knocking down walls here and there doesn't actually make a functional bit of difference. I'm not saying Musk isn't a stupid gorilla. I'm saying that anyone with a functioning brain could see from orbit that an organization at that size, with that product, with no customers, with that revenue and those expenses is deeply dysfunctional. The fact that Musk stepped in with real dollars to assume responsibility for that is somewhere between charitable and nonsensical.
  - ImPostingOnHN 3y ago
    emphasis on the first clause, because it seems to be something elmu struggles with Gallant humbly admit his lack of knowledge, seeks to learn by asking people he employs, doesn't take rash action without considering the consequences, and honors the work put in by all the people who work there Goofus immediately says everything and everyone is rotten without taking time to learn about them, fires most people without taking time to figure out what they do, singlehandedly causes site-wide infrastructure fires pretending he was as capable as the people he fired, never admits he was wrong to do so, continues blaming everyone but himself these are personal qualities, or results of personal qualities. It doesn't matter if elmu's thoughts on the size of the company are wrong or right. He did all the things goofus did, and that's bad. Like, your entire first paragraph is goofus: elmu had no idea of the state of things, he just kept causing problems, blaming them on others, and crying "look at the state of things!” he fired the people who knew the state of things without asking them what it was, and he's not an engineer (hence all the fires he causes), so he certainly can't discern it on his own and the fact that elmu bought twitter after desperately trying to renege on his commitment to do so was purely a result of court action (which he fought against) making the purchase inevitable
saghm 3y ago
If you're doing your job properly and you can't be expected to know something coming in, you shouldn't just make extreme changes wholesale on a whim without any sort of check that it won't break things.
- UncleMeat 3y ago
  And especially not on time limits. In TFA, people tell Musk that this is hard to do and will take time. Musk ignores this, demands that it be done at rapid speed, and doesn't even consider that there is user data on the machines. He was warned.
  - busterarm 3y ago
    And yet nothing catastrophic happened as a result. The platform stumbled for a few months. Some people got bigmad. 80% of their most active users have stayed their most active users since the purchase. I'm so reminded of Carl Icahn's story of firing an entire 9 story building full of people and not hearing a single complaint about it from anyone. Everyone loves to tell you how critical they are. They even feel it to be true, but companies lay important people off all the time and the world goes on. People shouldn't be so invested in their jobs to define their value.
    - fzeroracer 3y ago
      Their advertisers have been leaving in droves, Twitter itself has been breaking itself in new and profound ways since his take over and the site itself was broken (per Musk's own admission) as a result of the move. Profitability is in the dumpster, they're dealing with multiple court cases for refusing to abide by their legal obligations in regards to severance AND rent. And this entire critical move was caused by their lease running out with zero plans in place due to the financial crashing of Twitter. I can't tell exactly where you lie, because you blast Twitter for having incompetent engineers, yet defend Twitter for also having incompetent engineers but this time under Musk. He 'supposedly' cleaned up Twitter (well, the employees, never mind the moderation policies when he personally unbans someone that posts CSAM), but runs it worse than anyone pre-acquisition did.
yongjik 3y ago
> She started to give a lot more details, but after a minute, Musk interrupted. > “This is making my brain hurt,” he said. > “I’m sorry, that was not my intention,” she replied in a measured monotone. > “Do you know the head-explosion emoji?” he asked her. “That’s what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It’s trivial to move servers one place to another.” Have we read the same article?
- rsynnott 3y ago
  > It’s trivial to move servers one place to another Ah, yes, the “confident insistence on something you know nothing about” style of management. This is The Office-esque.

3seashells 3y ago

Reminds me of my first start up. Every time we ran out of money or good plans, we had to rearrange the office. Movement is activity is purpose is not dead&gone I guess.

EA-3167 3y ago

I have a sneaking sense that "I wasn't told" is Muskese for "I told people to shut up and didn't listen to them."

busterarm 3y ago
It's not an autobiography. Whatever is written to have been said here comes from multiple sources. That also means that whatever isn't written as having been said we know was not said.
- EA-3167 3y ago
  I have a lot of skepticism when it comes to biographers of living people, compromises are necessarily going to be made for the sake of access.
  - busterarm 3y ago
    What purpose is served by grossly misrepresenting the number of servers in a datacenter and who said what about their status with the very real risk that they might sue you? Elon is the kind of deep-pocketed person that lawyers take cases on pro-bono against.
- saghm 3y ago
  This is a ludicrous assertion. What if he said something to only one person in privacy, and there literally aren't multiple sources other than him (and the other party to the conversation might have incentive not to divulge anything he didn't approve of)? Not being an autobiography doesn't somehow imply that we know every single thing he's ever said.
  - busterarm 3y ago
    You're talking about a hypothetical scenario that doesn't even match the conversation described. The recounted conversation involved multiple individuals, many of whom have an adversarial relationship with the subject and would be inclined to sue. Misrepresentations of facts could have meaningful reputational damage. If you're the author you do not risk going to publication without verification that the gist of the conversation is accurate. Your editor literally will not let you. The publisher's reputation is on the line too.