6 ms·

Twitter, from day one, has been poorly architected. Defending that is pretty stupid and then to do it publicly is, well, kind of asking for it.

sho_hn 4y ago
I'm coming at this from a different angle. It should be possible to have a heated disagreement, even in public, and not have it result in a same-day firing. In fact, some countries have it explicitly on the law books. All this does is create an environment where people are afraid to voice opinions and address issues, e.g. because they're afraid of shoot-the-messenger. In my line of work (I make cars), dysfunctional discourse environments like that can get people killed.
- david927 4y ago
  Sure, and I'm a big proponent of labor. Musk is certainly not. That said, Twitter's architecture is a Rube Goldberg contraption and Musk is clearly on a warpath to fix it, so to come out publicly and yell, "It's fine," is kind of crazy. It's not fine and you're saying you're not interested in fixing it.
  - sho_hn 4y ago
    Looking at the exchange, he did confirm that it's slow on Android/iOS. He claimed it's for other reasons and gave examples. He might be wrong (I haven't looked at Twitter's architecture, commentary on the technical content of the exchange would be interesting), but I wouldn't say he said "It's fine".
    - david927 4y ago
      He said it's "net spend" and that's another way of saying, "It's fine." It's the equivalent of admitting the engine is running hot, but saying, it's not a problem with the engine, it just needs lots of oil." Frankly, I think the developer is embarrassed. There's no reason for it to be as slow as it is, other than he and the other engineers simply don't know what they're doing. I think Musk is calling it out because he wants to see who wants to fix it and who wants to defend it.
      - sho_hn 4y ago
        I think you misread "Ad spend" as "net spend" in his tweet (I can't locate any using the phrase "net spend"). He mentioned that performance work correlates well with "UAM and Ad spend", which to me reads as "it's worth working more on this problem" and is indicating the willingness you're looking for. From what I can see, he offered an opinion that the number of client requests (he gives a number of 20) and the number of internal RPCs (he gives 200) is not the primary reason the app is slow. I can't tell you whether he's right or wrong.
        david927 4y ago
        Thanks for the correction. You're right, and I can't tell you either, so I shouldn't really argue.
        sho_hn 4y ago
        FWIW, if he actually had written "just throw more money at it to make it go faster", I would understand your more negative opinion. :-)
        david927 4y ago
        I mean how does he know it's not "thousands of RPC calls" -- he spent six years and never got it to perform, so he doesn't know. "You're wrong!" ? Based on what? If I worked for six years on something and it still didn't work and someone came in and said it's "x", I would say, "I'm excited to see that data because I really want to fix this." I think I'm more upset at his attitude than anything. In all honesty, I think what he meant was "I spent six years and failed to fix something that is fixable, and then outside consultants came in a clearly found clear problems in a couple weeks, and now I'm humiliated," but he has too big of an ego for that.
        jurassic 4y ago
        The guy's job was not specifically performance; he didn't fail at anything. Product managers drive roadmaps so even if you know there's been a performance problem for years, you might not get to invest time in fixing it. That's exactly what Eric was advocating if you read the whole thread.
- viig99 4y ago
  Clearly these engineers have never done customer feedback for developing nations, twitter interactions clearly provide evidence for what musk said, but some engineers seem to be in denial, imho the firing is cause worthy.
  - sho_hn 4y ago
    I think that's misreading the exchange. He was challenging Musk on the fact that "1000 RPC calls" isn't what's happening and isn't the reason for the slowdown in his opinion, but did not deny that there are performance issues - and even cited other reasons it's slow. I don't see the denial you see.
    - viig99 4y ago
      Said engineer, said 1000 RPC calls are not the issue, and when questioned further on proving a tracepath / perf result casually exited the conversation, to me it looks like they are not ready to do better or listen to valid criticism, leading conversations on semantics is just creating noise instead of acting on signals. Also 1000 RPC's are a very valid performance bottleneck, 1000 * latency caused by serde+ network can compounds easily based on data locality + container network orchestration, no data was provided otherwise even when asked for.
      - fragmede 4y ago
        Except that you can open the dev console in a browser and load Twitter.com yourself and you won't find 1000's of XHR requests.
        sho_hn 4y ago
        That's client-side requests. Musk was talking about internal RPC calls to microservices needed to ultimately respond to the client-side requests. If the internal arch is heavy on microservices and you're putting together, say, 50 tweets you're at 1000 RPC calls with just 20 per tweet. It's not inconceivable. The developer puts the real number at 200.
- FormerBandmate 4y ago
  Good thing Musk isn't in charge of a car company
Comevius 4y ago
Many software are like that, especially those that have been around for some time. You don't see the CEO of Microsoft spewing nonsense about Windows publicly, and then punching down on an employee for correcting him. There is no need for the circus. Technical debt is a fact of business, it can be dealt with professionally. It's not a reason to treat people badly.
- david927 4y ago
  The developer, Eric, is in the wrong here. The product is especially bad. The new CEO is trying to reform it. Eric is saying "spend more," which is the refuge of bad architecture. That's not the answer. And instead of privately working together to review it, he's publicly arguing. I wouldn't accept that either.
  - Comevius 4y ago
    He called Musk's public statement out, because it was wrong. Twitter shouldn't be the place for this, but it's on Musk for creating such unprofessional atmosphere in the first place. Imagine the same thing with Microsoft. We all know Windows is held together with scotch tape, not to mention that 7 years after Windows 10 they still haven't quite manage to unify the old and the new interfaces, not that Windows 11's new interfaces are any good. What a mess. Except it's not, it's business as usual. Big software projects are messy and hard to manage, but these issues are unimportant when you get the bigger picture right. For Windows the bigger picture is that it runs all your application. This is what makes Windows a good product, the rest is details and can be fixed over time. Preferably in a professional manner.
  - fzeroracer 4y ago
    How is Musk trying to 'reform' it when since he's owned it Twitter has had worse performance than ever, degraded features due to his bull-headed decisions and is literally burning money because he's chasing advertisers? And the only one that started a public argument was Musk by making an incorrect claim and defacto blaming the engineers for something he clearly doesn't understand. He's trying to pull another Paypal and it's blowing up in his face now like it did then.
salad-fan 4y ago
Eric didn’t defend the entire architecture though. Space Karen made a specific claim (“>1000 RPCs to render the time line”) and Eric refuted just that one claim.
- david927 4y ago
  Eric essentially said, "That's not why the engine is running hot, it just needs lots of oil." That's a stupid response.
  - salad-fan 4y ago
    No he didn’t. Eric essentially said “the clients don’t make >1000 RPCs to build the time line”, a direct refutation of Space Karen’s original assertion.
    - david927 4y ago
      Yes, but he went on to say that it is truly slow and that it needs more spend, which is another way of saying, "You're wrong! It's not that! ... but I have no idea what it is," which isn't a very good response.
      - salad-fan 4y ago
        That’s a pretty uncharitable interpretation. I read it as “You’re wrong” (which is correct - Space Karen was wrong in his claim about why things were slow) “and we should figure out why, which will take time & effort (i.e. investment).”
        david927 4y ago
        You could be right. But how do you know the consultants Musk hired are wrong? How does Eric know? Maybe there was a situation found where that did, in fact, happen. Certainly with a large number of microservices, it could happen. Maybe Musk is saying "edge case" and Eric is saying "on average." Personally I feel outside consultants could have very well found problems that the internal team did not see. What everyone agrees on is that Twitter's architecture is deeply broken. Eric's been there six years, so he's probably at least a little culpable in that. If I'm Eric, and was part of a team that fucked up that badly for six years, I certainly wouldn't argue back with that much confidence.

sho_hn 4y ago

I'm coming at this from a different angle. It should be possible to have a heated disagreement, even in public, and not have it result in a same-day firing. In fact, some countries have it explicitly on the law books. All this does is create an environment where people are afraid to voice opinions and address issues, e.g. because they're afraid of shoot-the-messenger. In my line of work (I make cars), dysfunctional discourse environments like that can get people killed.

david927 4y ago
Sure, and I'm a big proponent of labor. Musk is certainly not. That said, Twitter's architecture is a Rube Goldberg contraption and Musk is clearly on a warpath to fix it, so to come out publicly and yell, "It's fine," is kind of crazy. It's not fine and you're saying you're not interested in fixing it.
- sho_hn 4y ago
  Looking at the exchange, he did confirm that it's slow on Android/iOS. He claimed it's for other reasons and gave examples. He might be wrong (I haven't looked at Twitter's architecture, commentary on the technical content of the exchange would be interesting), but I wouldn't say he said "It's fine".
  - david927 4y ago
    He said it's "net spend" and that's another way of saying, "It's fine." It's the equivalent of admitting the engine is running hot, but saying, it's not a problem with the engine, it just needs lots of oil." Frankly, I think the developer is embarrassed. There's no reason for it to be as slow as it is, other than he and the other engineers simply don't know what they're doing. I think Musk is calling it out because he wants to see who wants to fix it and who wants to defend it.
    - sho_hn 4y ago
      I think you misread "Ad spend" as "net spend" in his tweet (I can't locate any using the phrase "net spend"). He mentioned that performance work correlates well with "UAM and Ad spend", which to me reads as "it's worth working more on this problem" and is indicating the willingness you're looking for. From what I can see, he offered an opinion that the number of client requests (he gives a number of 20) and the number of internal RPCs (he gives 200) is not the primary reason the app is slow. I can't tell you whether he's right or wrong.
      - david927 4y ago
        Thanks for the correction. You're right, and I can't tell you either, so I shouldn't really argue.
        sho_hn 4y ago
        FWIW, if he actually had written "just throw more money at it to make it go faster", I would understand your more negative opinion. :-)
        david927 4y ago
        I mean how does he know it's not "thousands of RPC calls" -- he spent six years and never got it to perform, so he doesn't know. "You're wrong!" ? Based on what? If I worked for six years on something and it still didn't work and someone came in and said it's "x", I would say, "I'm excited to see that data because I really want to fix this." I think I'm more upset at his attitude than anything. In all honesty, I think what he meant was "I spent six years and failed to fix something that is fixable, and then outside consultants came in a clearly found clear problems in a couple weeks, and now I'm humiliated," but he has too big of an ego for that.
        jurassic 4y ago
        The guy's job was not specifically performance; he didn't fail at anything. Product managers drive roadmaps so even if you know there's been a performance problem for years, you might not get to invest time in fixing it. That's exactly what Eric was advocating if you read the whole thread.
viig99 4y ago
Clearly these engineers have never done customer feedback for developing nations, twitter interactions clearly provide evidence for what musk said, but some engineers seem to be in denial, imho the firing is cause worthy.
- sho_hn 4y ago
  I think that's misreading the exchange. He was challenging Musk on the fact that "1000 RPC calls" isn't what's happening and isn't the reason for the slowdown in his opinion, but did not deny that there are performance issues - and even cited other reasons it's slow. I don't see the denial you see.
  - viig99 4y ago
    Said engineer, said 1000 RPC calls are not the issue, and when questioned further on proving a tracepath / perf result casually exited the conversation, to me it looks like they are not ready to do better or listen to valid criticism, leading conversations on semantics is just creating noise instead of acting on signals. Also 1000 RPC's are a very valid performance bottleneck, 1000 * latency caused by serde+ network can compounds easily based on data locality + container network orchestration, no data was provided otherwise even when asked for.
    - fragmede 4y ago
      Except that you can open the dev console in a browser and load Twitter.com yourself and you won't find 1000's of XHR requests.
      - sho_hn 4y ago
        That's client-side requests. Musk was talking about internal RPC calls to microservices needed to ultimately respond to the client-side requests. If the internal arch is heavy on microservices and you're putting together, say, 50 tweets you're at 1000 RPC calls with just 20 per tweet. It's not inconceivable. The developer puts the real number at 200.
FormerBandmate 4y ago
Good thing Musk isn't in charge of a car company

Comevius 4y ago

Many software are like that, especially those that have been around for some time. You don't see the CEO of Microsoft spewing nonsense about Windows publicly, and then punching down on an employee for correcting him. There is no need for the circus. Technical debt is a fact of business, it can be dealt with professionally. It's not a reason to treat people badly.

david927 4y ago
The developer, Eric, is in the wrong here. The product is especially bad. The new CEO is trying to reform it. Eric is saying "spend more," which is the refuge of bad architecture. That's not the answer. And instead of privately working together to review it, he's publicly arguing. I wouldn't accept that either.
- Comevius 4y ago
  He called Musk's public statement out, because it was wrong. Twitter shouldn't be the place for this, but it's on Musk for creating such unprofessional atmosphere in the first place. Imagine the same thing with Microsoft. We all know Windows is held together with scotch tape, not to mention that 7 years after Windows 10 they still haven't quite manage to unify the old and the new interfaces, not that Windows 11's new interfaces are any good. What a mess. Except it's not, it's business as usual. Big software projects are messy and hard to manage, but these issues are unimportant when you get the bigger picture right. For Windows the bigger picture is that it runs all your application. This is what makes Windows a good product, the rest is details and can be fixed over time. Preferably in a professional manner.
- fzeroracer 4y ago
  How is Musk trying to 'reform' it when since he's owned it Twitter has had worse performance than ever, degraded features due to his bull-headed decisions and is literally burning money because he's chasing advertisers? And the only one that started a public argument was Musk by making an incorrect claim and defacto blaming the engineers for something he clearly doesn't understand. He's trying to pull another Paypal and it's blowing up in his face now like it did then.

salad-fan 4y ago

Eric didn’t defend the entire architecture though. Space Karen made a specific claim (“>1000 RPCs to render the time line”) and Eric refuted just that one claim.

david927 4y ago
Eric essentially said, "That's not why the engine is running hot, it just needs lots of oil." That's a stupid response.
- salad-fan 4y ago
  No he didn’t. Eric essentially said “the clients don’t make >1000 RPCs to build the time line”, a direct refutation of Space Karen’s original assertion.
  - david927 4y ago
    Yes, but he went on to say that it is truly slow and that it needs more spend, which is another way of saying, "You're wrong! It's not that! ... but I have no idea what it is," which isn't a very good response.
    - salad-fan 4y ago
      That’s a pretty uncharitable interpretation. I read it as “You’re wrong” (which is correct - Space Karen was wrong in his claim about why things were slow) “and we should figure out why, which will take time & effort (i.e. investment).”
      - david927 4y ago
        You could be right. But how do you know the consultants Musk hired are wrong? How does Eric know? Maybe there was a situation found where that did, in fact, happen. Certainly with a large number of microservices, it could happen. Maybe Musk is saying "edge case" and Eric is saying "on average." Personally I feel outside consultants could have very well found problems that the internal team did not see. What everyone agrees on is that Twitter's architecture is deeply broken. Eric's been there six years, so he's probably at least a little culpable in that. If I'm Eric, and was part of a team that fucked up that badly for six years, I certainly wouldn't argue back with that much confidence.