4 ms·

X Engineering Year Retrospective

63 points by sirobg 3y ago

rabf 3y ago
Changelog: https://twitter.com/cb_doge/status/1717668534078034024 https://twitter.com/cb_doge/status/1717668534078034024
outside1234 3y ago
"Shrank site traffic by 80%, yielding cost savings and cratering the company as a whole" I hope everyone there is getting paid in advance for their work...
- tymekpavel 3y ago
  I’m curious if you have a source for that? There has been lots of speculation on how much X traffic has reduced, but the best data I can find basically says that it might’ve dropped from the 2nd most visited website to the 5th most visited website in the world (according to SimilarWeb).
  - kenjackson 3y ago
    The numbers I see are declines in traffic of 15-20% with ad revenue declines of more than 50%. https://variety.com/2023/digital/news/musk-twitter-x-acquisition-one-year-user-revenue-decline-1235770297/ https://variety.com/2023/digital/news/musk-twitter-x-acquisi... I think for many sites this would be pretty horrible. I feel like the bar for X is that it still runs.
- yoav 3y ago
  They did this by shipping so many bugs (that I’ve seen) that the feeds only load 20% of the time. Saves a ton of bandwidth i bet.
tomohawk 3y ago
I've seen this in other places I've worked. Too many people and too much money to spend can be a lot worse than too few and not enough.
- VHRanger 3y ago
  Yes, twitter is doing so well currently as a business
  - add-sub-mul-div 3y ago
    It's garbage, yes, but all these companies overhired for years and it was a no-brainer that they all needed to fix that. Hard to criticize that decision in principle. Though the execution of it seemed like a clown show.
- marcinzm 3y ago
  If your main focus is growth or potential growth then inefficiencies are fine if they allow increased product velocity. A unified platform for N features is more efficient but communication overheads will cut velocity on improvements for all those N features. Of course if growth is no longer a focus that’s a different story.
  - whatasaas 3y ago
    Yea, I don’t know anything about cars but I’m pretty sure all these mechanics could work naked. Praise me for saving uniform cost.
imperialdrive 3y ago
That project must be so much fun to work on. I'd pay just to shadow it.
swimwiththebeat 3y ago
I'm inclined to believe that they consolidated their code across a lot of their microservices and simplified their architecture since that was stressed a lot from the moment Musk acquired Twitter. But we also can't really verify or disprove it. I'm a bit confused about their so-called improvements to video recommendation quality and bot detection. I've seen a lot of sentiment from people that they see more bots, hate speech, and irrelevant content on their timelines. Maybe what I'm hearing is just anecdotal evidence or stories in a bubble? The Sacramento data center migration to Portland is an entertaining story detailed here[1]. Here's the Hacker News thread on it[2]. They have a GPU supercompute cluster?? It seems like they have the capability to do training and inference with state-of-the-art algorithms at massive scales then. Why have Twitter's recommendations and ad revenue (even before the acquisition) been so poor then? [1] https://www.cnbc.com/2023/09/11/elon-musk-moved-twitter-servers-himself-in-the-night-new-biography-details-his-maniacal-sense-of-urgency.html https://www.cnbc.com/2023/09/11/elon-musk-moved-twitter-serv... [2] https://news.ycombinator.com/item?id=37470110 https://news.ycombinator.com/item?id=37470110
- marcinzm 3y ago
  >They have a GPU supercompute cluster?? It seems like they have the capability to do training and inference with state-of-the-art algorithms at massive scales then. Why have Twitter's recommendations and ad revenue (even before the acquisition) been so poor then? In general, assuming you are in some sort of decent state better ML doesn't make your revenue 50% better. It makes it 5% better per year.
- JohnFen 3y ago
  > I'm a bit confused about their so-called improvements to video recommendation quality and bot detection. Let's not dismiss the possibility that they're just lying. I don't think they're trustworthy enough to be given the benefit of the doubt.
  - DonHopkins 3y ago
    They're just running most of the bots in-house on the GPU supercompute cluster now, and billing Putin for it directly, now that the middleman Yevgeny Prigozhin is out of the picture.
insanitybit 3y ago
> Among the changes we made was a shift of all media/blob artifacts out of the cloud, which reduced our overall cloud data storage size by 60%, and separately, we succeeded in reducing cloud data processing costs by 75%. Huh. I feel like that's the one place to not leave cloud. 90% of why I want to use AWS is S3.
- Freedom2 3y ago
  That's why Musk is forward thinking, leading the charge for us techies to follow.
  - monkaiju 3y ago
    Lol, 'forward thinking' by moving one of the most expensive cloud costs internal in an app that clearly doesn't need that storage near their other cloud assets and operates at a scale few other services do? Woooow, he's sooooo smart. Nobody else has thought of this
- smaps 3y ago
  Cloud storage is expensive, I’m willing to bet that’s a lot of money saved at their scale.
  - insanitybit 3y ago
    Maintaining a massive scale infra with the same availability and durability as S3 is really expensive. It's possible but hard.
    - kalleboo 3y ago
      But do you need S3 durability for meme gifs that will be seen once and then just waste disk space for decades?
      - johannes1234321 3y ago
        At least temporarily old images weren't available and some people cared: https://www.forbes.com/sites/mattnovak/2023/08/19/twitter-deletes-all-user-photos-and-links-from-2011-2014/ https://www.forbes.com/sites/mattnovak/2023/08/19/twitter-de...
cratermoon 3y ago
"Shutdown the Sacramento data center and re-provisioned the 5,200 racks and 148,000 servers, which generated more than $100M in annual savings." Yeah, about that.. https://www.cnbc.com/2023/09/11/elon-musk-moved-twitter-servers-himself-in-the-night-new-biography-details-his-maniacal-sense-of-urgency.html https://www.cnbc.com/2023/09/11/elon-musk-moved-twitter-serv... "Musk turned to his security guard and asked to borrow his pocket knife. Using it, he was able to lift one of the air vents in the floor, which allowed him to pry open the floor panels. He then crawled under the server floor himself, used the knife to jimmy open an electrical cabinet, pulled the server plugs, and waited to see what happened. Nothing exploded. The server was ready to be moved."
- jprd 3y ago
  lol, sure. Dude is so capable, he is the RL MacGyver. Also, people still think he's Iron Man - even though Iron Man was semi-rational.
- quantified 3y ago
  This remains a fascinating, Howard Roark-like tale. I mean, it worked. I don't use Twixter at all. But it seems like he got rid of most of the engineering org without getting rid of most of the technical aspects of Twitter. It's not super stable, but it's not a smoldering wreck. If he hadn't scared off the advertisers with his personality and social behaviors, it might be financially in a much better place. The human side of content moderation was definitely going to cost more, I am steering away from the content impact on Twixter financials.
  - cratermoon 3y ago
    > it worked Yes, and it would probably work much of the time. The times when it doesn't work and you have to write off equipment worth more than the cost of a professional move, which shifts liability as well, is why more risk-averse companies don't do it.
  - lebean 3y ago
    Roark dynamited his own creation because he had no other recourse and was prepared to face the consequences. Musk dynamited something he bought out of foolhardiness and is expending his energy avoiding consequences. Similar in some regard, but different.
csteubs 3y ago
I hope their Trust & Security team has a roadmap that prioritizes topic and thread-jacking. The advent of GPT and their recent changes to creator compensation has created a massive drive for engagement by any means; the results are fairly predictable. Clicking on any trending topic usually surfaces completely unrelated videos with that topic's hashtags (and all of the other trending topic hashtags) appended to the tweet. Then you have the "content creators" who use GPT to summarize or add details to a post from a larger content aggregator in hopes of bandwagoning engagement. I see a lot of this type of behavior from popular History-focused accounts and the mega-accounts that post engagement bait content (think canned, "desert island"-type questions and polls). It's less malicious but certainly reinforces cynicism towards the state of Twitter/X and the broader social web.
deleted 3y ago
[deleted]

63 points by sirobg 3y ago

rabf 3y ago

Changelog: https://twitter.com/cb_doge/status/1717668534078034024 https://twitter.com/cb_doge/status/1717668534078034024

outside1234 3y ago

"Shrank site traffic by 80%, yielding cost savings and cratering the company as a whole" I hope everyone there is getting paid in advance for their work...

tymekpavel 3y ago
I’m curious if you have a source for that? There has been lots of speculation on how much X traffic has reduced, but the best data I can find basically says that it might’ve dropped from the 2nd most visited website to the 5th most visited website in the world (according to SimilarWeb).
- kenjackson 3y ago
  The numbers I see are declines in traffic of 15-20% with ad revenue declines of more than 50%. https://variety.com/2023/digital/news/musk-twitter-x-acquisition-one-year-user-revenue-decline-1235770297/ https://variety.com/2023/digital/news/musk-twitter-x-acquisi... I think for many sites this would be pretty horrible. I feel like the bar for X is that it still runs.
yoav 3y ago
They did this by shipping so many bugs (that I’ve seen) that the feeds only load 20% of the time. Saves a ton of bandwidth i bet.

tomohawk 3y ago

I've seen this in other places I've worked. Too many people and too much money to spend can be a lot worse than too few and not enough.

VHRanger 3y ago
Yes, twitter is doing so well currently as a business
- add-sub-mul-div 3y ago
  It's garbage, yes, but all these companies overhired for years and it was a no-brainer that they all needed to fix that. Hard to criticize that decision in principle. Though the execution of it seemed like a clown show.
marcinzm 3y ago
If your main focus is growth or potential growth then inefficiencies are fine if they allow increased product velocity. A unified platform for N features is more efficient but communication overheads will cut velocity on improvements for all those N features. Of course if growth is no longer a focus that’s a different story.
- whatasaas 3y ago
  Yea, I don’t know anything about cars but I’m pretty sure all these mechanics could work naked. Praise me for saving uniform cost.

imperialdrive 3y ago

That project must be so much fun to work on. I'd pay just to shadow it.

swimwiththebeat 3y ago

I'm inclined to believe that they consolidated their code across a lot of their microservices and simplified their architecture since that was stressed a lot from the moment Musk acquired Twitter. But we also can't really verify or disprove it. I'm a bit confused about their so-called improvements to video recommendation quality and bot detection. I've seen a lot of sentiment from people that they see more bots, hate speech, and irrelevant content on their timelines. Maybe what I'm hearing is just anecdotal evidence or stories in a bubble? The Sacramento data center migration to Portland is an entertaining story detailed here[1]. Here's the Hacker News thread on it[2]. They have a GPU supercompute cluster?? It seems like they have the capability to do training and inference with state-of-the-art algorithms at massive scales then. Why have Twitter's recommendations and ad revenue (even before the acquisition) been so poor then? [1] https://www.cnbc.com/2023/09/11/elon-musk-moved-twitter-servers-himself-in-the-night-new-biography-details-his-maniacal-sense-of-urgency.html https://www.cnbc.com/2023/09/11/elon-musk-moved-twitter-serv... [2] https://news.ycombinator.com/item?id=37470110 https://news.ycombinator.com/item?id=37470110

marcinzm 3y ago
>They have a GPU supercompute cluster?? It seems like they have the capability to do training and inference with state-of-the-art algorithms at massive scales then. Why have Twitter's recommendations and ad revenue (even before the acquisition) been so poor then? In general, assuming you are in some sort of decent state better ML doesn't make your revenue 50% better. It makes it 5% better per year.
JohnFen 3y ago
> I'm a bit confused about their so-called improvements to video recommendation quality and bot detection. Let's not dismiss the possibility that they're just lying. I don't think they're trustworthy enough to be given the benefit of the doubt.
- DonHopkins 3y ago
  They're just running most of the bots in-house on the GPU supercompute cluster now, and billing Putin for it directly, now that the middleman Yevgeny Prigozhin is out of the picture.

insanitybit 3y ago

> Among the changes we made was a shift of all media/blob artifacts out of the cloud, which reduced our overall cloud data storage size by 60%, and separately, we succeeded in reducing cloud data processing costs by 75%. Huh. I feel like that's the one place to not leave cloud. 90% of why I want to use AWS is S3.

Freedom2 3y ago
That's why Musk is forward thinking, leading the charge for us techies to follow.
- monkaiju 3y ago
  Lol, 'forward thinking' by moving one of the most expensive cloud costs internal in an app that clearly doesn't need that storage near their other cloud assets and operates at a scale few other services do? Woooow, he's sooooo smart. Nobody else has thought of this
smaps 3y ago
Cloud storage is expensive, I’m willing to bet that’s a lot of money saved at their scale.
- insanitybit 3y ago
  Maintaining a massive scale infra with the same availability and durability as S3 is really expensive. It's possible but hard.
  - kalleboo 3y ago
    But do you need S3 durability for meme gifs that will be seen once and then just waste disk space for decades?
    - johannes1234321 3y ago
      At least temporarily old images weren't available and some people cared: https://www.forbes.com/sites/mattnovak/2023/08/19/twitter-deletes-all-user-photos-and-links-from-2011-2014/ https://www.forbes.com/sites/mattnovak/2023/08/19/twitter-de...

cratermoon 3y ago

"Shutdown the Sacramento data center and re-provisioned the 5,200 racks and 148,000 servers, which generated more than $100M in annual savings." Yeah, about that.. https://www.cnbc.com/2023/09/11/elon-musk-moved-twitter-servers-himself-in-the-night-new-biography-details-his-maniacal-sense-of-urgency.html https://www.cnbc.com/2023/09/11/elon-musk-moved-twitter-serv... "Musk turned to his security guard and asked to borrow his pocket knife. Using it, he was able to lift one of the air vents in the floor, which allowed him to pry open the floor panels. He then crawled under the server floor himself, used the knife to jimmy open an electrical cabinet, pulled the server plugs, and waited to see what happened. Nothing exploded. The server was ready to be moved."

jprd 3y ago
lol, sure. Dude is so capable, he is the RL MacGyver. Also, people still think he's Iron Man - even though Iron Man was semi-rational.
quantified 3y ago
This remains a fascinating, Howard Roark-like tale. I mean, it worked. I don't use Twixter at all. But it seems like he got rid of most of the engineering org without getting rid of most of the technical aspects of Twitter. It's not super stable, but it's not a smoldering wreck. If he hadn't scared off the advertisers with his personality and social behaviors, it might be financially in a much better place. The human side of content moderation was definitely going to cost more, I am steering away from the content impact on Twixter financials.
- cratermoon 3y ago
  > it worked Yes, and it would probably work much of the time. The times when it doesn't work and you have to write off equipment worth more than the cost of a professional move, which shifts liability as well, is why more risk-averse companies don't do it.
- lebean 3y ago
  Roark dynamited his own creation because he had no other recourse and was prepared to face the consequences. Musk dynamited something he bought out of foolhardiness and is expending his energy avoiding consequences. Similar in some regard, but different.

csteubs 3y ago

I hope their Trust & Security team has a roadmap that prioritizes topic and thread-jacking. The advent of GPT and their recent changes to creator compensation has created a massive drive for engagement by any means; the results are fairly predictable. Clicking on any trending topic usually surfaces completely unrelated videos with that topic's hashtags (and all of the other trending topic hashtags) appended to the tweet. Then you have the "content creators" who use GPT to summarize or add details to a post from a larger content aggregator in hopes of bandwagoning engagement. I see a lot of this type of behavior from popular History-focused accounts and the mega-accounts that post engagement bait content (think canned, "desert island"-type questions and polls). It's less malicious but certainly reinforces cynicism towards the state of Twitter/X and the broader social web.

deleted 3y ago

[deleted]