3 ms·

No matter if it was or not and for better or worse: If Twitter survives this without any major harm it will have profound consequences for the whole software i

No matter if it was or not and for better or worse: If Twitter survives this without any major harm it will have profound consequences for the whole software industry.

ravloony 4y ago
This is exactly why this thread is full of slightly insecure comments making vague predictions. I'd suggest to most of them to get off HN and back to work, now is the time to make yourself useful!
pas 4y ago
That's tiny bit of an exaggeration, maaaybe. Maybe completely prophetic, though! That said, Instagram was run by just a dozen people back then, while it had hundreds of millions of users, right? So it's not a new data point. Storing, retrieving, indexing, managing 280 char blobs (with links, threads, embeds) is not exactly the most hardcore of a problem domains. Microblogs are the typical tutorial topic, and twitter's only "innovation" was to increase the text limit.
- gizmo 4y ago
  Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentation.
  - grishka 4y ago
    A lot of things in a typical social media service can be segmented trivially. The hardest part is the feed because it requires querying many database servers at once, but then it can be cached and served quicker.
    - wayfarer2s 4y ago
      Isn’t cache invalidation the hardest part of the feed? Once a user with millions of followers tweets, doesn’t that essentially invalidate millions of caches? That doesn’t sound like a trivial problem.
      - naasking 4y ago
        Because you can't edit tweets, the database is monotonically increasing (append-only) so scaling it isn't that hard. Cache invalidation basically just adds a new item. Deleting a tweet actually would be the harder/more expensive operation, but it's also less common.
        grishka 4y ago
        Editing a tweet is also easy because the feeds would only store its id. You'd only need to invalidate one cache where the tweet object itself is stored.
  - recuter 4y ago
    No, it really isn't. It is perfectly normalizable. Every twitterer is a newsletter. Most hardly ever tweet and sporadically at that. The followers are subscribers. They hardly even see most tweets they subscribe to as the whole thing is quite ephemeral. Same as never reading all your emails (very few people are inbox zero freaks). The timelines are just that, an email inbox. It is very soft "real-time" at best. Tweets older than 48 hours can just be archived to a blob store and served as a static website. Most people consume it as such, logged out and from the browser. You think brainyquote.com is hard to scale? Think of twitter.com as unbrainyquote.com. All the other froufrou built on top of it is more complicated but 2007 style twitter is dead simple. That is its (completely accidental) genius. There is no shortage of systems that are x10-100 times more "web scale".
    - evanelias 4y ago
      The approach you're describing (fully materialized inbox model) simply doesn't scale. I say this as someone who worked on social media databases and scalability for a decade. Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. At minimum you need to do a hybrid approach which special-cases the more widely-followed users. This problem has been well-known for quite a long time. Yahoo's "Feeding Frenzy" whitepaper came out in 2010, but the concepts were definitely known before that; I remember hearing about hybrid activity feed designs in 2009 from colleagues who formerly worked on LiveJournal.
      - recuter 4y ago
        > Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. I knew somebody would bring that up. Currently there are fewer than 500k "verified users". Not that many people have 1 million+ followers and they don't tweet all that often. > At minimum you need to do a hybrid approach which special-cases the more widely-followed users. Great, so we are in agreement that for 99.75% of users it is all quite trivial. I'll tell you what has happened since 2010: hardware has gotten a lot faster. 1 million iops is not a big deal anymore. Keeping that in mind you should refresh your assumptions. https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-millions-of-iops-and-tens-of-gb-s-to-any-amazon-ec2-instance/ https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-mi... https://spdk.io/news/2021/05/06/nvme-80m-iops/ https://spdk.io/news/2021/05/06/nvme-80m-iops/ We built a 2U Intel Xeon server system capable of 80 MILLION 512B random read I/O operations by combining the latest 3rd Generation Xeon Scalable Processor (code-named Ice Lake) with Intel Optane SSDs. https://twitter.com/axboe/status/1554115250588471297 https://twitter.com/axboe/status/1554115250588471297 122M IOPS in 2U, with > 80% of the system idle. Easy. Just to put this into perspective, at 4K random reads, this is 144GB/sec of bandwidth from storage, at 36M IOPS. Fancy 512b random reads? You now get ~120M IOPS. You could saturate streamed network traffic on 11-12 100Gbit NICs.
moduspol 4y ago
And it's almost inevitable. That's why the overwhelming "Elon is so dumb" narrative seems so odd to me. "Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users. Even if Elon literally has to re-hire half the roles he fired for, or Twitter is down for a few days or a week, or they struggle to get advertisers for a little while, that's nothing for the long-term. In six months, the chances that it'll look like these firings were a bad idea are minimal. And likewise with the "the only realistic way to moderate an online platform is the way Twitter was doing it" narrative. In six months, all it takes is the ship to still be floating without the old moderation to prove it out, and that's by far the most likely outcome.
- microtherion 4y ago
  > "Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users. With the takeover, Twitter has been saddled with massive new debt and would have to become significantly more profitable to service it. Instead, they are losing ad revenue, and any savings from reduced head count will not manifest four at least 4 months because of the severance payments for the laid off workers. Musk could of course keep the company going as a hobby for a few more years, but he strikes me as more likely to cut his losses early — preferably by finding somebody to take the problem off his hand, but, if necessary, by bankruptcy. > Even if Elon literally has to re-hire half the roles he fired for I have this nagging suspicion that the kind of people who line up for an opportunity to work without equity, lousy job security, and send weekly code printouts to the CEO are not necessarily the highest quality hires in the field. > or Twitter is down for a few days or a week The Twitter user base has quite a bit of inertia keeping them from moving to competing platforms — lots of people I follow have made backup plans on Mastodon, but not many have moved their focus there. But Twitter becoming literally impossible to use would probably be the one thing precipitating such a process. > they struggle to get advertisers for a little while And after that? Are the major brands going to decide that they like advertising on a Gab-lite Twitter after all? Or is Twitter going to strike it rich with ads for dietary supplements, gold coins, non-perishable meals, and tactical pants? > In six months, the chances that it'll look like these firings were a bad idea are minimal. The targeted firings were bad enough (To paraphrase someone else "Imagine ranking engineers by # of lines of code added, and then keeping the ones with the HIGHEST number"), but the voluntary departures are likely to be even more consequential. After the way the workforce was treated, what are the odds that the people left are the "best", rather than the ones with a lack of alternatives (e.g. for visa reasons), or Musk fans who may or may not be qualified?
hn2017 4y ago
It will be destructive to society in the amount of toxic speech. Lay off all the content moderators, ok buddy.. https://www.theguardian.com/technology/2022/nov/20/twitter-fails-to-delete-99-of-racist-tweets-aimed-at-footballers-in-run-up-to-world-cup https://www.theguardian.com/technology/2022/nov/20/twitter-f...

No matter if it was or not and for better or worse: If Twitter survives this without any major harm it will have profound consequences for the whole software industry.

ravloony 4y ago

This is exactly why this thread is full of slightly insecure comments making vague predictions. I'd suggest to most of them to get off HN and back to work, now is the time to make yourself useful!

pas 4y ago

That's tiny bit of an exaggeration, maaaybe. Maybe completely prophetic, though! That said, Instagram was run by just a dozen people back then, while it had hundreds of millions of users, right? So it's not a new data point. Storing, retrieving, indexing, managing 280 char blobs (with links, threads, embeds) is not exactly the most hardcore of a problem domains. Microblogs are the typical tutorial topic, and twitter's only "innovation" was to increase the text limit.

gizmo 4y ago
Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentation.
- grishka 4y ago
  A lot of things in a typical social media service can be segmented trivially. The hardest part is the feed because it requires querying many database servers at once, but then it can be cached and served quicker.
  - wayfarer2s 4y ago
    Isn’t cache invalidation the hardest part of the feed? Once a user with millions of followers tweets, doesn’t that essentially invalidate millions of caches? That doesn’t sound like a trivial problem.
    - naasking 4y ago
      Because you can't edit tweets, the database is monotonically increasing (append-only) so scaling it isn't that hard. Cache invalidation basically just adds a new item. Deleting a tweet actually would be the harder/more expensive operation, but it's also less common.
      - grishka 4y ago
        Editing a tweet is also easy because the feeds would only store its id. You'd only need to invalidate one cache where the tweet object itself is stored.
- recuter 4y ago
  No, it really isn't. It is perfectly normalizable. Every twitterer is a newsletter. Most hardly ever tweet and sporadically at that. The followers are subscribers. They hardly even see most tweets they subscribe to as the whole thing is quite ephemeral. Same as never reading all your emails (very few people are inbox zero freaks). The timelines are just that, an email inbox. It is very soft "real-time" at best. Tweets older than 48 hours can just be archived to a blob store and served as a static website. Most people consume it as such, logged out and from the browser. You think brainyquote.com is hard to scale? Think of twitter.com as unbrainyquote.com. All the other froufrou built on top of it is more complicated but 2007 style twitter is dead simple. That is its (completely accidental) genius. There is no shortage of systems that are x10-100 times more "web scale".
  - evanelias 4y ago
    The approach you're describing (fully materialized inbox model) simply doesn't scale. I say this as someone who worked on social media databases and scalability for a decade. Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. At minimum you need to do a hybrid approach which special-cases the more widely-followed users. This problem has been well-known for quite a long time. Yahoo's "Feeding Frenzy" whitepaper came out in 2010, but the concepts were definitely known before that; I remember hearing about hybrid activity feed designs in 2009 from colleagues who formerly worked on LiveJournal.
    - recuter 4y ago
      > Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. I knew somebody would bring that up. Currently there are fewer than 500k "verified users". Not that many people have 1 million+ followers and they don't tweet all that often. > At minimum you need to do a hybrid approach which special-cases the more widely-followed users. Great, so we are in agreement that for 99.75% of users it is all quite trivial. I'll tell you what has happened since 2010: hardware has gotten a lot faster. 1 million iops is not a big deal anymore. Keeping that in mind you should refresh your assumptions. https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-millions-of-iops-and-tens-of-gb-s-to-any-amazon-ec2-instance/ https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-mi... https://spdk.io/news/2021/05/06/nvme-80m-iops/ https://spdk.io/news/2021/05/06/nvme-80m-iops/ We built a 2U Intel Xeon server system capable of 80 MILLION 512B random read I/O operations by combining the latest 3rd Generation Xeon Scalable Processor (code-named Ice Lake) with Intel Optane SSDs. https://twitter.com/axboe/status/1554115250588471297 https://twitter.com/axboe/status/1554115250588471297 122M IOPS in 2U, with > 80% of the system idle. Easy. Just to put this into perspective, at 4K random reads, this is 144GB/sec of bandwidth from storage, at 36M IOPS. Fancy 512b random reads? You now get ~120M IOPS. You could saturate streamed network traffic on 11-12 100Gbit NICs.

moduspol 4y ago

And it's almost inevitable. That's why the overwhelming "Elon is so dumb" narrative seems so odd to me. "Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users. Even if Elon literally has to re-hire half the roles he fired for, or Twitter is down for a few days or a week, or they struggle to get advertisers for a little while, that's nothing for the long-term. In six months, the chances that it'll look like these firings were a bad idea are minimal. And likewise with the "the only realistic way to moderate an online platform is the way Twitter was doing it" narrative. In six months, all it takes is the ship to still be floating without the old moderation to prove it out, and that's by far the most likely outcome.

microtherion 4y ago
> "Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users. With the takeover, Twitter has been saddled with massive new debt and would have to become significantly more profitable to service it. Instead, they are losing ad revenue, and any savings from reduced head count will not manifest four at least 4 months because of the severance payments for the laid off workers. Musk could of course keep the company going as a hobby for a few more years, but he strikes me as more likely to cut his losses early — preferably by finding somebody to take the problem off his hand, but, if necessary, by bankruptcy. > Even if Elon literally has to re-hire half the roles he fired for I have this nagging suspicion that the kind of people who line up for an opportunity to work without equity, lousy job security, and send weekly code printouts to the CEO are not necessarily the highest quality hires in the field. > or Twitter is down for a few days or a week The Twitter user base has quite a bit of inertia keeping them from moving to competing platforms — lots of people I follow have made backup plans on Mastodon, but not many have moved their focus there. But Twitter becoming literally impossible to use would probably be the one thing precipitating such a process. > they struggle to get advertisers for a little while And after that? Are the major brands going to decide that they like advertising on a Gab-lite Twitter after all? Or is Twitter going to strike it rich with ads for dietary supplements, gold coins, non-perishable meals, and tactical pants? > In six months, the chances that it'll look like these firings were a bad idea are minimal. The targeted firings were bad enough (To paraphrase someone else "Imagine ranking engineers by # of lines of code added, and then keeping the ones with the HIGHEST number"), but the voluntary departures are likely to be even more consequential. After the way the workforce was treated, what are the odds that the people left are the "best", rather than the ones with a lack of alternatives (e.g. for visa reasons), or Musk fans who may or may not be qualified?

hn2017 4y ago

It will be destructive to society in the amount of toxic speech. Lay off all the content moderators, ok buddy.. https://www.theguardian.com/technology/2022/nov/20/twitter-fails-to-delete-99-of-racist-tweets-aimed-at-footballers-in-run-up-to-world-cup https://www.theguardian.com/technology/2022/nov/20/twitter-f...