3 ms·

Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentat

by gizmo 4y ago

Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentation.

grishka 4y ago
A lot of things in a typical social media service can be segmented trivially. The hardest part is the feed because it requires querying many database servers at once, but then it can be cached and served quicker.
- wayfarer2s 4y ago
  Isn’t cache invalidation the hardest part of the feed? Once a user with millions of followers tweets, doesn’t that essentially invalidate millions of caches? That doesn’t sound like a trivial problem.
  - naasking 4y ago
    Because you can't edit tweets, the database is monotonically increasing (append-only) so scaling it isn't that hard. Cache invalidation basically just adds a new item. Deleting a tweet actually would be the harder/more expensive operation, but it's also less common.
    - grishka 4y ago
      Editing a tweet is also easy because the feeds would only store its id. You'd only need to invalidate one cache where the tweet object itself is stored.
recuter 4y ago
No, it really isn't. It is perfectly normalizable. Every twitterer is a newsletter. Most hardly ever tweet and sporadically at that. The followers are subscribers. They hardly even see most tweets they subscribe to as the whole thing is quite ephemeral. Same as never reading all your emails (very few people are inbox zero freaks). The timelines are just that, an email inbox. It is very soft "real-time" at best. Tweets older than 48 hours can just be archived to a blob store and served as a static website. Most people consume it as such, logged out and from the browser. You think brainyquote.com is hard to scale? Think of twitter.com as unbrainyquote.com. All the other froufrou built on top of it is more complicated but 2007 style twitter is dead simple. That is its (completely accidental) genius. There is no shortage of systems that are x10-100 times more "web scale".
- evanelias 4y ago
  The approach you're describing (fully materialized inbox model) simply doesn't scale. I say this as someone who worked on social media databases and scalability for a decade. Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. At minimum you need to do a hybrid approach which special-cases the more widely-followed users. This problem has been well-known for quite a long time. Yahoo's "Feeding Frenzy" whitepaper came out in 2010, but the concepts were definitely known before that; I remember hearing about hybrid activity feed designs in 2009 from colleagues who formerly worked on LiveJournal.
  - recuter 4y ago
    > Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. I knew somebody would bring that up. Currently there are fewer than 500k "verified users". Not that many people have 1 million+ followers and they don't tweet all that often. > At minimum you need to do a hybrid approach which special-cases the more widely-followed users. Great, so we are in agreement that for 99.75% of users it is all quite trivial. I'll tell you what has happened since 2010: hardware has gotten a lot faster. 1 million iops is not a big deal anymore. Keeping that in mind you should refresh your assumptions. https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-millions-of-iops-and-tens-of-gb-s-to-any-amazon-ec2-instance/ https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-mi... https://spdk.io/news/2021/05/06/nvme-80m-iops/ https://spdk.io/news/2021/05/06/nvme-80m-iops/ We built a 2U Intel Xeon server system capable of 80 MILLION 512B random read I/O operations by combining the latest 3rd Generation Xeon Scalable Processor (code-named Ice Lake) with Intel Optane SSDs. https://twitter.com/axboe/status/1554115250588471297 https://twitter.com/axboe/status/1554115250588471297 122M IOPS in 2U, with > 80% of the system idle. Easy. Just to put this into perspective, at 4K random reads, this is 144GB/sec of bandwidth from storage, at 36M IOPS. Fancy 512b random reads? You now get ~120M IOPS. You could saturate streamed network traffic on 11-12 100Gbit NICs.
    - 4y ago

by gizmo 4y ago

Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentation.

grishka 4y ago

A lot of things in a typical social media service can be segmented trivially. The hardest part is the feed because it requires querying many database servers at once, but then it can be cached and served quicker.

wayfarer2s 4y ago
Isn’t cache invalidation the hardest part of the feed? Once a user with millions of followers tweets, doesn’t that essentially invalidate millions of caches? That doesn’t sound like a trivial problem.
- naasking 4y ago
  Because you can't edit tweets, the database is monotonically increasing (append-only) so scaling it isn't that hard. Cache invalidation basically just adds a new item. Deleting a tweet actually would be the harder/more expensive operation, but it's also less common.
  - grishka 4y ago
    Editing a tweet is also easy because the feeds would only store its id. You'd only need to invalidate one cache where the tweet object itself is stored.

recuter 4y ago

No, it really isn't. It is perfectly normalizable. Every twitterer is a newsletter. Most hardly ever tweet and sporadically at that. The followers are subscribers. They hardly even see most tweets they subscribe to as the whole thing is quite ephemeral. Same as never reading all your emails (very few people are inbox zero freaks). The timelines are just that, an email inbox. It is very soft "real-time" at best. Tweets older than 48 hours can just be archived to a blob store and served as a static website. Most people consume it as such, logged out and from the browser. You think brainyquote.com is hard to scale? Think of twitter.com as unbrainyquote.com. All the other froufrou built on top of it is more complicated but 2007 style twitter is dead simple. That is its (completely accidental) genius. There is no shortage of systems that are x10-100 times more "web scale".

evanelias 4y ago
The approach you're describing (fully materialized inbox model) simply doesn't scale. I say this as someone who worked on social media databases and scalability for a decade. Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. At minimum you need to do a hybrid approach which special-cases the more widely-followed users. This problem has been well-known for quite a long time. Yahoo's "Feeding Frenzy" whitepaper came out in 2010, but the concepts were definitely known before that; I remember hearing about hybrid activity feed designs in 2009 from colleagues who formerly worked on LiveJournal.
- recuter 4y ago
  > Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. I knew somebody would bring that up. Currently there are fewer than 500k "verified users". Not that many people have 1 million+ followers and they don't tweet all that often. > At minimum you need to do a hybrid approach which special-cases the more widely-followed users. Great, so we are in agreement that for 99.75% of users it is all quite trivial. I'll tell you what has happened since 2010: hardware has gotten a lot faster. 1 million iops is not a big deal anymore. Keeping that in mind you should refresh your assumptions. https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-millions-of-iops-and-tens-of-gb-s-to-any-amazon-ec2-instance/ https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-mi... https://spdk.io/news/2021/05/06/nvme-80m-iops/ https://spdk.io/news/2021/05/06/nvme-80m-iops/ We built a 2U Intel Xeon server system capable of 80 MILLION 512B random read I/O operations by combining the latest 3rd Generation Xeon Scalable Processor (code-named Ice Lake) with Intel Optane SSDs. https://twitter.com/axboe/status/1554115250588471297 https://twitter.com/axboe/status/1554115250588471297 122M IOPS in 2U, with > 80% of the system idle. Easy. Just to put this into perspective, at 4K random reads, this is 144GB/sec of bandwidth from storage, at 36M IOPS. Fancy 512b random reads? You now get ~120M IOPS. You could saturate streamed network traffic on 11-12 100Gbit NICs.
  - 4y ago