4 ms·

No, it really isn't. It is perfectly normalizable. Every twitterer is a newsletter. Most hardly ever tweet and sporadically at that. The followers are subscri

No, it really isn't. It is perfectly normalizable. Every twitterer is a newsletter. Most hardly ever tweet and sporadically at that. The followers are subscribers. They hardly even see most tweets they subscribe to as the whole thing is quite ephemeral. Same as never reading all your emails (very few people are inbox zero freaks). The timelines are just that, an email inbox. It is very soft "real-time" at best. Tweets older than 48 hours can just be archived to a blob store and served as a static website. Most people consume it as such, logged out and from the browser. You think brainyquote.com is hard to scale? Think of twitter.com as unbrainyquote.com. All the other froufrou built on top of it is more complicated but 2007 style twitter is dead simple. That is its (completely accidental) genius. There is no shortage of systems that are x10-100 times more "web scale".

evanelias 4y ago
The approach you're describing (fully materialized inbox model) simply doesn't scale. I say this as someone who worked on social media databases and scalability for a decade. Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. At minimum you need to do a hybrid approach which special-cases the more widely-followed users. This problem has been well-known for quite a long time. Yahoo's "Feeding Frenzy" whitepaper came out in 2010, but the concepts were definitely known before that; I remember hearing about hybrid activity feed designs in 2009 from colleagues who formerly worked on LiveJournal.
- recuter 4y ago
  > Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. I knew somebody would bring that up. Currently there are fewer than 500k "verified users". Not that many people have 1 million+ followers and they don't tweet all that often. > At minimum you need to do a hybrid approach which special-cases the more widely-followed users. Great, so we are in agreement that for 99.75% of users it is all quite trivial. I'll tell you what has happened since 2010: hardware has gotten a lot faster. 1 million iops is not a big deal anymore. Keeping that in mind you should refresh your assumptions. https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-millions-of-iops-and-tens-of-gb-s-to-any-amazon-ec2-instance/ https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-mi... https://spdk.io/news/2021/05/06/nvme-80m-iops/ https://spdk.io/news/2021/05/06/nvme-80m-iops/ We built a 2U Intel Xeon server system capable of 80 MILLION 512B random read I/O operations by combining the latest 3rd Generation Xeon Scalable Processor (code-named Ice Lake) with Intel Optane SSDs. https://twitter.com/axboe/status/1554115250588471297 https://twitter.com/axboe/status/1554115250588471297 122M IOPS in 2U, with > 80% of the system idle. Easy. Just to put this into perspective, at 4K random reads, this is 144GB/sec of bandwidth from storage, at 36M IOPS. Fancy 512b random reads? You now get ~120M IOPS. You could saturate streamed network traffic on 11-12 100Gbit NICs.
  - evanelias 4y ago
    You can't just hand-wave away those edge cases. That's not how anything works. If your potential write amplification for a single operation is anywhere remotely near a factor of 1 million, you have a serious problem and need to completely change your approach to the problem, and use different data structures and algorithms. Hardware hasn't gotten that much faster really. PCIe flash cards started to get used over a decade ago -- yes, modern storage is better than those were, but not by a huge multiple. And meanwhile max CPU frequencies today aren't much higher at all. What we have instead is more cores. And a lot more RAM per box. Faster networking, sure. But none of this lets you get away with massive write amplification from choosing an overly-naive algorithm. And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about. Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency. I'm not making "assumptions" that I need to "refresh". I've literally spent the majority of my career working on this stuff at extreme scale, both in 2010 and today, and all times in between.
    - recuter 4y ago
      > And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about. The twitter firehose is usually bellow 50MB/s. 200GB a day of tweets will bankrupt no one. An 100TB Nimbus Exadrive that does 100,000+ iops costs about $30,000. 1 year of tweets. Thousands of twitter employees fired probably saves $3+ billion/year in salaries, I'm sure they have a hefty enough hardware budget. > Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency. Does it really? I don't think so. Not with tweets. For DR you can stream into a blob store like s3 in the background and have an automated process that stands up a fresh shadow cluster from it every couple of hours. Hardly costs anything with this volume of data. That is cold data, doesn't need the fancy blazing fast storage.
      - 4y ago

evanelias 4y ago

The approach you're describing (fully materialized inbox model) simply doesn't scale. I say this as someone who worked on social media databases and scalability for a decade. Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. At minimum you need to do a hybrid approach which special-cases the more widely-followed users. This problem has been well-known for quite a long time. Yahoo's "Feeding Frenzy" whitepaper came out in 2010, but the concepts were definitely known before that; I remember hearing about hybrid activity feed designs in 2009 from colleagues who formerly worked on LiveJournal.

recuter 4y ago
> Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. I knew somebody would bring that up. Currently there are fewer than 500k "verified users". Not that many people have 1 million+ followers and they don't tweet all that often. > At minimum you need to do a hybrid approach which special-cases the more widely-followed users. Great, so we are in agreement that for 99.75% of users it is all quite trivial. I'll tell you what has happened since 2010: hardware has gotten a lot faster. 1 million iops is not a big deal anymore. Keeping that in mind you should refresh your assumptions. https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-millions-of-iops-and-tens-of-gb-s-to-any-amazon-ec2-instance/ https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-mi... https://spdk.io/news/2021/05/06/nvme-80m-iops/ https://spdk.io/news/2021/05/06/nvme-80m-iops/ We built a 2U Intel Xeon server system capable of 80 MILLION 512B random read I/O operations by combining the latest 3rd Generation Xeon Scalable Processor (code-named Ice Lake) with Intel Optane SSDs. https://twitter.com/axboe/status/1554115250588471297 https://twitter.com/axboe/status/1554115250588471297 122M IOPS in 2U, with > 80% of the system idle. Easy. Just to put this into perspective, at 4K random reads, this is 144GB/sec of bandwidth from storage, at 36M IOPS. Fancy 512b random reads? You now get ~120M IOPS. You could saturate streamed network traffic on 11-12 100Gbit NICs.
- evanelias 4y ago
  You can't just hand-wave away those edge cases. That's not how anything works. If your potential write amplification for a single operation is anywhere remotely near a factor of 1 million, you have a serious problem and need to completely change your approach to the problem, and use different data structures and algorithms. Hardware hasn't gotten that much faster really. PCIe flash cards started to get used over a decade ago -- yes, modern storage is better than those were, but not by a huge multiple. And meanwhile max CPU frequencies today aren't much higher at all. What we have instead is more cores. And a lot more RAM per box. Faster networking, sure. But none of this lets you get away with massive write amplification from choosing an overly-naive algorithm. And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about. Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency. I'm not making "assumptions" that I need to "refresh". I've literally spent the majority of my career working on this stuff at extreme scale, both in 2010 and today, and all times in between.
  - recuter 4y ago
    > And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about. The twitter firehose is usually bellow 50MB/s. 200GB a day of tweets will bankrupt no one. An 100TB Nimbus Exadrive that does 100,000+ iops costs about $30,000. 1 year of tweets. Thousands of twitter employees fired probably saves $3+ billion/year in salaries, I'm sure they have a hefty enough hardware budget. > Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency. Does it really? I don't think so. Not with tweets. For DR you can stream into a blob store like s3 in the background and have an automated process that stands up a fresh shadow cluster from it every couple of hours. Hardly costs anything with this volume of data. That is cold data, doesn't need the fancy blazing fast storage.
    - 4y ago