5 ms·

> Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would cr

> Otherwise, every time Obama or Musk tweets, you need to do one hundred million writes. That amount of write amplification is completely ludicrous and would crush any system. I knew somebody would bring that up. Currently there are fewer than 500k "verified users". Not that many people have 1 million+ followers and they don't tweet all that often. > At minimum you need to do a hybrid approach which special-cases the more widely-followed users. Great, so we are in agreement that for 99.75% of users it is all quite trivial. I'll tell you what has happened since 2010: hardware has gotten a lot faster. 1 million iops is not a big deal anymore. Keeping that in mind you should refresh your assumptions. https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-millions-of-iops-and-tens-of-gb-s-to-any-amazon-ec2-instance/ https://aws.amazon.com/blogs/storage/aws-san-in-the-cloud-mi... https://spdk.io/news/2021/05/06/nvme-80m-iops/ https://spdk.io/news/2021/05/06/nvme-80m-iops/ We built a 2U Intel Xeon server system capable of 80 MILLION 512B random read I/O operations by combining the latest 3rd Generation Xeon Scalable Processor (code-named Ice Lake) with Intel Optane SSDs. https://twitter.com/axboe/status/1554115250588471297 https://twitter.com/axboe/status/1554115250588471297 122M IOPS in 2U, with > 80% of the system idle. Easy. Just to put this into perspective, at 4K random reads, this is 144GB/sec of bandwidth from storage, at 36M IOPS. Fancy 512b random reads? You now get ~120M IOPS. You could saturate streamed network traffic on 11-12 100Gbit NICs.

evanelias 4y ago
You can't just hand-wave away those edge cases. That's not how anything works. If your potential write amplification for a single operation is anywhere remotely near a factor of 1 million, you have a serious problem and need to completely change your approach to the problem, and use different data structures and algorithms. Hardware hasn't gotten that much faster really. PCIe flash cards started to get used over a decade ago -- yes, modern storage is better than those were, but not by a huge multiple. And meanwhile max CPU frequencies today aren't much higher at all. What we have instead is more cores. And a lot more RAM per box. Faster networking, sure. But none of this lets you get away with massive write amplification from choosing an overly-naive algorithm. And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about. Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency. I'm not making "assumptions" that I need to "refresh". I've literally spent the majority of my career working on this stuff at extreme scale, both in 2010 and today, and all times in between.
- recuter 4y ago
  > And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about. The twitter firehose is usually bellow 50MB/s. 200GB a day of tweets will bankrupt no one. An 100TB Nimbus Exadrive that does 100,000+ iops costs about $30,000. 1 year of tweets. Thousands of twitter employees fired probably saves $3+ billion/year in salaries, I'm sure they have a hefty enough hardware budget. > Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency. Does it really? I don't think so. Not with tweets. For DR you can stream into a blob store like s3 in the background and have an automated process that stands up a fresh shadow cluster from it every couple of hours. Hardly costs anything with this volume of data. That is cold data, doesn't need the fancy blazing fast storage.
  - evanelias 4y ago
    > The twitter firehose is usually bellow 50MB/s That's a single copy of all tweets. Not fanning out to up to 100 million inboxes. Completely different problems. You keep citing raw hardware speeds of a single machine, yet we're talking about the feasibility of a distributed system being able to sustain random bursts of write amplification factor of 100 million across a decentralized database, with ideally exactly-once write semantics even if a failure occurs mid-way -- and that's all in addition to whatever the normal baseline write activity of all "normal" users with more reasonable follower counts. Again, completely different problems. > I don't think so. Not with tweets. So in your design, if the singular data center that maintains all users' inboxes goes offline for a long period of time, the entire product just goes down. And you think that's acceptable for a business valued in the tens of billions of dollars? You seem absolutely convinced that a massive social network can be run on a shoestring budget with tiny staff, and no amount of evidence from someone like me (who actually worked on this stuff in depth, and posts with my real name, and expertise in profile) will convince you otherwise, so I suppose I should just stop replying to you.
    - recuter 4y ago

evanelias 4y ago

You can't just hand-wave away those edge cases. That's not how anything works. If your potential write amplification for a single operation is anywhere remotely near a factor of 1 million, you have a serious problem and need to completely change your approach to the problem, and use different data structures and algorithms. Hardware hasn't gotten that much faster really. PCIe flash cards started to get used over a decade ago -- yes, modern storage is better than those were, but not by a huge multiple. And meanwhile max CPU frequencies today aren't much higher at all. What we have instead is more cores. And a lot more RAM per box. Faster networking, sure. But none of this lets you get away with massive write amplification from choosing an overly-naive algorithm. And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about. Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency. I'm not making "assumptions" that I need to "refresh". I've literally spent the majority of my career working on this stuff at extreme scale, both in 2010 and today, and all times in between.

recuter 4y ago
> And iops aside, even just the storage capacity from full inbox materialization (along with necessary indexing overhead) will bankrupt you, especially on that blazing fast storage you keep talking about. The twitter firehose is usually bellow 50MB/s. 200GB a day of tweets will bankrupt no one. An 100TB Nimbus Exadrive that does 100,000+ iops costs about $30,000. 1 year of tweets. Thousands of twitter employees fired probably saves $3+ billion/year in salaries, I'm sure they have a hefty enough hardware budget. > Keep in mind everything needs to be replicated to multiple regions / data centers for DR/HA, as well as keeping the data closer to users to lower the latency. Does it really? I don't think so. Not with tweets. For DR you can stream into a blob store like s3 in the background and have an automated process that stands up a fresh shadow cluster from it every couple of hours. Hardly costs anything with this volume of data. That is cold data, doesn't need the fancy blazing fast storage.
- evanelias 4y ago
  > The twitter firehose is usually bellow 50MB/s That's a single copy of all tweets. Not fanning out to up to 100 million inboxes. Completely different problems. You keep citing raw hardware speeds of a single machine, yet we're talking about the feasibility of a distributed system being able to sustain random bursts of write amplification factor of 100 million across a decentralized database, with ideally exactly-once write semantics even if a failure occurs mid-way -- and that's all in addition to whatever the normal baseline write activity of all "normal" users with more reasonable follower counts. Again, completely different problems. > I don't think so. Not with tweets. So in your design, if the singular data center that maintains all users' inboxes goes offline for a long period of time, the entire product just goes down. And you think that's acceptable for a business valued in the tens of billions of dollars? You seem absolutely convinced that a massive social network can be run on a shoestring budget with tiny staff, and no amount of evidence from someone like me (who actually worked on this stuff in depth, and posts with my real name, and expertise in profile) will convince you otherwise, so I suppose I should just stop replying to you.
  - recuter 4y ago