4 ms·

For reference, based on quick Googling, Twitter publishes around 10 000 tweets per second on average.

• 83457 4y ago
  On average I guess that makes sense. The peaks must be ridiculously high though.
  • woevdbz 4y ago
    I'd assume it's not so much the peaks that are a challenge - most of these 10k tweets/second aren't critical to serve to anyone fast, and that scales horizontally- it's the hot spots, that one tweet thread in the spotlight right now that everybody wants to read and jump on - that doesn't scale by just adding more servers
• ndriscoll 4y ago
  i.e. my laptop could handle the write load. A retired nerd with a real server they put in a data center for bandwidth could easily run that level of traffic on 2022 hardware. For reference my work laptop (8 logical cores, so 4+HT? 32GB RAM) can handle 100k rows/second sustained inserts into postgres 14 with some batch jobs I'm working on. You can buffer http requests into batches and easily handle way more than 10k/s on a server while still providing synchronous semantics and reasonable latency to the client (e.g. flush batches every 10-100 ms). I doubt Mastodon is designed for that kind of scalability, but most techies could probably afford to run Twitter as a hobby if they knew what they're doing and they weren't trying to do all the analytics and advertising stuff to monetize it/just wanted to provide the service.
  • brazzy 4y ago
    Yeah, you have not the faintest clue what you're talking about.
  • 0x457 4y ago
    I think you don't understand that 10,000 tweets isn't 10,000 inserts. It's not enough to just write tweet, you need to fan-out writes to every user's feed. It's also not batched into neat transactions like your batch jobs. You have no idea what are you talking about.
    • ndriscoll 4y ago
      You don't need to fan out writes to feeds. Users subscribe to other users, not tweets. You can attempt to send out the notification to subscribed users, and if it fails, that's fine. You don't need to record notification status. Have a worker that records (in the db) the last tweet id it's processed, and just regularly joins N new tweets to authors to subscribers and attempts to send. You can play with how that query works to limit total work done in a chunk rather than N new tweets (in case of someone with many subscribers), but the idea is straightforward. You can easily batch writes into neat transactions: make a queue, have your POST handler write (row, callback) onto the queue and await the callback. Have the queue reader grab a chunk, push it to the db in batch, and execute all of the callbacks on commit. The callbacks return a 200 to the client, or 500 if the commit failed. This can all happen fast enough to be done in "real time" (however fast you want your queue worker to flush batches). You can do all of this in a couple dozen lines of code with something like Scala/ZIO. Computationally, it is totally doable. The biggest constraint is the cost of storage.
      • 0x457 4y ago
        I've worked in a social network with feeds. Feeds are hard. Both fan-out on write vs fan-in on read both have upsides and downsides. Either way, it's "cheap" problem to solve in a couple of dozen lines of code. When I said that you can't batch, I meant that each tweet will be at least one transaction. Async write with batching like you suggested will have a horrible user-experience, also your client often is a mobile device or browser - how do you deliver a callback from server there? Nah, how many inserts your laptop can make and how many tweets created in a unit of time are two irrelevant metrics.
        • ndriscoll 4y ago