4 ms·

I don't work at twitter, but one thing you can know for sure is that the tweet needs to be distributed to multiple machines to meet this performance requirement

I don't work at twitter, but one thing you can know for sure is that the tweet needs to be distributed to multiple machines to meet this performance requirement. A single machine probably could probably send three million trivial messages in three seconds, but if you mix in any business logic it's not gonna happen. What I'd do is have a shared highfanout queue. Once you have more than say 10k followers your tweets go to the high fanout cluster. You'd have hundreds or low thousands of machines, each of which serves a slice of consumers. When the tweet is sent, you write it to this queue. Each consumer is pulling from one of the shards. If you have a thousand workers that means each worker only needs to send a thousand messages in three seconds, which is very doable. Only about 180k twitter users have more than 20k followers. If each users tweets every 200 seconds, which seems like a high estimate, then that implies a load of about 1kwps for this system, which seems doable, especially if you have a small intermediate layer of distributors which consolidates the write load. That's just my sketch.

brianwawok 6y ago
Right. Seems very doable to verbally sketch it if you have ever seen something like a fan out. If you only worked in single Wordpress servers, I could see it a big leap.
- asdfasgasdgasdg 6y ago
  To be fair, having worked in a twitter-like domain is a valid selection criterion for any internet company planning to handle lots of traffic. It is possible that working on wordpress servers would not prepare you very well for such a job. Being unable to even articulate a basic strategy for handling a problem like this would be something I would consider a yellow flag for a potential teammate. It would make me question whether the person would be able to help me in my work.
  - alexbanks 6y ago
    If someone's resume was only wordpress sites, and they somehow made it into a technical interview for a team that managed twitter-like systems, I'd consider that a critical failure of the selection process. As an interviewer, I would try and make that experience as painless as possible for the candidate, and if not totally shocked by their abilities, have very direct conversations about the shortcomings of the selection process with whoever brought them in in the first place.
    - asdfasgasdgasdg 6y ago
      > I'd consider that a critical failure of the selection process. Me too. Unfortunately, I have no control over recruiters or the questionable candidates they sometimes choose to send in. I just have to do my best to ascertain whether the person is a good potential teammate.
      - alexbanks 6y ago
        Yes, and my point is to not take that critical failure out on the candidate.
        asdfasgasdgasdg 6y ago
        I wouldn't "take it out" on the candidate. I always treat my candidates with due respect and courtesy. I would no-hire them, though, if they were not a match for the team.
lefrenchy 6y ago
Follow up question for a little clarification. Would you write the tweet into the queue once or append a message to the queue for each follower containing the tweet id? Seems to me you’d need to do the latter right? That way you ensure you process each follower at least once, but each worker doesn’t need to be aware of others it’s just pulling messages from the queue.
- asdfasgasdgasdg 6y ago
  I don't think so. Maybe for the active users, but maybe not even then. Most twitter users are inactive and so you wouldn't want to fan that kind of data out so widely. You'd either have to pay enormous gc costs or enormous storage costs for someone like Donald Trump (0.5 GB per tweet assuming 64 bit tweet IDs). But I think I would just have a sharded in memory cache of recent tweets for the big dogs and if someone loads their feed you would do lookups into the sharded db for each subscription. You'd then cache the feed for the user so you don't have to do that lookup very often. If you were going to write, you'd write into the cache, rather than persisting anything to disk. That's my guess at least.

brianwawok 6y ago

Right. Seems very doable to verbally sketch it if you have ever seen something like a fan out. If you only worked in single Wordpress servers, I could see it a big leap.

asdfasgasdgasdg 6y ago
To be fair, having worked in a twitter-like domain is a valid selection criterion for any internet company planning to handle lots of traffic. It is possible that working on wordpress servers would not prepare you very well for such a job. Being unable to even articulate a basic strategy for handling a problem like this would be something I would consider a yellow flag for a potential teammate. It would make me question whether the person would be able to help me in my work.
- alexbanks 6y ago
  If someone's resume was only wordpress sites, and they somehow made it into a technical interview for a team that managed twitter-like systems, I'd consider that a critical failure of the selection process. As an interviewer, I would try and make that experience as painless as possible for the candidate, and if not totally shocked by their abilities, have very direct conversations about the shortcomings of the selection process with whoever brought them in in the first place.
  - asdfasgasdgasdg 6y ago
    > I'd consider that a critical failure of the selection process. Me too. Unfortunately, I have no control over recruiters or the questionable candidates they sometimes choose to send in. I just have to do my best to ascertain whether the person is a good potential teammate.
    - alexbanks 6y ago
      Yes, and my point is to not take that critical failure out on the candidate.
      - asdfasgasdgasdg 6y ago
        I wouldn't "take it out" on the candidate. I always treat my candidates with due respect and courtesy. I would no-hire them, though, if they were not a match for the team.

lefrenchy 6y ago

Follow up question for a little clarification. Would you write the tweet into the queue once or append a message to the queue for each follower containing the tweet id? Seems to me you’d need to do the latter right? That way you ensure you process each follower at least once, but each worker doesn’t need to be aware of others it’s just pulling messages from the queue.

asdfasgasdgasdg 6y ago
I don't think so. Maybe for the active users, but maybe not even then. Most twitter users are inactive and so you wouldn't want to fan that kind of data out so widely. You'd either have to pay enormous gc costs or enormous storage costs for someone like Donald Trump (0.5 GB per tweet assuming 64 bit tweet IDs). But I think I would just have a sharded in memory cache of recent tweets for the big dogs and if someone loads their feed you would do lookups into the sharded db for each subscription. You'd then cache the feed for the user so you don't have to do that lookup very often. If you were going to write, you'd write into the cache, rather than persisting anything to disk. That's my guess at least.