9 ms·

Fanout is soft realtime, but yeah, we don't block lady gagas tweets to 40 million people until they're all delivered. Note that she counts as 1 in the tps calcu

Fanout is soft realtime, but yeah, we don't block lady gagas tweets to 40 million people until they're all delivered. Note that she counts as 1 in the tps calculation when she tweets. Btw, we're hiring if you'd like to help us make twitter even better.

itchyouch 13y ago
Not sure what the original comment was, but essentially trading firms build their own ticker plants to get their "curated" view. Obviously, twitter followers won't be building their own tweet plants. I'd imagine that the main challenges that are faced with twitter's fan out is in data persistence and scale of data involved. We could talk about how trading system architecture does not apply, but there are potentially many ways that it does. The main difference in web architecture versus trading-system architecture is the messaging data model. Trading systems are architected for low-latency by building messaging systems that focus on transmitting events as opposed to fully fleshed-out data structures. For the uninitiated, a typical web-app will commit data to a DB, then request all data. A low-latency trading system does not build an orderbook data structure and commit that to a DB, then request it from the DB, it merely listens to a stream of events (e.g. firehose) consisting of buys, sells & executions and builds its own book, so that the orderbook data-structure is always available locally in-memory as opposed to being a DB request away. The beauty with this architecture is that special data structures need not be a complex select/join away with DB's and cache layers taking on a ton of work. It allows just the events to be propagated and let each app decide for itself what it necessary and what isn't necessary and it allows the relevant data to be built in realtime, with low-latency and a high amount of determinism. Modern systems on higher-end commodity hardware that I have dealt with, with the right tuning can push around 2M 120Byte msgs/sec with an average latency of 6-7 microseconds per host. I'm told with extra/better tuning hitting 3M 120Bytes messages/sec sustained per host. Of course this is doing everything in-memory w/out disk, but even having a disk as a bottleneck doesn't preclude this kind of performance. It just introduced 5-20ms outliers in the message processing workflow as pages are flushed to disk. The way I've envisioned applying trading system arch to Twitter is where there are tweet submitters that get aggregated into a sequencer that then a bunch of fan-out machines read and committing the fanned out data structures to a memcache and/or data persistence cluster of some type. Each fan-out can be its own shard discarding irrelevant tweets. Other apps that want data can get them via other apps that provide a firehose listening directly from the sequenced data from the firehose or merely subscribing/joining the firehose feeds and processing the data appropriately. This should be able to handle the hundreds of millions of fanouts per second that are necessary in a distributed manner with great efficiency in the number of hosts involved. If you need to expand capacity, it's just a matter of bringing up extra capacity to the fan-out hosts and persister hosts.
- erichocean 13y ago
  Our architecture is similar to that for a Twitter-esque system. Event streams are really pretty awesome. :)

itchyouch 13y ago

Not sure what the original comment was, but essentially trading firms build their own ticker plants to get their "curated" view. Obviously, twitter followers won't be building their own tweet plants. I'd imagine that the main challenges that are faced with twitter's fan out is in data persistence and scale of data involved. We could talk about how trading system architecture does not apply, but there are potentially many ways that it does. The main difference in web architecture versus trading-system architecture is the messaging data model. Trading systems are architected for low-latency by building messaging systems that focus on transmitting events as opposed to fully fleshed-out data structures. For the uninitiated, a typical web-app will commit data to a DB, then request all data. A low-latency trading system does not build an orderbook data structure and commit that to a DB, then request it from the DB, it merely listens to a stream of events (e.g. firehose) consisting of buys, sells & executions and builds its own book, so that the orderbook data-structure is always available locally in-memory as opposed to being a DB request away. The beauty with this architecture is that special data structures need not be a complex select/join away with DB's and cache layers taking on a ton of work. It allows just the events to be propagated and let each app decide for itself what it necessary and what isn't necessary and it allows the relevant data to be built in realtime, with low-latency and a high amount of determinism. Modern systems on higher-end commodity hardware that I have dealt with, with the right tuning can push around 2M 120Byte msgs/sec with an average latency of 6-7 microseconds per host. I'm told with extra/better tuning hitting 3M 120Bytes messages/sec sustained per host. Of course this is doing everything in-memory w/out disk, but even having a disk as a bottleneck doesn't preclude this kind of performance. It just introduced 5-20ms outliers in the message processing workflow as pages are flushed to disk. The way I've envisioned applying trading system arch to Twitter is where there are tweet submitters that get aggregated into a sequencer that then a bunch of fan-out machines read and committing the fanned out data structures to a memcache and/or data persistence cluster of some type. Each fan-out can be its own shard discarding irrelevant tweets. Other apps that want data can get them via other apps that provide a firehose listening directly from the sequenced data from the firehose or merely subscribing/joining the firehose feeds and processing the data appropriately. This should be able to handle the hundreds of millions of fanouts per second that are necessary in a distributed manner with great efficiency in the number of hosts involved. If you need to expand capacity, it's just a matter of bringing up extra capacity to the fan-out hosts and persister hosts.

erichocean 13y ago
Our architecture is similar to that for a Twitter-esque system. Event streams are really pretty awesome. :)