11 ms·
Processing billions of events in real time
- deleted 5y ago[deleted]
- feketegy 5y agoDo you remember the Fail Whale? Pepperidge Farm remembers.
- fuddle 5y agoIs it just me or are most of the links broken in the blog post?
- zdwolfe 5y agoWorks for me
- arthurcolle 5y agowhat DNS server are you using, old sport?
- deleted 5y ago[deleted]
- fuddle 5y agoOops I meant to comment on this HN post: "Handling five billion sessions a day in real time" https://news.ycombinator.com/item?id=29231932 https://news.ycombinator.com/item?id=29231932
- blakesterz 5y agoThere's an HN post from a few years ago with a bunch of interesting reading from the "old days" at Twitter. https://news.ycombinator.com/item?id=17147404 https://news.ycombinator.com/item?id=17147404 I think somewhere in there is a link to a story about how only one popular user (Was it Ashton Kutcher?) could tweet at a time. I seem to recall it ran on a singly MySQL server for quite a while too.
- sharps_xp 5y agoi think it was justin bieber. a former engineer who founded Signal went on the joe rogan podcast and talked about how when bieber tweeted, the building would shake or lights flicker or something to that effect
- Supermancho 5y agoWhere I work, they use more and more google services every day, to my chagrin (since I've spent years learning the quirks of AWS). Amazon's solutions can be fit into various architectures, but are more generalist tools than the BigQuery, DataFlow and BigTable application. Google's solutions are also cheaper and/or easier to work with, for very large data processing.
- chrisjc 5y agoWhat does this mean? > but are more generalist tools than the BigQuery, DataFlow and BigTable application Doesn't AWS provide similar tools? BigQuery == Redshift DataFlow == Kinesis (Analytics/Flink/EMR-Spark/EMR-BEAM) BigTable == Redshift Spectrum
- Supermancho 5y agoSimilar in that they are building blocks that you have to tailor to be efficient for your use case. "Kinesis (Analytics/Flink/EMR-Spark/EMR-BEAM)" - this is kinda the point. In AMZ you're building out the dataflow process that fits your use-case and optimizes where you would like, rather than submitting to an existing (Google) design that is optimized+priced for very large dataset.
- arecurrence 5y agoCloud Service Usage Price matters a lot. I actively move systems off of AWS to save costs. Take Cloudfront as an example. Bunny is half a cent per GB and has no per-request cost. I've seen Cloudfront cost over 20x more. A $10,000 bill instead of $200,000 per month is well worth the engineering work. Btw, Bunny is also $10 flat rate for unlimited image processing... and they only bill the post-processed bandwidth cost. The price of my AWS equivalent for that pipeline was quite the comparison. :)
- tedmiston 5y agoI misread your comment as "Bunny" being a GCP service competing with AWS CloudFront. Is Bunny Optimizer (bunny.net) [1] what you are referring to? [1]: https://bunny.net/optimizer/#pricing-details https://bunny.net/optimizer/#pricing-details
- javajosh 5y agoPhilosophically, isn't it wrong to process billions of events in real time? I mean, the magnitude of the moral hazard is astounding. The alternative isn't to stop sending messages, but rather to not put them through a bottleneck like Twitter. This would have the dual benefit of a) not giving central control over messages and b) not requiring exotic solutions. I fear that technologists (including myself) are fascinated by the exotic solutions required by extreme centralization, and are more than happy to solve those rather than question the need for them in the first place.
- LindyTalker 5y agoHuh? People are using Twitter to create these events? Who else would you want to process it? It's not a bottleneck to put things through Twitter if people are literally using Twitter
- staticassertion 5y agoI assume they're suggesting some sort of peer to peer system.
- javajosh 5y agoYes. p2p systems would be the alternative and would, as a whole, process billions of messages but in a distributed way where each node only processes ~100. I'm further suggesting there there is a relationship between scale and ethics, via moral hazard, that is worth exploring. Too bad I'm getting downvoted instead of engaged with.
- burnished 5y agoYou haven't really explored what or why the moral hazard could be, so your current comment really sounds like "twitter big, ergo twitter evil". Its sort of on you to surface your idea instead of expecting strangers to engage with you - you seem perfectly pleasant but it frequently leads to very negative interactions.
- endisneigh 5y agoThis is an interesting blog post, but one thing I really wish more blog posts would do is compare the actual user experience from the simpler architecture. For example this new architecture, from the end-users perspective, compared to 2010 Twitter would be interesting. I'm sure much of the technology is needed for monetization, but it would be a fascinating look anyway.
- rozap 5y agoIt's true, the product is much worse than it was 10 years ago (I don't use it anymore since it's such a chore), but I don't think a company that has been obsessed with growth for growth's sake is really capable of that level of introspection. That'd be a difficult pill to swallow. Billions of dollars and thousands of human-hours spent to make something...worse. And since all that we really have at the end of the day is a desire/need to work on something that we feel is valuable, what's the point in asking the question?
- deleted 5y ago[deleted]
- saagarjha 5y agoThis infrastructure work seems to be very far disconnected from anything a user would interact with. What kind of use experience would you like it to talk about?
- rileyphone 5y ago> For the interaction and engagement pipeline, we collect and process data from various real-time streams and server and client logs, to extract Tweet and user interaction data with various levels of aggregations, time granularities, and other metrics dimensions. That aggregated interaction data is particularly important and is the source of truth for Twitter’s ads revenue services and data product services to retrieve information on impression and engagement metrics. All this engineering talent and effort to track dwell times and clicks for ads. At some point, I think we need to take a critical look at the ad economy and whether continued investments actually help anybody, or rather just give certain parties competitive advantages in selling ads. Smart regulations can perhaps make ads "dumber" and drive a lot of the engineering away, as well as being a boon to privacy, while still leaving sites with a way to generate revenue from content.
- natded 5y agoThere are probably several software engineers optimizing ads that would have otherwise made breakthroughs in some other technology, like I don't know, nuclear reactors. It is bizarre situation.
- kibblesalad 5y agoAds seem like kind of a generic and misleading term for it. It's less about showing you a flashy clip for an advertised product in the classic sense, and more about building complex personality and interest profiles on users based around their engagement patterns across sites and services. That's the real resource which organizations pay money for, especially once it gets aggregated into constantly growing targeted data collections on individual users and the increasingly monopolized share of their digital fingerprints and track records.
- eointierney 5y agoIt's* an ecology-shattering waste of resources to satisfy the profit demands of a small number of globally distributed artificial legal personalities. It's a fabulously inefficient yet monetarily useful model of pernicious theft of human attention. It's bad Physics. * It being the Advertising "Industry"
- tyingq 5y agoUndeniably impressive, but it's funny that it all lands in a UI where you can no longer tie together a thread (of said events) into something readable. Doesn't twitter get why all those 3rd party sites that string things back together are popular?
- glitchcrab 5y agoI won't use the official app or twitter.com for exactly this reason - I want to read things chronologically, not in a disjointed fashion with occasional adverts thrown in too.
- saagarjha 5y agoConsidering the recent acquisition of Threader, I’d say they are at least aware of this area: https://threader.app/ https://threader.app/
- nemothekid 5y agoI'm pretty surprised that Twitter is moving their homegrown stack to GCP. Twitter has a storied big data platform (Storm, Summingbird, Lambda Platform, Manhattan), and they have completely moved to the cloud. I wonder how the finances worked out; I'm sure Twitter paid a lot for their bespoke platform and Google must've been competitive. It was probably also easier as I imagine a lot of the engineers who considered those projects their children may have already moved on.
- secondcoming 5y agoWell, Google paid for my employer to migrate to GCP from AWS.
- geodel 5y agoWell all these projects seems Java/Scala codebase. I wouldn't be surprised as amount of data grew these systems would have started behaving unpredictably and unceremoniously dumped later on. Another thing as Twitter was (still is?) big on Scala so solid working software becomes secondary to circlejerking around Scala features and products eventually would have suffered. As a user the only Scala software I have used is Kafka and needless to say it comes across as half-assed cobbled system and not a thoughtfully designed, solid, reliable and performant system. So Google being competitive is not a must as Twitter itself does not anything reasonable for their growing needs.
- funstuff007 5y agomaybe they got a sweetheart deal being a marquee customer. Although, not as sweet as a deal that the CME got. > Google also makes $1B equity investment in CME Group https://www.cmegroup.com/media-room/press-releases/2021/11/04/cme_group_signs_10-yearpartnershipwithgooglecloudtotransformglob.html https://www.cmegroup.com/media-room/press-releases/2021/11/0...
- chillfox 5y agoOr it's the result of some good wine, dinners and gifts. The spreadsheets justifying it can always be engineered afterwards.
- EdwardDiego 5y ago> There are various event sources we consume data from, and they are produced in different platforms and storage systems, such as Hadoop, Vertica, Manhattan distributed databases, Kafka, Twitter Eventbus, GCS, BigQuery, and PubSub. I'm surprised that they mentioned Twitter EventBus, I thought that they were migrating away from that to Apache Kafka entirely. [1] Mind you, they've got a lot of tech going on, so not surprising if it's still present in legacy systems. Fun fact, if you dig into the architecture of Twitter EventBus, it will seem awfully familiar to the architecture of Apache Pulsar (storage separated from brokers, storage based on Bookkeeper), and that's no coincidence, as Sijie Guo, CEO of StreamNative developed EventBus at Twitter (also a main dev of BookKeeper). StreamNative is to Pulsar as Confluent is to Kafka. And the reasons that Twitter moved from EventBus to Kafka also apply to Pulsar, which is worth keeping in mind the next time an HN commenter proclaims "Le roi (Kafka) est mort, vive le roi! (Pulsar)". [1]: https://blog.twitter.com/engineering/en_us/topics/insights/2018/twitters-kafka-adoption-story https://blog.twitter.com/engineering/en_us/topics/insights/2...
- ntonozzi 5y agoThey use the term EventBus, but really it's a thin EventBus client library that uses Kafka under the hood. I think all EventBus clients at Twitter have been migrated to use Kafka.
- EdwardDiego 5y agoAh, cheers, that makes sense, I didn't see it mentioned further in the architecture.
- abraxas 5y agoTheir whole platform sounds like an unholy mess by the sound of that. A big pile of dung that just accreted over time and they never cleaned out their technical debt. I just left a job like that a few weeks ago. It was such a nightmare I took a slight paycut.
- EdwardDiego 5y ago
- mgaunard 5y agoStopped reading at Kafka. Java technology that combines bad networking with bad messaging and bad queueing. I would have expected one of the big Internet websites to use better technology.
- eatYourFood 5y agoElaborate?
- okigan 5y agoYes, please elaborate - what’s better and why?
- albertopv 5y agoBetter tech? Which is?
- mgaunard 5y agoWell to begin with, Java is not a serious programming language to do systems programming in, since it has poor control of networking, threads and memory. Then it's a message bus built on top of TCP. Anyone with basic understanding of networking can see that if you have a producer that wants to send the same data to multiple consumers efficiently, you should use multicast. Kafka also lacks proper mechanisms to throttle the speed of producers when consumers are too slow, which is the first thing you should ever be concerned about whenever you introduce a queue. If you want something somewhat decent within Javaland you could try Aeron.
- nikhilsimha 5y agoImpressive, but I don't get why folks don't normalize it to events per second instead. It is 463k eps for anyone wondering.
- jeffbee 5y agoAnd the nameplate capacity - up to 1GB per second - is easily within the nominal abilities of one single machine.
- bob1029 5y agoI love reading these threads and thinking this sort of stuff to myself... How much of twitter's infrastructure is self-perpetuated or otherwise exists just to serve other infrastructure? You obviously cant fit everything their business does on a single machine, but you could probably get dangerously close with enough determination. Especially, if you just focused on the core 280 character tweet abstraction. An options order consists of more data than an average tweet, and we can certainly process them at a higher rate than twitter would need to go in practice. Many financial exchanges operate on a single thread. 1-100 million transactions per second with jitter measured in tens of microseconds. I don't see why other software products & services can't leverage similar concepts.
- abraxas 5y agoYep, probably a single good mainframe would handle all of it with a smile. And provide the uptime that most "cloud" vendors can dream of. But that wouldn't be nowhere near as cool as building "cloud infra", right?
- saagarjha 5y agoThere’s certainly lots of room for improvement in this kind of system, but I think it’s a bit reductionist to compare this kind of work to HFT, which hits one hot path that is ultra optimized and doesn’t have to deal with complexities like “this data needs to be replicated and kept consistent”. Your question is a bit like “planes go fast, why can’t my car use some of the same tricks to go fast as well?”: yes, your car can probably go faster, and it could probably steal some aerodynamics and material science from your plane, but reducing the problem to something simple that they should “just” do is absurd.
- rsync 5y agoGenuinely curious ... of the 463k events-per-second that are implied here, how many of them are actual events (like a tweet or a like or retweet, etc.) and how many of them are advertising/tracking events ?
- crate_barre 5y agoThose could all legitimately be tweets. The amount of spam tweet bots makes that number almost seem small.
- deleted 5y ago[deleted]
- eismcc 5y agoWhen I was at Twitter (6yrs ago now) they averaged around 5k tweets/s. There were MANY more notifications and other system type events.
- stillicidious 5y agoSeems surprisingly low, and I imagine definitely at least timezone dependent. Any indication of what the largest kinds of spike traffic looked like?
- eismcc 5y agoI was involved in World Cup features and it definitely would spike around goals. Maybe 2-3x as I recall but it was always surprising to me how “low” the tweet volume was relative to the read volume.
- Kiro 5y agoIsn't it 4.63 million eps?
- jimmytucson 5y agoI’d love to hear more about why they deployed Google-managed message brokers and event processing protocols downstream from Kafka. If you’re already using Kafka as a broker, why introduce PubSub? For event processing, why Dataflow and not Kafka streams or ksqldb? If it’s that your company signed a sponsorship with Google, I understand. But then why not replace Kafka with Google-managed services altogether? Are there things that Kafka does that PubSub doesn’t do that you really need (such as unlimited message retention)?
- geodel 5y agoI'd think it is simply not changing all moving parts at the same time. It is good to be conservative while making such large move to cloud.
- pupdogg 5y agoTitle maybe a little misleading. After reading the document, the chart shows the new architecture as processing approx. 4-million events/sec…I think they are deriving the 400 billion events in real time by taking 4MM events/s x 86,400s/day. The actual number comes out to 345 billion but I’m guessing if we’re that high, might as well round it up-to 400 billion. I personally would consider 4-million events/s as the benchmark for processed events in real-time.
- hardwaresofton 5y agoIt looks like Kafka is by far and away the way to handle persistent logs/events at scale. AFAIK a company here in Japan called LINE has all their messaging flowing through a large kafka cluster themselves. Wonder if anyone is running large NATS Jetstream[0]/Liftbridge[1] or Pulsar[2] (yahoo runs those) clusters. I guess Pulsar might be #2 in terms of adoption at large scale? [0]: https://docs.nats.io/jetstream/jetstream https://docs.nats.io/jetstream/jetstream [1]: https://liftbridge.io/ https://liftbridge.io/ [2]: https://pulsar.apache.org/ https://pulsar.apache.org/
- wdb 5y agoAre they any alternative that aren't using the Java runtime?
- hardwaresofton 5y agoNATS is built with golang which is one of the reasons I like it… https://github.com/nats-io/nats-server https://github.com/nats-io/nats-server
- abraxas 5y agoKafka is a wonderful tool. I built a few systems on top of it and all of them delivered the scale that was promised and more. With surprisingly little hardware. I'm very hostile to a lot of hipster tech but Kafka is one of the few genuinely good pieces of tech from the whole "Big Data" craze of the past decade.
- RhodesianHunter 5y agoPulsar is a much better fit when your architecture absolutely requires many queues ex: you need one queue per customer across 100's of thousands of customers. This architecture certainly exists, but is a lot more burdensome and less frequent than partitioning by customer id across a Kafka topic.
- kortex 5y agoHow well does Kafka handle high density data (e.g. A/V and images)? I'm scouting out systems for our computer vision pipeline and Kafka would simplify the aggregation/collimation step for marshalling to GPUs, and it would be simplest if I can just send raw frames vs some alternate transport.
- mmmmkay 5y agoIt's funny, while I was reading the "before" architecture I found myself thinking "my god, why don't they just move to a cloud already? this would be much simpler if they did". I "turned the page" and there was GCP :D. Also, I wonder why they went with GCP instead of AWS. Does Twitter have a deal with Google that I'm not aware of?
- mmmmkay 5y agooh, also, what's next, the iota architecture?
- czbond 5y agoIt appears to me they left a lot of the real time processing in house, and the pub / sub to web clients and processing of that data to GCP where I'm guessing the rest of their web delivery stack is. I think they're wise to keep the real time processing in house (eg: kafka streams, etc)
- mmmmkay 5y agoagreed! the real-time processing should be bespoke and is where I think Twitter previously shined. The data processing wasn't a point of differentiation for them and so it makes sense for them to offload it to a cloud provider and let someone else deal with the operations associated with it.
- polskibus 5y agoI wonder what is the monthly cost on running their pipeline on new Google-backed system vs the old system they wrote themselves (but could've been running on public cloud before). I bet they got a massive discount, so would be good to know the unit prices (GB/s etc.) before that discount so that one can relate it to another company's problems and solutions.