2 ms·
I love reading these threads and thinking this sort of stuff to myself... How much of twitter's infrastructure is self-perpetuated or otherwise exists just to s
by bob1029 5y ago
I love reading these threads and thinking this sort of stuff to myself... How much of twitter's infrastructure is self-perpetuated or otherwise exists just to serve other infrastructure? You obviously cant fit everything their business does on a single machine, but you could probably get dangerously close with enough determination. Especially, if you just focused on the core 280 character tweet abstraction.
An options order consists of more data than an average tweet, and we can certainly process them at a higher rate than twitter would need to go in practice. Many financial exchanges operate on a single thread. 1-100 million transactions per second with jitter measured in tens of microseconds. I don't see why other software products & services can't leverage similar concepts.
- abraxas 5y agoYep, probably a single good mainframe would handle all of it with a smile. And provide the uptime that most "cloud" vendors can dream of. But that wouldn't be nowhere near as cool as building "cloud infra", right?
- saagarjha 5y agoThere’s certainly lots of room for improvement in this kind of system, but I think it’s a bit reductionist to compare this kind of work to HFT, which hits one hot path that is ultra optimized and doesn’t have to deal with complexities like “this data needs to be replicated and kept consistent”. Your question is a bit like “planes go fast, why can’t my car use some of the same tricks to go fast as well?”: yes, your car can probably go faster, and it could probably steal some aerodynamics and material science from your plane, but reducing the problem to something simple that they should “just” do is absurd.
- bob1029 5y ago> doesn’t have to deal with complexities like “this data needs to be replicated and kept consistent” I'm sure there are other excuses available, but this is not a good one. Non-repudiation is a critical system requirement of any financial exchange. The figures stated include persistence to durable media, and the whole point of running everything on a single thread is to ensure serialized processing of all activities (i.e. consistency). I would argue the need for replicating tweets is less urgent than ensuring 7-8 figure financial transactions don't go unaccounted for. We could probably make some compromises for the twitter use case to make this even faster.