3 ms·

Hi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =) Steve and John were both good developers that wrote

Hi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =) Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with no idea what they were talking about to write articles about what was going on inside of Twitter, yet do so with authority that they didn't really have. By the time a team of reliability focused people started to get hired Twitter was finally getting smart about not letting employees/outsides control the messaging sent to Techcrunch and the like. Twitter has completely reinvented the architecture since I left so that is not impossible by any means. Lets be clear about the rate that Twitter was growing though. In any 9 month window while I worked there they never failed to grow an order of magnitude in request volume. When I started we would have spikes of a few thousand requests per second. We had a major site outage at 451 tweets per second and we simply couldn't keep up. 9 months later we were spiking 50k requests per second, and we never dropped below 451 tps (barring outage). By the time I left both of those numbers simply couldn't be seen in the graphs anymore. Every time we got technology in place to deal with the scaling issues we were already behind the curve because of how fast things were growing. It felt like being in a car that has amazing 0-60 time, but it just kept accelerating well beyond 60mph.

dingaling 9y ago
If Twitter couldn't keep up with scaling then it wasn't being architected to scale, only to grow. There are financial systems that handle millions of transactions per second because their designers sat down and designed systems that could scale by magnitudes.
- liquidgecka 9y ago
  Yea but those designers knew the magnitudes that they were dealing with and built them into every single layer of the system. Its easy to build a system that handles "x million transactions a second". Its MUCH harder to build a system that handles somewhere between x, and x*100 million messages a second. Especially when in startup mode because cost and complexity matter greatly. Ultimately it wasn't "build a db that handles x writes" though. That is actually a fairly known and solvable problem. Our issue was that we were migrating from a design model that encouraged rabid development by using existing patterns to a model that would scale up much better but required a lot more planning and design, all while dealing with management that expected the feature development cycle to not slow down at all during or after the transition.

dingaling 9y ago

If Twitter couldn't keep up with scaling then it wasn't being architected to scale, only to grow. There are financial systems that handle millions of transactions per second because their designers sat down and designed systems that could scale by magnitudes.

liquidgecka 9y ago
Yea but those designers knew the magnitudes that they were dealing with and built them into every single layer of the system. Its easy to build a system that handles "x million transactions a second". Its MUCH harder to build a system that handles somewhere between x, and x*100 million messages a second. Especially when in startup mode because cost and complexity matter greatly. Ultimately it wasn't "build a db that handles x writes" though. That is actually a fairly known and solvable problem. Our issue was that we were migrating from a design model that encouraged rabid development by using existing patterns to a model that would scale up much better but required a lot more planning and design, all while dealing with management that expected the feature development cycle to not slow down at all during or after the transition.