3 ms·

I followed this story back in 2006 to 2008. I’m sorry I don’t have time to write a real comment with the crucial details. The basic plot is that Twitter initial

I followed this story back in 2006 to 2008. I’m sorry I don’t have time to write a real comment with the crucial details. The basic plot is that Twitter initially set out to be a CMS sort of like WordPress, but then they realized they had actually built a messaging platform, which needed a different architecture. You can catch a glimpse of the issues in “Amateur hour is over at Twitter” from 2008. https://www.google.com/amp/s/techcrunch.com/2008/04/23/amateur-hour-over-at-twitter/amp/ https://www.google.com/amp/s/techcrunch.com/2008/04/23/amate... Of course, it is a stretch to suggest that in ten years they couldn’t reinvent the architecture. It took them many years to get beyond the “fail whale”. And it is surprising who they thought of as experts at scaling: “And this week they hired two scaling experts – John Kalucki and Steve Jenson (“known for his work scaling Blogger and Blogspot”).” No offense to those guys, but Blogger and Blogspot were also notorious failures at scaling. Bloggers problems were not fixed till a year or two after they were bought by Google.

liquidgecka 9y ago
Hi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =) Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with no idea what they were talking about to write articles about what was going on inside of Twitter, yet do so with authority that they didn't really have. By the time a team of reliability focused people started to get hired Twitter was finally getting smart about not letting employees/outsides control the messaging sent to Techcrunch and the like. Twitter has completely reinvented the architecture since I left so that is not impossible by any means. Lets be clear about the rate that Twitter was growing though. In any 9 month window while I worked there they never failed to grow an order of magnitude in request volume. When I started we would have spikes of a few thousand requests per second. We had a major site outage at 451 tweets per second and we simply couldn't keep up. 9 months later we were spiking 50k requests per second, and we never dropped below 451 tps (barring outage). By the time I left both of those numbers simply couldn't be seen in the graphs anymore. Every time we got technology in place to deal with the scaling issues we were already behind the curve because of how fast things were growing. It felt like being in a car that has amazing 0-60 time, but it just kept accelerating well beyond 60mph.
- dingaling 9y ago
  If Twitter couldn't keep up with scaling then it wasn't being architected to scale, only to grow. There are financial systems that handle millions of transactions per second because their designers sat down and designed systems that could scale by magnitudes.
  - liquidgecka 9y ago
    Yea but those designers knew the magnitudes that they were dealing with and built them into every single layer of the system. Its easy to build a system that handles "x million transactions a second". Its MUCH harder to build a system that handles somewhere between x, and x*100 million messages a second. Especially when in startup mode because cost and complexity matter greatly. Ultimately it wasn't "build a db that handles x writes" though. That is actually a fairly known and solvable problem. Our issue was that we were migrating from a design model that encouraged rabid development by using existing patterns to a model that would scale up much better but required a lot more planning and design, all while dealing with management that expected the feature development cycle to not slow down at all during or after the transition.

liquidgecka 9y ago

Hi! I am one of the dudes that actually helped kill the fail whale so I feel I can speak to this topic. =) Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with no idea what they were talking about to write articles about what was going on inside of Twitter, yet do so with authority that they didn't really have. By the time a team of reliability focused people started to get hired Twitter was finally getting smart about not letting employees/outsides control the messaging sent to Techcrunch and the like. Twitter has completely reinvented the architecture since I left so that is not impossible by any means. Lets be clear about the rate that Twitter was growing though. In any 9 month window while I worked there they never failed to grow an order of magnitude in request volume. When I started we would have spikes of a few thousand requests per second. We had a major site outage at 451 tweets per second and we simply couldn't keep up. 9 months later we were spiking 50k requests per second, and we never dropped below 451 tps (barring outage). By the time I left both of those numbers simply couldn't be seen in the graphs anymore. Every time we got technology in place to deal with the scaling issues we were already behind the curve because of how fast things were growing. It felt like being in a car that has amazing 0-60 time, but it just kept accelerating well beyond 60mph.

dingaling 9y ago
If Twitter couldn't keep up with scaling then it wasn't being architected to scale, only to grow. There are financial systems that handle millions of transactions per second because their designers sat down and designed systems that could scale by magnitudes.
- liquidgecka 9y ago
  Yea but those designers knew the magnitudes that they were dealing with and built them into every single layer of the system. Its easy to build a system that handles "x million transactions a second". Its MUCH harder to build a system that handles somewhere between x, and x*100 million messages a second. Especially when in startup mode because cost and complexity matter greatly. Ultimately it wasn't "build a db that handles x writes" though. That is actually a fairly known and solvable problem. Our issue was that we were migrating from a design model that encouraged rabid development by using existing patterns to a model that would scale up much better but required a lot more planning and design, all while dealing with management that expected the feature development cycle to not slow down at all during or after the transition.