14 ms·
In Twitter’s early days, only one celebrity could tweet at a time
- gringoDan 8y agoThis isn't shocking - Twitter was notorious for being held together with Scotch tape technically. Honestly this hands-on approach is an impressive example of doing things that don't scale.
- giancarlostoro 8y agoI found it amusing that Twitter was Rails' biggest advertisement. Everyone wanted to use Rails but Twitter turned into a franken app with different stacks to keep it running
- phaedryx 8y agoTwitter was Rails' worst advertisement. They used Rails as a scapegoat to hide their bad tech. I still hear things like "Rails can't scale; remember Twitter?"
- joering2 8y agoAt least 3 large companies I know my friends developers work at, over the last 5 years switched from Rails to PHP. All told me same story "after twitter, noone wants to work or touch rails anymore"
- jensvdh 8y agoAirbnb is still on rails.. Probably the largest rails app out there for now.
- Jagat 8y agoExcept that I'd bet Airbnb's read qps requirement is less than 1% of Twitter's. Write qps would be even smaller.
- MBCook 8y agoTrue, but on an absolute scale AirBNB is still very big.
- ksec 8y agoBy RPS, I imagined Shopify is largest. But then from what I understand, Shopify operate in such a way that every shop is its own app. i.e there is 100s of thousands of Basecamp / AirBnb running on the same code base and every shop is somewhat isolated. Purely in terms of App in think the largest would be Cookpad. AirBnB never shared their numbers so I don't know. https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-...
- segmondy 8y agoI don't think so. Seriously think about it. How many active users does AirBnB have? How many put their home up for rent, and how many rent in a year? I reckon you get more tweets in a day than AirBnB rentals in a year. That's how big Twitter is or small AirBnB is. :D
- MBCook 8y agoTwitter was bigger but they don’t run on Rails anymore where I gather AirBNB does.
- myth_drannon 8y agoI would imagine Shopify on Black Friday is under heavier load.
- lfxyz 8y agoBut Rails is nice to work with and whispers 99% of websites will never need to reach Twitter-scale.
- fanpuns 8y agoI had the same thought when I read this. I wasn't into Rails back then (or development) so I don't have a sense of context for what the framework was like at that point in time, but the more articles I read about Twitter in the early days, the more of a sense I get that maybe they didn't write the best code.
- scarface74 8y agoFrom 2008: Scaling is fundamentally about the ability of a system to easily support many servers. So something is scalable if you can easily start with one server and go easily to 100, 1000, or 10,000 servers and get performance improvement commensurate with the increase in resources. When people talk about languages scaling, this is silly, because it is really the architecture that determines the scalability. One language may be slower than another, but this will not affect the ability of the system to add more servers. Typically one language could be two or three, or even ten times slower. But all this would mean in a highly scalable system is that you would need two or three or ten times the number of servers to handle a given load. Servers aren't free (just ask Facebook), but a well-capitalized company can certainly afford them. http://www.businessinsider.com/2008/5/why-can-t-twitter-scale-blaine-cook-tries-to-explain http://www.businessinsider.com/2008/5/why-can-t-twitter-scal...
- sulam 8y agoYes, well, that's a nice idea in theory. In practice, you could get over 10x (sometimes 100x) the rps off a box running the new, JVM-based services vs their Rails-equivalents. Orders of magnitude probably matter a little less when you're well-funded and have hundreds of servers, but when you are thinking about trying to go public and your bottom line is being scrutinized and you have 10's of thousands of servers, it starts to matter.
- scarface74 8y agoThat's exactly what he said. but a well-capitalized company can certainly afford them But these days when you don't have to buy servers and make a long term capital commitment and you can use something like AWS, if you have a scalable but not efficient architecture and you have the faith of the investors, you can get enough servers to get you over the hump temporarily, slowly start replacing the most performance sensitive part of your architecture and then scale down. Look at what HN darling Dropbox did, they bootstrapped on AWS, got big and then when the time was right, they moved to a cheaper architecture - off of AWS and built their own infrastructure.
- 8y ago
- evanweaver 8y agoRails, and the way it used MySQL/Postgres, was designed for building CMSs. Very amenable to CDN caching to scale. Twitter was a realtime messaging platform (fundamentally not edge-cacheable) that evolved from that CMS foundation. So the reason for the difficult evolution should be clear. It's not really a coincidence that before Twitter I worked at CNET on Urbanbaby.com which was also a realtime, threaded, short message web chat implemented in Rails. Anyway the point is: use my new project/company https://fauna.com/ https://fauna.com/ to scale your operational data. :-)
- Operyl 8y agoI take it from a quick glance: you either own/work for Fauna and it’s a completely proprietary product? If so .. I’ll pass, I like building things atop stuff I can control. :)
- evanweaver 8y agoYes, it's not open source (for now). Managed cloud and on-premises edition. We will have a free download soon.
- Operyl 8y agoMight I suggest disclosing it when you post about it? Clicking through your profile revealed it, but it seems like it could read off better for you in the future. :) EDIT: either it was edited above, or I’m blind. I think it was the earlier?
- evanweaver 8y agoYeah I fixed it. Thanks.
- truncate 8y agoJust wondering how is engineering stack at Twitter now? Is it still bandaging things like everyone is talking about here (in Rails days) or they have matured to good engineering practices. Interested, as I was considering applying for engineering job there.
- myth_drannon 8y agoIf you check their Github account it looks like they built a lot of interesting Scala tech.
- sulam 8y agoThe fail whale image was retired over 4 years ago now because it almost never showed up anymore. That should tell you enough.
- evanweaver 8y agoIt's fine. Very Scala/Java/Mesos heavy. Big company integration problems now instead of startup scaling problems. They still use some of the distributed storage we built almost a decade ago though.
- tonystubblebine 8y agoOh good. This is where I get to chime in. Twitter wouldn’t have existed with out Rails. It was a side project inside of a failing startup. It took Jack and Florian like two weeks to have a solid version up that we could all use and love. That’s two weeks of designing and then having Rails make it super easy to prototype. I was in the management meetings and my read was that if they’d wasted any time fighting with technology or gold plating infrastructure then we would have moved on. So I do see Twitter as a huge Rails success story. To use the Rails term, Rails was the scaffolding. Replacing any part of Rails later isn’t a failure of Rails.
- molecule 8y ago2010: "At any moment, Justin Bieber uses 3% of our infrastructure. Racks of servers are dedicated to him" https://gizmodo.com/5632095/justin-bieber-has-dedicated-servers-at-twitter https://gizmodo.com/5632095/justin-bieber-has-dedicated-serv...
- fizx 8y agoBieber-related conversation might reasonably have been 3%. Updating his friend count was a constant source of lock contention on a particular mysql. There were never explicitly dedicated databases/servers, unless you count the hotspot caused by whatever shard he hashed into.
- lwansbrough 8y agoYou would think a HyperLogLog would be approximately good enough when follower counts reach tens of millions.
- fizx 8y agoHyperloglog was only just invented, and hadn't really made its way into the popular developer consciousness. Anyhow, the solution is easier than that (implement lock striping, i.e. https://stackoverflow.com/questions/16151606/need-simple-explanation-how-lock-striping-works-with-concurrenthashmap https://stackoverflow.com/questions/16151606/need-simple-exp...), but I was reorg'd away before my code hit production, so I'm not sure what happened there.
- lorenzhs 8y agoApproximate counters are much much older than HyperLogLog. Here’s something from 1978: https://www.inf.ed.ac.uk/teaching/courses/exc/reading/morris.pdf https://www.inf.ed.ac.uk/teaching/courses/exc/reading/morris.... Of course this solves a different problem because HyperLogLog is an, uh, interesting way to count followers (why would you need to do a count-distinct query? You can’t follow someone multiple times). In any case, the Flajolet Martin sketch dates back to 1985 and solves the same problem as HyperLogLog: https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algorithm https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algori...
- lunchbreak 8y agoWhat frameworks would you use to handle such an steep growth curve? Most startups I know of start of with rails or the like - and obviously they couldn't handle the strain. So what would you use?
- christophilus 8y agoThe one my team knows and / or likes best. When you hit really big scale, architecture is more of a differentiator than language, but you can always move inefficiencies into their own services written in a more performant technology. The thing you need most when starting is the ability to produce, test, and iterate quickly.
- phaedryx 8y agoI would use Rails, but I wouldn't do things like try to write an asynchronous message queue in Ruby (like Twitter did). I mean sites like Shopify, Hulu, YellowPages.com, etc. "handle the strain" just fine.
- mbreese 8y agoHow much of the content in those sites change as fast as Twitter does? Mainly static content can be displayed using about any framework/language. Moreso when you include caching. I don’t think Rails was simply a good choice for the workload Twitter needed to support (not that they knew it at the time).
- nkozyra 8y agoThe first question should be: do I need a framework at all. There's almost always overhead, you have to ask if it's worth it
- sulam 8y agoYou're getting downvoted, but this is one of the arguments for going with unikernels.
- tschellenbach 8y ago
- tschellenbach 8y agoIf you want to read more about activity feeds there are a ton of papers listed here: https://github.com/tschellenbach/stream-framework https://github.com/tschellenbach/stream-framework I've been working on this stuff for years. Recently I've also enjoyed reading Linkedin's posts about their feed tech. Three are a few different posts but here's one of them: https://engineering.linkedin.com/blog/2016/03/followfeed--linkedin-s-feed-made-faster-and-smarter https://engineering.linkedin.com/blog/2016/03/followfeed--li... Scaling a social network is just inherently a very hard problem. Especially if you have a large userbase with a few very popular users. Stackshare recently did a nice blogpost about how we at Stream solve this for 300 million users with Go, RocksDB and Raft: https://stackshare.io/stream/stream-and-go-news-feeds-for-over-300-million-end-users https://stackshare.io/stream/stream-and-go-news-feeds-for-ov... I think the most important part is using a combination of push and pull. So you keep the most popular users in memory and for the other users you use the traditional fanout on-write approach. The other thing that helped us scale was using Go+RocksDB. The throughput is just so much higher compared to traditional databases like Cassandra. It's also interesting to note how other companies solved it. Instagram used a fanout on write approach with Redis, later on Cassandra and eventually a flavor of Cassandra based on RocksDB. They managed to use a full fanout approach using a combination of great optimization, a relatively lower posting volume (compared to Twitter at least) and a ton of VC money. Friendster and Hyves are two stories of companies that didn't really manage to solve this and went out of business. (there were probably other factors as well, but still.) I also heard one investor mention how Tumblr struggled with technical debt related to their feed. A more recent example is Vero that basically collapsed under scaling issues.
- obelix_ 8y agoSocial network scaling is not a technology problem. Technology has imposed scaling on societies and patted it self on the head for the unintended benefits derived, while burying its head in the sand on the unintended consequences. The roman empire, Genghis Khan, the East India Company, Napoleon, Hitler etc etc etc achieved scale and then what happened?
- ma2rten 8y agoI used to work at Hyves. Hyves overcame it's scalability issues, but went out of business for other reasons. Hyves used MySQL and Memcache similar to facebook at that time. By the way, RocksDB which is now Facebook's main database (afaik) is written on top of LevelDB. So both Google and Facebook run on software written by Jeff Dean...
- JohnJamesRambo 8y agoIt would be fascinating if they returned to this capability. One celebrity gets the podium at a time.
- deleted 8y ago[deleted]
- tschellenbach 8y agoFun fact, Twitter's feed is still kinda broken. If you visit the site after being gone for a week or so it tells your timeline is empty. It recovers after a few minutes, but its still a pretty poor user experience.
- staticassertion 8y agoNot really much of a fun fact
- RockyMcNuts 8y agoI see a fair amount of stuff that's badly broken. - Analytics.twitter.com numbers changing unexpectedly, being zeroed out temporarily, inconsistent numbers - Deleting tweets and having them not show up on my timeline but still show up in search weeks later. - The repeating timeline bug, you're going down the timeline and after a while it loses its place and repeats tweets you saw a couple of minutes ago - People who have blocked me (gah) but I still see their tweets in certain apps. tbh I see some strange stuff on Facebook and Reddit sometimes and granted it's hard to do what they do at scale. But Facebook and Google products seem to evolve a lot more over the years.
- donttrack 8y agoI always get a message saying the request took too long to execute, if I click a link leading to Twitter. Have to reload the page a couple of times to make it work. Usually don't click on Twitter links for this reason.
- eythian 8y agoYes, I get "rate limited" on a single request on mobile web quite often. It's strange.
- emmanuel_1234 8y agoIf I go to Twitter on my mobile phone (using Brave, not their interface), I consistently got an error message preventing me from seeing anything. It says I apparently hit Twitter too many times (when I try to load a single tweet).
- iamaelephant 8y ago> Jason Goldman, who served as Vice President of Product at Twitter between 2007 and 2010, responded to Weaver’s tweets with the observation that early Twitter was “held together by sheer force of will.” I would dispute that, I don't think they can take that much credit. Regardless of their "sheer force of will" the site was down very, very frequently.
- liquidgecka 8y agoMuch of that was just bad design decisions in the early days creating a momentum that was unstoppable and unreplacable at the speed we were growing. In the early days they implemented a Kitama system for memcache. It worked great so long as nothing failed since a node coming in and going out could lead to bad results. In the old days they would just flush caches when that happened. Later, when memcache was the only way we could serve the loads we had it became imperative that memcache never be restarted. A single memcache restarting would overload the mysql backend so bad that the site would be down for hours. Adding memcache had more or less the same issues though it was a little easier to prewarm things. We wrote a kernel module that allowed us to change the ulimits of a running process so we could increase the file descriptor limits for memcache without restarting it.. Replacing it completely was damn near impossible given the growth rate and inability to get the data out of mysql quickly enough. Us reliability guys worked 16 hour days 7 days a week for years trying to keep things working well enough to not fail completely. Some of the crazy hacks we did just to survive were fantastic and impressive and I am still proud and disgusted by them to this day. =)
- evanweaver 8y agoIndeed. "It was amazing that it worked, and terrible that we had to do it."
- liquidgecka 8y agoI believe it was James May that said: "You have come up with a clever solution to a problem you shouldn't have had in the first place!" =)
- anyfoo 8y agoAh, I read that Twitter thread a few days (weeks?) ago and it was much longer. As far as I remember, it started with someone asking Twitter ops people, former and current, to share some stories about things that went spectacularly wrong. It contained a lot of Twitter ops battle stories, some very interesting. I was pretty impressed to read Twitter internals in the given level of detail, but now it seems that the thread that held them all together is protected (probably didn't expect it to be so popular, or just wanted to continue more privately).
- exikyut 8y agoAww. If only there was an archive of it somewhere... (I'm curious if there is)
- liquidgecka 8y agoAnd yet I bet nobody mentioned the "fire rain" in our first data center, the load bearing mac mini, or the surprise Snoop Dog visit/party! =)
- anyfoo 8y agoHah, I don't think anybody has, but I'd love to hear those stories!
- city41 8y agoIn the early days of Twitter the "fail whale" was so common it got assimilated into culture as a term to use for anytime any site gets overloaded. Nowadays it seems like that term is "hugged to death" https://www.theatlantic.com/technology/archive/2015/01/the-story-behind-twitters-fail-whale/384313/ https://www.theatlantic.com/technology/archive/2015/01/the-s...
- paulcole 8y agoFor us old timers the term is “slashdotted.” https://en.m.wikipedia.org/wiki/Slashdot_effect https://en.m.wikipedia.org/wiki/Slashdot_effect
- throw_away 8y agoAs slashdot's active user count dwindles and as it has gotten a bit easier to build a site that can handle decent traffic, I wonder if there was (or will be) a specific last site to get literally slashdotted.
- flurdy 8y agoI doubt you get many/any sites slashdotted from the actual slashdot these days. But you do see sites slashdotted from HN articles every day. And you will be going forwards as well. People will not create horizontally autoscalable, cachachable sites as that cost money and effort for 99.9999% sites that will never need it. True we are at an age were it is easier, more well known how to do it, and less and less costly, but still not free in money and time.
- liquidgecka 8y agoEverybody knows about the "fail whale".. nobody knows about the "moan cone".. The image is lost to history but it was captured by this account ages ago: https://twitter.com/moan_cone https://twitter.com/moan_cone IT was thrown when the system was failing at the rails layer rather than the apache layer. I believe that Ryan King and I were the last people to ever see the moan cone in production. =)
- collinf 8y agoI haven't seen anyone touch on this, but I remember reading about this in Data Intensive Applications[1]. The way that they solved the celebrity feed issue was to decouple users with high amounts of followers from normal users. Here is a quick excerpt, this book is filled to the brim with these gems. > The final twist of the Twitter anecdote: now that approach 2 is robustly implemented,Twitter is moving to a hybrid of both approaches. Most users’ tweets continue to be fanned out to home timelines at the time when they are posted, but a small number of users with a very large number of followers (i.e., celebrities) are excepted from this fan-out. Tweets from any celebrities that a user may follow are fetched separately and merged with that user’s home timeline when it is read, like in approach 1. This hybrid approach is able to deliver consistently good performance. Approach 1 is a global collection of tweets, the tweets are discovered and merged in that order. Approach 2 involves posting a tweet from each user into each follower's timeline, with a cache similar to how a mailbox would work. [1] https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321/ref=sr_1_1?ie=UTF8&qid=1527213498&sr=8-1&keywords=data+intensive+application&dpID=51PjhtI9VRL&preST=_SX218_BO1,204,203,200_QL40_&dpSrc=srch https://www.amazon.com/Designing-Data-Intensive-Applications...
- hinkley 8y agoIt’s an oft overlooked inequality in these systems. People get so wrapped up in some whiz bang thing that they don’t stop to think if they should. At the end of the day, one of the most important aspects of your information architecture is how many times is each write to the system actually observed? That answer can dictate a lot about your best processing strategies.
- tmd83 8y agoThis. It took me a while to learn to look at this and eventually focusing less on caching on some items that don't enough read in it to justify the caching or pre calculation.
- sewercake 8y agoThere was a fun high-scalability article around their 'fan-out' approaches to disseminating tweets by popular users etc : http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html http://highscalability.com/blog/2013/7/8/the-architecture-tw... . When I was working on something with similar technical requirements I also came across this paper (http://jeffterrace.com/docs/feeding-frenzy-sigmod10-web.pdf http://jeffterrace.com/docs/feeding-frenzy-sigmod10-web.pdf) that outlined the approach in a more 'formal' manner.
- evanweaver 8y agoMissed that this went to the front page! I will answer questions if I can. I am now CEO @ https://fauna.com/ https://fauna.com/, making whales and fails a thing of the past for everybody. We are hiring, if you want to work on distributed databases.
- liquidgecka 8y agoLong time no see man! =)
- evanweaver 8y agoYou too! Hope Zookeeper isn't still keeping you up.
- liquidgecka 8y agoFunny thing is that the place I am at now uses Zookeeper.. luckily it has not been trouble for us though. =)
- evanweaver 8y agoIt's saving it up for you...remember deleting thousands of Nagios notifications on the bus? Those were the days.
- liquidgecka 8y agoI remember all those iPhone suckers having to download special apps to remove conversations in the early days since it only allowed you to delete them one at a time. =) I also remember getting on a bend to make all fake alerting go away after I started only to find that like 50% of the configured alerts couldn't ever succeed and had been paging every 10 minutes for over a year. We got that stuff sorted out pretty quick. Once we started getting good visibility it helped us make much better decisions about where the problems actually were. The first six months felt like everything was just on fire constantly and nobody know what was going on, but then things started falling in place and it felt like everything was still on fire, but some people actually had fire extinguishers for a change. =)
- liquidgecka 8y agoIf anybody is interested in random Twitter internal stuff I might bang up a medium post one day. I was neck deep in the infra side of things for years and have all sorts of funny stories. Our managed hosting provider wouldn't let us use VPNs or anything that allowed direct access to the managed network they provided, but we wanted to make internal only services that were not on the internet so I setup a simple little system that used DNS to point to private space in the office and a SSH tunnel to forward the ports to the right places. Worked great, but over time the internal stuff grew up, and our IT team refused to let me have a server in the office so it was all running of a pair of mac mini's. We called them the "load bearing mac minis" since basically 90% of the production management traffic went over the SSH tunnels they hosted. =)
- kornish 8y agoExtremely interested! Sounds like some great stories from the trenches.
- klausjensen 8y agoPlease make that post - would love to read it.
- Improvotter 8y agoPlease do! Sounds very interesting!
- lagadu 8y agoPosting that here is like showing a big juicy steak to a pit full of hungry lions: of course we're interested!
- mpfundstein 8y agoMake a mailing list please. Don’t want to miss that
- Fradow 8y agoI like the name of "load bearing mac minis". Reading that I wonder how much mac minis are used by actual user vs used headless in offices for various purpose. There is one such mac mini in my office (named "buildbot", because that was its original purpose) that is heavily used by all the dev team (thankfully not connected in anyway to production, apart from simple uptime monitoring). Obviously, like any other I'm interested in those funny stories you have.
- logicallee 8y agoSay what you will, Twitter managed to make a $25.23B company out of printing char whatever[140];
- tombell 8y agoshould be 141 if you want to print out a string that would be 140 characters, NULL termination is a thing.
- deleted 8y ago[deleted]
- adrianhel 8y agoThis made me laugh and feel better. Being in a startup is tough!