6 ms·
How can over 100 people be needed for basically a scaled blog system that consists only of 140 character titles (not even any article content) and RSS feeds? It
by ck2 16y ago
How can over 100 people be needed for basically a scaled blog system that consists only of 140 character titles (not even any article content) and RSS feeds? It doesn't even have a search feature (past the previous week) or html, or tags, etc.
Doesn't WordPress.com do it with like a dozen people and their system is way, way, way more complex?
update: no, wait, in December 2010 they announced it's over 350 people now
http://blog.twitter.com/2010/12/stocking-stuffer.html http://blog.twitter.com/2010/12/stocking-stuffer.html
What on earth are they all doing?
- noahlt 16y agoiPhone, Android, and Mac clients. Maintaining as well as improving the web interface and the API. Lots more scaling - people check Twitter way more than they do Wordpress-hosted blogs. I'm sure there's more, this is just what came off the top of my head.
- benologist 16y agoWasn't their iPhone stuff a single guy who had a nice/leading client they bought last year?
- robryan 16y agoThat client would then touch a lot of other people though, even if it was primarily the one guy. Changes that are rolled out on the platform would need to be considered in terms of this app along with the all other platform apps. Sure when independent a single guy can sit there and create something like that solo, inside Twitter though there would be a lot more process and interaction to achieve the goals.
- Samuel_Michon 16y agoYup, Loren Brichter, who wrote Tweetie. Twitter bought his company, Atebits, and hired Loren to continue working on the iOS Twitter client. http://en.wikipedia.org/wiki/Tweetie http://en.wikipedia.org/wiki/Tweetie
- shaunxcode 16y agoI initially upvoted you because I agree - it is mind boggling but really I think the reality is that scaling at levels like that is a much bigger job/problem than you/I can imagine. Also I doubt all of the people are technical?
- ck2 16y agoI just have to start wondering when it's over 100 though and now they are at 350. They don't have an ad sales force right? They don't have a billing department right? Maybe they have housekeeping and food prep people included in that 350? I honestly don't know and my instinct could be very wrong, but 350 seems crazy high.
- ganjianwei 16y agoThey do have an ad sales force. http://bits.blogs.nytimes.com/2010/08/24/new-hires-at-twitter-put-the-focus-on-making-money/?dbk http://bits.blogs.nytimes.com/2010/08/24/new-hires-at-twitte...
- sahillavingia 16y agoI upvoted you because I have no technical proficiency in the (engineering) challenges they face and would love an explanation by someone from Twitter about their apparently ridiculous employee numbers.
- scarmig 16y agoQuora to the rescue: http://www.quora.com/Twitter-Inc-company/Why-does-Twitter-need-300-employees http://www.quora.com/Twitter-Inc-company/Why-does-Twitter-ne...
- BenSchaechter 16y agoHuzzah, I asked this question!
- ck2 16y agoWow, very nice find. I took a stab at the question via Google but gave up. I also discovered they have their own (but singular) datacenter: http://engineering.twitter.com/2010/07/room-to-grow-twitter-data-center.html http://engineering.twitter.com/2010/07/room-to-grow-twitter-... But they are planning on adding more, which leads me to believe maybe that's what some of the new hires are for.
- wh-uws 16y agoWow, very nice find. I took a stab at the question via Google but gave up. Exactly why quora is compelling. If you didn't know now you do
- js2 16y agoIf you ever wonder about why a company needs X developers, please come back to this list.
- tzs 16y agoHandling on average something like 76000 status updates per second (much higher when something interesting happens in the world), each of which must quickly become available to anywhere from 1 to a million clients. Wordpress.com is trivial compared to Twitter.
- ck2 16y agoYes but that's handled by hardware, not by people, and I am sure only a few critical coders handle the software performace. WordPress.com is not as trivial as you might think http://en.wordpress.com/stats/ http://en.wordpress.com/stats/ Certainly it's just a matter of scaling once you hit a certain level of volume, you just have to be able to bring more servers online into the grid. Scaling from 10,000 users to 1 million is probably very hard. Scaling from 1 million to 100 million, well you better have a pattern down that works with easy hardware replication (like google does).
- __Joker 16y agohttp://www.quora.com/Twitter-Inc-company/Why-does-Twitter-need-300-employees http://www.quora.com/Twitter-Inc-company/Why-does-Twitter-ne...
- jacques_chester 16y agoBased on the stats on the site, I calculate that Wordpress.com is averaging about 10 writes and around 8800 reads per second. edit: Correction, ~880 reads per second.
- ck2 16y agoLooking at the posts per day detail I think it's more like 30 per second with bursts that probably go to twice that, while nighttime is idle depending on timezones. But I agree it's a fraction of 7k/sec peak for twitter. However, twitter does not have to parse html, has no plugins to execute or templates, and has a max string length of 140 characters which is trivial. Each post/comment published on wp.com takes many, many more cpu cycles than twitter.
- deleted 16y ago[deleted]
- x5315 16y agoOne thing that CEO Dick Costolo has said is the amount of technical debt we have (http://kara.allthingsd.com/20110117/full-dces-interview-video-twitter-ceo-dick-costolo/ http://kara.allthingsd.com/20110117/full-dces-interview-vide...). The scale is simple tremendous, and you've got to remember: Twitter isn't just a website; it's an SMS service, clients and an API.
- ra 16y agoTwitter.com is probably one of the great engineering feats of todays social Internet. You tweet from your mobile phone app (maintained by twitter) or twitter.com, and that tweet gets posted on your public timeline. The tweet is parsed for mentions, and it is also copied on to the mentionee's incoming timeline and their mentions timeline. The tweet is parsed for hashtags, and active searches (i.e. ones open on twitter.com, or via the APIs) and is also copied onto those timelines. Also, you have followers. So Twitter also copies the tweet onto the inbox timeline of each of your followers. Not only is all this information published in real time on Twitter.com, but it is also made available via JSON API, Streaming API and Firehose API. All tweets (on all timelines) are stored forever. Twitter's scale is hard to fathom - all of this processing is way beyond what your Rails / Django app could process with a MySQL backend. MySQL replication wouldn'e even come close to keeping up with the sheer volume of events to be processed. To make matters more complicated - Twitter is expected to scale to meet the demand of emerging world events (eg: Egypt, Iran, snowstorms, hurricanes, earthquakes, bushfires). These events don't evenly spread traffic across Twitter's network, but instead provide "storm surges" of localised intense traffic. Oh, and Twitter haven't just launched an analytics product? I think for sheer engineering at scale, there are maybe only about half a dozen other companies in the same league. (Edit: grammer changes for readability)
- ck2 16y agoThat's a good breakdown of the processes but I am positive all that is done in stages and queues. So with a good design you have groups of servers doing the different stages in the queue. One you've got the pattern down for 100 tweets per second, the pattern should be reproducible by scaling servers in each queue to 1000 tweets per second, and eventually 10,000 tweets per second. The database requirements may explain why it's all done in one datacenter instead of trying to do replication across the country/world.
- jacques_chester 16y agoIn the research literature this is called SEDA: http://en.wikipedia.org/wiki/Staged_event-driven_architecture http://en.wikipedia.org/wiki/Staged_event-driven_architectur...
- ry0ohki 16y agoOnce you hire over 15 people, you need to start hiring people to manage the HR issues that it brings, and then once you have 20 why you need middle management, and at 40 you need someone to write internal tools and manage hardware etc... then for every 40.... you get the point. Past a small team, employee count snowballs.
- cletus 16y agoWell you need 100-200 engineers just to try and make Cassandra scale. The rest do the production site with more proven technologies.
- whopa 16y agoDespite the facetious nature of the comment, there's a kernel of truth there: Twitter actually did hire 3 people just to work on Cassandra.