4 ms·
I have actually built and scaled a system similar to Twitter called Plurk (which has millions of users, billions of data items and similar use cases as Twitter)
by amix 13y ago
I have actually built and scaled a system similar to Twitter called Plurk (which has millions of users, billions of data items and similar use cases as Twitter). We built this with a team of a few developers and one sys-admin. While Twitter is at a much larger scale, I really doubt you need 1000 engineers to solve that problem, because even at Plurk's scale we built automated systems for scaling, monitoring, backups etc. (e.g. sharded the databases, used master-master MySQL setups that supported auto-failovers, nagios for alerts etc.)
You may say: but they have mobile apps? But so do a lot of other companies that don't have 1000 engineers. You need a few really great mobile devs to build great mobile apps, again based on my experience.
I think what has happened for them is that their structure isn't that good and they have scaled too fast (and in this process they have hired a lot of B and C people that aren't that good). It's a least my conclusion based on the limited information I know about them.
- lifeisstillgood 13y agoI for one would be (honestly) interested in reading your article in the same vein as the original - what would you say is and is not necessary in his take?
- Irishsteve 13y agoTake a look at https://github.com/twitter https://github.com/twitter this should give some insight into where they may be spending their time. There are a few projects that would be well known (Storm, Kestrel, Bootstrap, FlockDB). Although I'm 100% if they were created at Twitter or just hired in.
- hythloday 13y agoIf Plurk has 5 developers serving 1 million daily users, and Twitter has 1000 developers serving 100 million daily active users, you only have to believe that scaling by 2 orders of magnitude is twice as hard for it to be a tie in terms of developer:user ratio. (I suspect the write load is rather different - the figures I can find for Plurk suggest 8 billion messages in total - Twitter writes half a billion a day.)
- csbrooks 13y agoThe big advantage of software though is that you don't have to maintain a fixed ratio of developer:user like that, though. Sure, your employee count should increase as user count does, but far less than linearly.
- parbo 13y agoHe suggests that it might scale logarithmically, not linearly.
- hythloday 13y agoI'm not sure this is in practice true. Twitter have to start writing or tuning their own scale-specific software, need more analytics, application security, developer productivity, partner and sales engineering etc. proportionally than they would do at 1% of their size. Certainly they could run 100 Plurks with many fewer than 1000 engineers but I'm not sure that's the best way to characterize Twitter.
- amix 13y agoI don't think there's a huge differences serving 1 million daily users or serving 100 million. In both cases you need highly scaleable systems. In 100 million case you need a lot more computers, but I think the overall structure of the systems could be very similar (e.g. sharded databases, auto fail-overs, extensive monitoring etc.) Of course you need a larger team serving 100million users, but I don't think you need to scale the team as much as Twitter has done.
- erobbins 13y agoone of the problems there is that everything is extremely convoluted.... every single thing is a huge rat's nest of cross team dependencies. I found it an extremely frustrating environment to try and work in.