20 ms·

Twitter Shifting More Code to JVM, Citing Performance and Encapsulation

228 points by colin_jack 15y ago

raganwald 15y ago
I like this. I recall a startup some years back with a vision to build something complex. The founder sat down with everyone and said that we had an important decision to make: Do we build something that grows or do we build something that evolves? Neither way forward was presented as an obvious win. Building something that grows meant overengineering and trying to forecast the future. Building something that evolves meant constantly rewriting things, taking time away from new development. Twitter seems to have taken the "evolution" route, and it has served them well. I'm not at all sure they would be this successful if they'd tried to build on the JVM right from the start using the technologies available at the time.
- div 15y ago
  Building something that evolves feels like the smarter intuitive decision. It allows you to focus on features and pivoting over ceremony (boilerplatey / architecture stuff) and when you finally start hitting those performance limits, you have a luxury problem. It will also be more apparent where to invest time spent scaling your product, whereas if you try to optimize out of the gate, you may still hit unforeseen performance issues.
  - kenjackson 15y ago
    It allows you to focus on features and pivoting over ceremony This always sounds nice, but I've found that ceremony almost never really reduces productivity by much. The real wins/losses almost never have to do with ceremony related features of language, but rather architectural and framework components. You build something that evolves by bringing in the smallest architecture and least fx components and then build. The language you use may dictate the fx components to some extent -- otherwise it's typically just to make developers feel happy (which is important, but really is just about morale more than anything truly inherent in the productivity of the language).
    - div 15y ago
      When I said ceremony, I did mean working around architectural up-front decisions. I was arguing in favor of 'growing something' as opposed to 'designing something' with regards to a hopefully growing userbase. Language choice may be of lesser importance as you say, but I do feel that some languages fit the growing strategy better while others have a more design up front feel to them.
- nupark2 15y ago
  I'm not at all sure they would be this successful if they'd tried to build on the JVM right from the start using the technologies available at the time. I'm not sure why you think that, or why you think Twiiter had a 'grow' vs 'evolve' strategy. Their approach seemed not nearly so self-aware, and resulted in serious, core deficiencies that required considerable time and expense to correct. From an external position, it very much did not look like a rational considered approach to solving he problem through 'evolution'.
  - raganwald 15y ago
    I never suggested they were self-aware, only that they evolved. Therefore, I do not suggest that they decided anything. All evolution strategies involve serious, core deficiencies, that's the trade-off, that's why it isn't 100% obviously better to evolve than to grow. But I do suggest that evolution--however it came about at Twitter HQ--has worked out for them.
    - deleted 15y ago
      [deleted]
- angryasian 15y ago
  don't know why everyone assumes jvm language = over engineering. I agree that some jvm language code can be more robust, but the reality is most of us should live in our test suites. I would even say that Java test suites start and run faster than the standard rails tests ( without all the things to make it faster like spork). The frameworks argument of configuration vs convention I think is moot as most jvm language frameworks are taking a more conventional approach, but allowing for greater configuration now. on the other side the reality of a startup is you're going to push out so much code / features / product based on demand so fast that a lot of the things that you consider building for the future will be thrown out the window pretty fast. Just build it, see if it works, and move to the next thing.
- eropple 15y ago
  These aren't incompatible, though. Building something that "grows", even to very large scale, is not that difficult, and still leaves the door open to evolutionary improvements and changes to the fundamental architecture over time if they become necessary. The key is, when designing your growth architecture (and I don't consider it overengineering--a few simple decisions allow you to scale pretty well to a fairly large load), is to code to interface rather than code to implementation. If things evolve/change over time, you've already got half the work done for you because things don't spontaneously break on change.
  - raganwald 15y ago
    It is incredibly easy to design for growth when looking in a rear view mirror. For example, it is easy NOW to see that Twitter's problem was scaling, and that they should have planned to grow their scalability while evolving their user experience. However, at the time it might have gone the other way: Perhaps their user base and volume might have increased at a slower place, leaving plenty of time to evolve their scalability, while their user experience might have required relentless change. In which case, they should have planned for their user experience to "scale" rather than their infrastructure.
    - eropple 15y ago
      True. That said, designing for scalability isn't that hard, and does not take that much more time than just bunging out whatever comes first to mind.
      - jshen 15y ago
        You're assuming that you know ahead of time what the bottlenecks are, and more importantly, which trade offs to make, because scaling is almost always about trade offs. A scalable solution is usually a more rigid solution as well, and that can be a problem for a startup that may need to try a few ideas before they hit on what works.
  - dasil003 15y ago
    This is armchair architect thinking. Sure, coding to the interface makes scaling easier, it also makes maintenance, testing and everything else easier: it's just a good idea. But the devil is in the details. When you have to scale like Twitter you are getting into the territory where you have to do a lot of legwork just to figure out how to scale a particular interface. Performance and consistency issues start to gnaw away at the ideal interface you carefully designed back when you were 20k users and 10 reqs/s.
- grok2 15y ago
  Twitter's approach (atleast in the early stages) has been more of Darwinian evolution ("Shit, we have to do something about this or we're screwed") rather than planned evolution :-). It has probably served them well -- the growth they saw in usage meant they could without thinking too much whether something was worth it, evolve to more sophisticated solutions as time went by.
- bborud 15y ago
  The problem with Java is mainly that people think they have to use all the horribly complex, intrusive, badly written, underperforming frameworks that exist in the Java sphere. And then, of course, you will inevitably be screwed. To this day it takes real effort to convince Java programmers that a lot of "best practices" are anything but. For instance I still see Java programmers use frameworks that require them to maintain mountains of flimsy XML configs for things that they ought to have done in code. Both to get the benefit of letting the compiler do the work of weeding out boneheaded errors and to get rid of unnecessary flexibility that just leads to more work and more confusing, hard to read code. Java is a great language in which you can be very productive. But productivity means that you have to crack down on people who drag along J2EE-crap, or whatever crap was invented to make J2EE-crap slightly less crap. It also means you have to mentor people actively and harass anyone who even thinks of doing in XML what can be accomplished much faster in testable, compiled Java code.
  - nerd_in_rage 15y ago
    Yep, I saw this same thing. Mountains of XML deployment descriptors / "component wiring" (some "Spring" garbage), needlessly verbose and usually unneeded mapper classes, useless interfaces (my rule is if you are only going to have ONE implementation save the interface for later after you know you got it right and need it), extra layers that do nothing (except "map", of course), etc. kill me. Makes me want to kill myself.
    - bluelu 15y ago
      That's one of the reasons I left my first and last employer. Tons of compnontent wiring written in xml, tons of configuration directives, but the end only the defaults were used anyway. Impossible to do updates because everything could break.
    - GrooveStomp 15y ago
      This exactly describes my very brief experience working with Java professionally. Back in university it was no big deal - the differences between C++ and Java weren't that big. In a production environment, though; ... XML, XML, XML! All the XML-wrapper tools for Java are supposedly the reason why there is so much XML config stuff going on - but the result is that you end up writing soo much code in XML instead of the actual language that you're supposed to be using. It's like somebody looked quickly at the MVC pattern, got it all wrong, and now Java has very tight coupling between XML and Java code for any production environment that hopes to leverage existing tools.
- 6ren 15y ago
  By "something that grows", I think you mean "something that scales". It confused me, because Brooks talks about "growing' software (as opposed to building/planning it), which is I think what you mean by "something that evolves". FWIW, I think the standard wisdom today is to evolve (your term) software (e.g. agile ideas of YAGNI, DTSTTCPW), and only to scale/plan it if you have a very clear idea of what you're doing (e.g. frozen specs, which exist in some government/military domains; long-term standards; mathematics) - and you also know how to do it, having done it a few times before.
- amurmann 15y ago
  I don't think Twitter would have existed, if they had had to take the "grow"/plan for scale path. As I understand, Twitter originally was little tool, that the founder just wanted to have for them selves. If they had thought, they would have to take months to implement it in a scalable version right away, they might not have done it. I also think it's a good idea to throw a bunch of small, simple stuff on the wall and see what sticks. If everything is planned to be a big engineering effort, made to scale to millions of users, you spend a lot of time building stuff nobody will care about. That's why frameworks like Rails are great. They probably aren't perfect for your use case, esp. once you want to start scaling to crazy user numbers (although high user number will probably work very well), but they help you get a useable version out of the door crazy fast, compared to an all custom implementation.
clintjhill 15y ago
In my opinion, the best part of their infrastructure is their willingness to specialize in languages within each stack. It isn't simply "Java everywhere". "To allow developers to choose the best language for the job, Twitter has invested a lot of effort in writing internal frameworks which encapsulate common concerns." The single best investment companies can make is to allow developers to choose their specialty, and their language. Otherwise you have a huge overhead of skill set mismatch. And your talent pool can be bigger if you're open to more than one language. It's a refreshing view.
- badmash69 15y ago
  Twitter can afford the to have multiple languages in their stack but most start-ups simple don't have that luxury. If you have a early stage start up with about 10 developers and you need to quickly build a set of server side component or services for your product, would you want to use multiple languages to develop that or use one programming language ? Having developers choose best languages for the job would be disastrous if you end up with your stack written in 5 different languages. I would not discount the importance keeping codebase easy to manage.
  - shazow 15y ago
    It depends on how much effort it would take to rewrite any of said components. In my personal experience, for reasonably isolated and small components, 80% of time spent creating a component of code is spent on making decisions. By the end of it, when all the decisions have been made, I could probably rewrite the entire thing from scratch in 20% of the time—and it would likely be better in every way. It would seem that the first iteration is largely prototyping. If you can save a significant multiplier of time by choosing a different language than the rest of the stack for the prototype and potentially rewrite it later if necessary, why not? Perhaps by the end of it all, you'd break even on time but end up with multiple implementations and better code.
  - div 15y ago
    I think it's important to look at how twitter got there though. They'd been using just Rails for a long time before introducing Scala. At that point, it's fair to say they weren't a 10 dev startup anymore. This, imo, is part of having developers choose the best language for the job. They chose ruby to get their mvp out quickly and be able to grow their userbase. After that, they pinpointed trouble areas in their architecture and made pragmatic choices in fixing up those areas. While I suppose it's possible to "let developers pick the best tool for the job" and end up with 5 different languages in your stack, most good developers are likely to steer clear of such an endgame, unless the trade-offs are very, very clear.
sehugg 15y ago
Netty is a godsend. Java threads are extremely memory-hungry, so async I/O is a must for handling many connections. We routinely handle 200K simultaneous connections on our push servers without breaking a sweat.
- fictorial 15y ago
  Mind if I ask what the specs are for those servers?
  - sehugg 15y ago
    AWS m1.large. We don't use nearly the entire CPU or memory footprint, but m1.small just is a wee too tiny and there's nothing in between.
lemming 15y ago
This sounds slightly misleading to me: The primary driver is honestly encapsulation, so we can iterate faster as a company. Having a single, monolithic application codebase is not amenable to quick movement on a per-team basis. So when we decide to encapsulate something, then because of our performance concerns, its better to rewrite it in the JVM for most systems, than to write a new Ruby system. It sounds from that like their primary driver for using the JVM is actually performance, but that they only decide to rewrite components when encapsulation drives them to do so. I can't see how the JVM provides any encapsulation benefits over Ruby for new systems.
msie 15y ago
Holy name-soup, Batman! Stuff that was new to me: Gizzard, Finagle, Blender, Netty and Earlybird.
jrockway 15y ago
jrockway's law: add enough developers to a project and it eventually becomes Java.
- aristidb 15y ago
  Except it seems to be Scala here?
  - aphexairlines 15y ago
    "In the case of the search team, since they do a lot of work on Lucene, which is Java-based, they have a lot of experience in writing Java code. As such it is more convenient for them to work in Java than Scala or another language."
SoftwareMaven 15y ago
One of the important lessons, IMO, is that you should always be making pragmatic decisions that work today and into the near future. You can't predict how your system will change over time, so engineer in today's needs and let tomorrow take care of itself. Pragmatic failure inevitably leads to analysis paralysis. Just worry about getting stuff done. :)
gary4gar 15y ago
I wonder how the story had been, if twitter had opted for Python, instead of ruby. Do they still have these problems or in these aspects python is better than ruby?
- hkarthik 15y ago
  Given the huge growth of the site in the first few years and the relative inexperience of the founding team with problems of this magnitude, I expect the story would have been the same with Python, Java, .NET, etc. The simple Web Server -> ORM -> Relational database architecture that most modern web frameworks utilize can easily break down under tremendous concurrent load, especially if you attempt to run it on commodity hardware.
- glassx 15y ago
  If I recall correctly, Google switched their crawler from Python to C++ for performance and for library compatibility etc. And eBay moved the whole website from C++ to Java in 2002. IMO, the problem is that Twitter was Ruby's poster while both were still in ascension (might as well say they still are). The first migration they did was porting their message queue from Ruby to Scala. As mentioned on a TheGuardian article, they migrated because Starling was crashing too often and dropping tweets, so they had to stop the site and migrate manually. Also, another member claimed performance problems on some blog posts. However, Starling doesn't perform that well to Ruby standards either, so it's very difficult to say. I'd say the migration was for "social" reasons. The team is clearly waaaay more comfortable and enthusiastic about Scala, also they seem to prefer Static typing, so using it is the best choice indeed...
fizx 15y ago
One of the key things that was kinda skimmed over: You should check out http://github.com/twitter/finagle http://github.com/twitter/finagle. It takes async network programming with netty into a functional programming paradigm. Programming scala/finagle network services is much nicer IMO than coffeescript, ruby/em/fibers, raw netty. I can't wait until we release our finagle-based cassandra client. It's been really nice to work with. Here's some sample code: https://github.com/twitter/finagle/tree/master/finagle-example/src/main/scala/com/twitter/finagle/example https://github.com/twitter/finagle/tree/master/finagle-examp...
- jorgeortiz85 15y ago
  I've been using Finagle over the last couple of weeks to build a small little side-project. It's been almost a religious experience for me. This is how network programming should be done.
- grncdr 15y ago
  I clicked that link, because it seemed like it might be interesting. Nice to hear that it's still working for you after implementing something in it. Also, I may be in need of a nice async cassandra client very soon, any chance of an alpha preview? (e-mail in profile)
petenixey 15y ago
I wonder if the reason Twitter never sees any significant evolution in product is because they've weighed themselves down with too much iron cladding on their services or if they've had more time to iron clad their services because they never evolve the product. Neither may be related but for a large company with very little product they seem to produce astoundingly little.
- akronim 15y ago
  They may have very little functionality, but do it at scale. e.g. even searching tweets (which doesn't seem to work that great...) is a massive undertaking (not astoundingly little) that they're obviously still working at.
  - petenixey 15y ago
    That's a fair point and I don't trivialise the engineering however I am taken aback by the just how little product they produce. In a similar timeframe Facebook had created vast swathes of product and dealt with far greater user numbers than Twitter.
riprock 15y ago
As a JRuby fan, I have my fingers crossed that they will switch to JRuby and help it become "more mature." :)
russellperry 15y ago
"...static typing becomes a big convenience in enforcing coherency across all the systems. You can guarantee that your dataflow is more or less going to work, and focus on the functional aspects...But as we move into a light-weight Service Oriented Architecture model, static typing becomes a genuine productivity boon." A 'productivity boon'? I don't understand. At the risk of invoking the ancient static vs. dynamic religious war, this statement makes no sense to me. I get that if your codebase is tangled enough, and your unit test suite is inadequate to "guarantee that your dataflow is more or less going to work" that maybe rewriting significant portions of it in a type-safe system makes sense. I guess. But without specific code examples it's hard to say exactly what he's talking about. Myself, I've spent many years in both static and dynamic environments and I know exactly where I'm more productive -- and it's not wrestling complex parameterized types to the ground, pulling up abstract classes or interfaces, and/or configuring IOC containers, abstract factories and the like. I wonder though -- this has echoes of Alex Payne's criticisms a couple of years ago, which I think Obie Fernandez addressed pretty well: http://blog.obiefernandez.com/content/2009/04/my-reasoned-response-about-scala-at-twitter.html http://blog.obiefernandez.com/content/2009/04/my-reasoned-re...
- colin_jack 15y ago
  Yeah I found this bit odd too, definitely feels like it could be explored further.
- wpietri 15y ago
  A 'productivity boon'? I don't understand. At the risk of invoking the ancient static vs. dynamic religious war, this statement makes no sense to me. I don't know what they mean either, but my first guess has to do with company size and mobility of staff. I love dynamic languages most when I'm coding solo or with small teams. I don't need to express a lot of things to the computer when they're so clear in my head. But if I'm going to take over an adequately maintained code base, I'd rather it be in a static language, because more of the intent is explicit. At this point Twitter has a lot of engineers and is still growing, and they're in a very dynamic business. It's plausible to me that they get a global productivity boost even though static languages could feel like a productivity hit to each individual engineer.
rockarage 15y ago
This is similar to facebook going from php to c++ via hip hop. ( http://developers.facebook.com/blog/post/358/ http://developers.facebook.com/blog/post/358/ ) Very few companies will ever reach the scale of twitter and facebook. Building with ruby, php, python.. whatever your team is comfortable with is still ok.
cheez 15y ago
Hopefully they don't have any Java experts on board.

228 points by colin_jack 15y ago

I like this. I recall a startup some years back with a vision to build something complex. The founder sat down with everyone and said that we had an important decision to make: Do we build something that grows or do we build something that evolves? Neither way forward was presented as an obvious win. Building something that grows meant overengineering and trying to forecast the future. Building something that evolves meant constantly rewriting things, taking time away from new development. Twitter seems to have taken the "evolution" route, and it has served them well. I'm not at all sure they would be this successful if they'd tried to build on the JVM right from the start using the technologies available at the time.

div 15y ago
Building something that evolves feels like the smarter intuitive decision. It allows you to focus on features and pivoting over ceremony (boilerplatey / architecture stuff) and when you finally start hitting those performance limits, you have a luxury problem. It will also be more apparent where to invest time spent scaling your product, whereas if you try to optimize out of the gate, you may still hit unforeseen performance issues.
- kenjackson 15y ago
  It allows you to focus on features and pivoting over ceremony This always sounds nice, but I've found that ceremony almost never really reduces productivity by much. The real wins/losses almost never have to do with ceremony related features of language, but rather architectural and framework components. You build something that evolves by bringing in the smallest architecture and least fx components and then build. The language you use may dictate the fx components to some extent -- otherwise it's typically just to make developers feel happy (which is important, but really is just about morale more than anything truly inherent in the productivity of the language).
  - div 15y ago
    When I said ceremony, I did mean working around architectural up-front decisions. I was arguing in favor of 'growing something' as opposed to 'designing something' with regards to a hopefully growing userbase. Language choice may be of lesser importance as you say, but I do feel that some languages fit the growing strategy better while others have a more design up front feel to them.
nupark2 15y ago
I'm not at all sure they would be this successful if they'd tried to build on the JVM right from the start using the technologies available at the time. I'm not sure why you think that, or why you think Twiiter had a 'grow' vs 'evolve' strategy. Their approach seemed not nearly so self-aware, and resulted in serious, core deficiencies that required considerable time and expense to correct. From an external position, it very much did not look like a rational considered approach to solving he problem through 'evolution'.
- raganwald 15y ago
  I never suggested they were self-aware, only that they evolved. Therefore, I do not suggest that they decided anything. All evolution strategies involve serious, core deficiencies, that's the trade-off, that's why it isn't 100% obviously better to evolve than to grow. But I do suggest that evolution--however it came about at Twitter HQ--has worked out for them.
  - deleted 15y ago
    [deleted]
angryasian 15y ago
don't know why everyone assumes jvm language = over engineering. I agree that some jvm language code can be more robust, but the reality is most of us should live in our test suites. I would even say that Java test suites start and run faster than the standard rails tests ( without all the things to make it faster like spork). The frameworks argument of configuration vs convention I think is moot as most jvm language frameworks are taking a more conventional approach, but allowing for greater configuration now. on the other side the reality of a startup is you're going to push out so much code / features / product based on demand so fast that a lot of the things that you consider building for the future will be thrown out the window pretty fast. Just build it, see if it works, and move to the next thing.
eropple 15y ago
These aren't incompatible, though. Building something that "grows", even to very large scale, is not that difficult, and still leaves the door open to evolutionary improvements and changes to the fundamental architecture over time if they become necessary. The key is, when designing your growth architecture (and I don't consider it overengineering--a few simple decisions allow you to scale pretty well to a fairly large load), is to code to interface rather than code to implementation. If things evolve/change over time, you've already got half the work done for you because things don't spontaneously break on change.
- raganwald 15y ago
  It is incredibly easy to design for growth when looking in a rear view mirror. For example, it is easy NOW to see that Twitter's problem was scaling, and that they should have planned to grow their scalability while evolving their user experience. However, at the time it might have gone the other way: Perhaps their user base and volume might have increased at a slower place, leaving plenty of time to evolve their scalability, while their user experience might have required relentless change. In which case, they should have planned for their user experience to "scale" rather than their infrastructure.
  - eropple 15y ago
    True. That said, designing for scalability isn't that hard, and does not take that much more time than just bunging out whatever comes first to mind.
    - jshen 15y ago
      You're assuming that you know ahead of time what the bottlenecks are, and more importantly, which trade offs to make, because scaling is almost always about trade offs. A scalable solution is usually a more rigid solution as well, and that can be a problem for a startup that may need to try a few ideas before they hit on what works.
- dasil003 15y ago
  This is armchair architect thinking. Sure, coding to the interface makes scaling easier, it also makes maintenance, testing and everything else easier: it's just a good idea. But the devil is in the details. When you have to scale like Twitter you are getting into the territory where you have to do a lot of legwork just to figure out how to scale a particular interface. Performance and consistency issues start to gnaw away at the ideal interface you carefully designed back when you were 20k users and 10 reqs/s.
grok2 15y ago
Twitter's approach (atleast in the early stages) has been more of Darwinian evolution ("Shit, we have to do something about this or we're screwed") rather than planned evolution :-). It has probably served them well -- the growth they saw in usage meant they could without thinking too much whether something was worth it, evolve to more sophisticated solutions as time went by.
bborud 15y ago
The problem with Java is mainly that people think they have to use all the horribly complex, intrusive, badly written, underperforming frameworks that exist in the Java sphere. And then, of course, you will inevitably be screwed. To this day it takes real effort to convince Java programmers that a lot of "best practices" are anything but. For instance I still see Java programmers use frameworks that require them to maintain mountains of flimsy XML configs for things that they ought to have done in code. Both to get the benefit of letting the compiler do the work of weeding out boneheaded errors and to get rid of unnecessary flexibility that just leads to more work and more confusing, hard to read code. Java is a great language in which you can be very productive. But productivity means that you have to crack down on people who drag along J2EE-crap, or whatever crap was invented to make J2EE-crap slightly less crap. It also means you have to mentor people actively and harass anyone who even thinks of doing in XML what can be accomplished much faster in testable, compiled Java code.
- nerd_in_rage 15y ago
  Yep, I saw this same thing. Mountains of XML deployment descriptors / "component wiring" (some "Spring" garbage), needlessly verbose and usually unneeded mapper classes, useless interfaces (my rule is if you are only going to have ONE implementation save the interface for later after you know you got it right and need it), extra layers that do nothing (except "map", of course), etc. kill me. Makes me want to kill myself.
  - bluelu 15y ago
    That's one of the reasons I left my first and last employer. Tons of compnontent wiring written in xml, tons of configuration directives, but the end only the defaults were used anyway. Impossible to do updates because everything could break.
  - GrooveStomp 15y ago
    This exactly describes my very brief experience working with Java professionally. Back in university it was no big deal - the differences between C++ and Java weren't that big. In a production environment, though; ... XML, XML, XML! All the XML-wrapper tools for Java are supposedly the reason why there is so much XML config stuff going on - but the result is that you end up writing soo much code in XML instead of the actual language that you're supposed to be using. It's like somebody looked quickly at the MVC pattern, got it all wrong, and now Java has very tight coupling between XML and Java code for any production environment that hopes to leverage existing tools.
6ren 15y ago
By "something that grows", I think you mean "something that scales". It confused me, because Brooks talks about "growing' software (as opposed to building/planning it), which is I think what you mean by "something that evolves". FWIW, I think the standard wisdom today is to evolve (your term) software (e.g. agile ideas of YAGNI, DTSTTCPW), and only to scale/plan it if you have a very clear idea of what you're doing (e.g. frozen specs, which exist in some government/military domains; long-term standards; mathematics) - and you also know how to do it, having done it a few times before.
amurmann 15y ago
I don't think Twitter would have existed, if they had had to take the "grow"/plan for scale path. As I understand, Twitter originally was little tool, that the founder just wanted to have for them selves. If they had thought, they would have to take months to implement it in a scalable version right away, they might not have done it. I also think it's a good idea to throw a bunch of small, simple stuff on the wall and see what sticks. If everything is planned to be a big engineering effort, made to scale to millions of users, you spend a lot of time building stuff nobody will care about. That's why frameworks like Rails are great. They probably aren't perfect for your use case, esp. once you want to start scaling to crazy user numbers (although high user number will probably work very well), but they help you get a useable version out of the door crazy fast, compared to an all custom implementation.

clintjhill 15y ago

In my opinion, the best part of their infrastructure is their willingness to specialize in languages within each stack. It isn't simply "Java everywhere". "To allow developers to choose the best language for the job, Twitter has invested a lot of effort in writing internal frameworks which encapsulate common concerns." The single best investment companies can make is to allow developers to choose their specialty, and their language. Otherwise you have a huge overhead of skill set mismatch. And your talent pool can be bigger if you're open to more than one language. It's a refreshing view.

badmash69 15y ago
Twitter can afford the to have multiple languages in their stack but most start-ups simple don't have that luxury. If you have a early stage start up with about 10 developers and you need to quickly build a set of server side component or services for your product, would you want to use multiple languages to develop that or use one programming language ? Having developers choose best languages for the job would be disastrous if you end up with your stack written in 5 different languages. I would not discount the importance keeping codebase easy to manage.
- shazow 15y ago
  It depends on how much effort it would take to rewrite any of said components. In my personal experience, for reasonably isolated and small components, 80% of time spent creating a component of code is spent on making decisions. By the end of it, when all the decisions have been made, I could probably rewrite the entire thing from scratch in 20% of the time—and it would likely be better in every way. It would seem that the first iteration is largely prototyping. If you can save a significant multiplier of time by choosing a different language than the rest of the stack for the prototype and potentially rewrite it later if necessary, why not? Perhaps by the end of it all, you'd break even on time but end up with multiple implementations and better code.
- div 15y ago
  I think it's important to look at how twitter got there though. They'd been using just Rails for a long time before introducing Scala. At that point, it's fair to say they weren't a 10 dev startup anymore. This, imo, is part of having developers choose the best language for the job. They chose ruby to get their mvp out quickly and be able to grow their userbase. After that, they pinpointed trouble areas in their architecture and made pragmatic choices in fixing up those areas. While I suppose it's possible to "let developers pick the best tool for the job" and end up with 5 different languages in your stack, most good developers are likely to steer clear of such an endgame, unless the trade-offs are very, very clear.

sehugg 15y ago

Netty is a godsend. Java threads are extremely memory-hungry, so async I/O is a must for handling many connections. We routinely handle 200K simultaneous connections on our push servers without breaking a sweat.

fictorial 15y ago
Mind if I ask what the specs are for those servers?
- sehugg 15y ago
  AWS m1.large. We don't use nearly the entire CPU or memory footprint, but m1.small just is a wee too tiny and there's nothing in between.

lemming 15y ago

This sounds slightly misleading to me: The primary driver is honestly encapsulation, so we can iterate faster as a company. Having a single, monolithic application codebase is not amenable to quick movement on a per-team basis. So when we decide to encapsulate something, then because of our performance concerns, its better to rewrite it in the JVM for most systems, than to write a new Ruby system. It sounds from that like their primary driver for using the JVM is actually performance, but that they only decide to rewrite components when encapsulation drives them to do so. I can't see how the JVM provides any encapsulation benefits over Ruby for new systems.

msie 15y ago

Holy name-soup, Batman! Stuff that was new to me: Gizzard, Finagle, Blender, Netty and Earlybird.

jrockway 15y ago

jrockway's law: add enough developers to a project and it eventually becomes Java.

aristidb 15y ago
Except it seems to be Scala here?
- aphexairlines 15y ago
  "In the case of the search team, since they do a lot of work on Lucene, which is Java-based, they have a lot of experience in writing Java code. As such it is more convenient for them to work in Java than Scala or another language."

SoftwareMaven 15y ago

One of the important lessons, IMO, is that you should always be making pragmatic decisions that work today and into the near future. You can't predict how your system will change over time, so engineer in today's needs and let tomorrow take care of itself. Pragmatic failure inevitably leads to analysis paralysis. Just worry about getting stuff done. :)

gary4gar 15y ago

I wonder how the story had been, if twitter had opted for Python, instead of ruby. Do they still have these problems or in these aspects python is better than ruby?

hkarthik 15y ago
Given the huge growth of the site in the first few years and the relative inexperience of the founding team with problems of this magnitude, I expect the story would have been the same with Python, Java, .NET, etc. The simple Web Server -> ORM -> Relational database architecture that most modern web frameworks utilize can easily break down under tremendous concurrent load, especially if you attempt to run it on commodity hardware.
glassx 15y ago
If I recall correctly, Google switched their crawler from Python to C++ for performance and for library compatibility etc. And eBay moved the whole website from C++ to Java in 2002. IMO, the problem is that Twitter was Ruby's poster while both were still in ascension (might as well say they still are). The first migration they did was porting their message queue from Ruby to Scala. As mentioned on a TheGuardian article, they migrated because Starling was crashing too often and dropping tweets, so they had to stop the site and migrate manually. Also, another member claimed performance problems on some blog posts. However, Starling doesn't perform that well to Ruby standards either, so it's very difficult to say. I'd say the migration was for "social" reasons. The team is clearly waaaay more comfortable and enthusiastic about Scala, also they seem to prefer Static typing, so using it is the best choice indeed...

fizx 15y ago

One of the key things that was kinda skimmed over: You should check out http://github.com/twitter/finagle http://github.com/twitter/finagle. It takes async network programming with netty into a functional programming paradigm. Programming scala/finagle network services is much nicer IMO than coffeescript, ruby/em/fibers, raw netty. I can't wait until we release our finagle-based cassandra client. It's been really nice to work with. Here's some sample code: https://github.com/twitter/finagle/tree/master/finagle-example/src/main/scala/com/twitter/finagle/example https://github.com/twitter/finagle/tree/master/finagle-examp...

jorgeortiz85 15y ago
I've been using Finagle over the last couple of weeks to build a small little side-project. It's been almost a religious experience for me. This is how network programming should be done.
grncdr 15y ago
I clicked that link, because it seemed like it might be interesting. Nice to hear that it's still working for you after implementing something in it. Also, I may be in need of a nice async cassandra client very soon, any chance of an alpha preview? (e-mail in profile)

petenixey 15y ago

I wonder if the reason Twitter never sees any significant evolution in product is because they've weighed themselves down with too much iron cladding on their services or if they've had more time to iron clad their services because they never evolve the product. Neither may be related but for a large company with very little product they seem to produce astoundingly little.

akronim 15y ago
They may have very little functionality, but do it at scale. e.g. even searching tweets (which doesn't seem to work that great...) is a massive undertaking (not astoundingly little) that they're obviously still working at.
- petenixey 15y ago
  That's a fair point and I don't trivialise the engineering however I am taken aback by the just how little product they produce. In a similar timeframe Facebook had created vast swathes of product and dealt with far greater user numbers than Twitter.

riprock 15y ago

As a JRuby fan, I have my fingers crossed that they will switch to JRuby and help it become "more mature." :)

russellperry 15y ago

"...static typing becomes a big convenience in enforcing coherency across all the systems. You can guarantee that your dataflow is more or less going to work, and focus on the functional aspects...But as we move into a light-weight Service Oriented Architecture model, static typing becomes a genuine productivity boon." A 'productivity boon'? I don't understand. At the risk of invoking the ancient static vs. dynamic religious war, this statement makes no sense to me. I get that if your codebase is tangled enough, and your unit test suite is inadequate to "guarantee that your dataflow is more or less going to work" that maybe rewriting significant portions of it in a type-safe system makes sense. I guess. But without specific code examples it's hard to say exactly what he's talking about. Myself, I've spent many years in both static and dynamic environments and I know exactly where I'm more productive -- and it's not wrestling complex parameterized types to the ground, pulling up abstract classes or interfaces, and/or configuring IOC containers, abstract factories and the like. I wonder though -- this has echoes of Alex Payne's criticisms a couple of years ago, which I think Obie Fernandez addressed pretty well: http://blog.obiefernandez.com/content/2009/04/my-reasoned-response-about-scala-at-twitter.html http://blog.obiefernandez.com/content/2009/04/my-reasoned-re...

colin_jack 15y ago
Yeah I found this bit odd too, definitely feels like it could be explored further.
wpietri 15y ago
A 'productivity boon'? I don't understand. At the risk of invoking the ancient static vs. dynamic religious war, this statement makes no sense to me. I don't know what they mean either, but my first guess has to do with company size and mobility of staff. I love dynamic languages most when I'm coding solo or with small teams. I don't need to express a lot of things to the computer when they're so clear in my head. But if I'm going to take over an adequately maintained code base, I'd rather it be in a static language, because more of the intent is explicit. At this point Twitter has a lot of engineers and is still growing, and they're in a very dynamic business. It's plausible to me that they get a global productivity boost even though static languages could feel like a productivity hit to each individual engineer.

rockarage 15y ago

This is similar to facebook going from php to c++ via hip hop. ( http://developers.facebook.com/blog/post/358/ http://developers.facebook.com/blog/post/358/ ) Very few companies will ever reach the scale of twitter and facebook. Building with ruby, php, python.. whatever your team is comfortable with is still ok.

cheez 15y ago

Hopefully they don't have any Java experts on board.