7 ms·

Twitter survives election after Ruby-to-JVM move

131 points by tawman 14y ago

eta_carinae 14y ago
The rumors of Java's death are, indeed, greatly exaggerated.
- esusatyo 14y ago
  They used Scala, not Java, on JVM.
  - trimbo 14y ago
    They use both. "Last week, we launched a replacement for our Ruby-on-Rails front-end: a Java server we call Blender. We are pleased to announce that this change has produced a 3x drop in search latencies and will enable us to rapidly iterate on search features in the coming months." http://engineering.twitter.com/2011/04/twitter-search-is-now-3x-faster_1656.html http://engineering.twitter.com/2011/04/twitter-search-is-now...
    - mnutt 14y ago
      At the same time they did that, they replaced MySQL with a real-time version of Lucene. Almost every one of these "we switched from A to B and got a 3x speed increase" articles conflates a lot of different variables. The first version you build when you have no traffic and product/market fit is the most important thing. Performance is a low priority. Eventually it hits a bottleneck and you begin to look at performance. Perhaps there is another language out there that is faster than the one you're using. At this point nobody says "let's do an exact code translation from A to B". As you rewrite, you keep a constant eye on performance. It often involves ripping out abstractions and moving closer to the metal. The system you end up with usually looks nothing like the one you started out with, nor should it since it is the product of all of your experience scaling up to that point.
  - mercurial 14y ago
    Actually, TFA states they use a mix of both.
- 46Bit 14y ago
  Java isn't sexy any more. You use Java because your problems are, and so it doesn't matter to anyone.
elchief 14y ago
JVM is a wonderful thing.
- camus 14y ago
  Github and the thousands of Rails powered websites have no problem scaling, so i'm pretty sure neither Ruby nor Rails are a problem.
  - thebluesky 14y ago
    With enough hardware you can make virtually anything scale, for a price...
    - camus 14y ago
      Well github is quite profitable , maybe that's twitter's problem... what service does twitter provides that people are willing to pay for ? None...
      - deleted 14y ago
        [deleted]
  - elchief 14y ago
    500 million users is more a little more demanding than 2 million users.
  - jacques_chester 14y ago
    Github is an easier problem: generate pages. Twitter has to deal with very high fanout and low cacheability.
Cowen 14y ago
> None of this will be welcome news to the army of fanatical Ruby developers who believe the language's syntax, its high developer productivity, and its overall philosophy far outweigh any performance disadvantage it might have compared to other languages. I'm a relative newbie to the Ruby world, but as best I can tell, the Ruby and Rails communities both accepted long ago that they weren't made for Twitter levels of traffic. Fact is, almost no one has Twitter levels of traffic besides Twitter. That's why Ruby and Rails are still so popular, because for ~99% of performance needs, they're more than capable and also extremely pleasant to work with.
- robryan 14y ago
  Yeah, that line feels like lazy reporting to me. Any involvement at any in the Ruby community over the past few years and they would know that very few Ruby developers are concerned about the performance issues past a certain scale. Seems like a no brainer that something that is a lot more strict and static will outperform something that has to deal with many more possibilities at runtime.
  - mercurial 14y ago
    > Yeah, that line feels like lazy reporting to me. Now, now, it's The Register. The tabloid style is part of its identity.
erichocean 14y ago
Would have been nice if the title was "Twitter survives election after Ruby-to-Scala move", since that's what actually happened. (They do use a bit of regular Java, but the majority of the core code is Scala, and it's Scala that should be getting the top line credit here, not Java.)
- taligent 14y ago
  No. It is the JVM that should be getting the credit.
  - otibom 14y ago
    Magnetised needles and steady hands.
  - erichocean 14y ago
    If that were the case, they could have just run JRuby and avoided rewriting a bunch of code. The truth is, Scala is actually helpful with the architecture they came up with AND its great that it runs on the JVM.
    - suresk 14y ago
      I'm not necessarily disagreeing with you (static typed languages are going to be faster on the JVM), but JRuby - partially with the help of invokedynamic, added in Java 7 - has come a long way performance-wise since Twitter began their rewrite. I wonder if they would have done anything different, had JRuby been where it is now?
      - spullara 14y ago
        JRuby, even when using a lot of Java based libraries, doesn't get you the performance boost you need — it is just incremental over Ruby and even then only with some code.
    - thebluesky 14y ago
      ...except that JRuby is 3-100x slower than Java: http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=jruby&lang2=java http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...
kevinconroy 14y ago
The novice programmer says: "My language is better than yours." The wise programmer says: "Use the right tool for the job." Please HN, we're wiser than trending stories would suggest.
- verroq 14y ago
  The wise programmer rewrites his stack in another language to scale right?
  - pirateking 14y ago
    The wise programmer does not write programs. Therefore, the wise programmer does not exist.
  - philwelch 14y ago
    If your software is Twitter, then you're constantly rewriting the stack anyway just to keep up, and switching to different languages, frameworks, technologies, and tools component-by-component is often the best way to do that.
- pg 14y ago
  Do you think it's impossible for one language to be better than another? If so, it's impossible for a language designer to do a good job, because it's impossible to improve a programming language.
  - deleted 14y ago
    [deleted]
  - fizx 14y ago
    It seems to me that he is arguing that languages do not have a total ordering. As such, novices will argue that a language is strictly better than another, when a broader perspective will reveal that this is only true for some use cases. The "no free lunch" theorem would indicate to me that there is no ultimate language, merely languages that are better for common (to you) use cases.
pyrotechnick 14y ago
This move marks the beginning of the end of a four year long effort for Twitter to rid themselves of Ruby. History will remember the entire Ruby industry as a series of compounding failures. The de facto formalisation and specifications. The black-box behaviour of core development. The broken-linked, un-versioned docs. The rampant cargo-cult mentality. The arcane exceptions. The meta-frameworks. Gem hell. 1.9/2.0 Rails.
- SoftwareMaven 14y ago
  I'm not a Ruby developer, but I still can't help thinking, "WTF". No technology stack is perfect, but I've yet to meet a stack that was pure evil. There may be some cargo-cult personalities in the Ruby community, but, if there is, it's only because there is value in it. An infinitesimal number of sites have to deal with Twitter's scale problems. The rest can work on getting crap done instead of worrying about Maseratti problems.
  - knowtheory 14y ago
    Allow me to be withering in my criticism of GP. Cargo-culting is what you accuse others of when they are learning and you do not like them. The appropriate way to deal with newbies who do not fully understand the consequences of the decisions they have made (perhaps even while they are advocating others join them), is to explain their rhetorical and technological shortcomings in a way that others can learn from. Accusing someone of "cargo-culting" is just unhelpful character assassination. Also, weak-sauce.
  - pyrotechnick 14y ago
    argumentum ad populum
    - deleted 14y ago
      [deleted]
  - thebluesky 14y ago
    > No technology stack is perfect, but I've yet to meet a stack that was pure evil. As someone who has seen Ruby cause major problems I'd have to say it's not too far off "pure evil"
timothya 14y ago
It's interesting that when it comes time to scale to serve enormous loads, you have to be willing to change fundamental parts of your stack which you've made a huge investment in. Ruby holds up well enough on the majority of the sites that use it, but when you have traffic the size that Twitter does, it's just not good enough. And it turns out that Java provides a nice tradeoff with high performance and high-level code. It's also interesting to see different companies approach this problem differently - Facebook famously recreated a way to run their PHP source code (by compiling it to C and then running it natively) instead of actually rewriting the source to a different language. I wonder if something similar would have been possible for Twitter, or if they weren't happy with how their existing code was structured in the first place which may have made the rewrite more attractive.
- adamsmith 14y ago
  Facebook also has many more features, which probably means more code, which probably means larger switching costs. Take anything: photos, privacy logic, news feed, etc., and Facebook has the more sophisticated feature set.
- jshen 14y ago
  It's not primarily about the size of traffic, but the ability or inability to cache. At work we serve a ton of traffic with MRI ruby and 3 small VMs. Most requests are served by varnish and never hit the ruby stack. Most people do a terrible job at caching (edit: I'm not saying that twitter is bad at caching).
  - kayoone 14y ago
    Twitter is hugely write intensive and needs realtime data, so their caching needs are probably vastly different from yours.
    - jshen 14y ago
      That was my point. Their caching needs are vastly different from almost everyone!
gary4gar 14y ago
Twitter's better performance is because of a better architecture than Java language. Further, the performance is from JVM, so you can use any of the JVM language including JRuby to get similar effects. This articles seems more like link-bait to me.
- jrockway 14y ago
  I agree. I think I could write a scalable Twitter in mostly Ruby. And I've never written a line of Ruby in my life.
  - deleted 14y ago
    [deleted]
  - scribu 14y ago
    Have you worked on other sites with a similar scale to Twitter? If not, how would you know?
    - jrockway 14y ago
      I work at Google :)
      - shanelja 14y ago
        Well, there's pretty much no response he can give to that then. Unless of course, you work in the marketing department or clean the buildings?
      - gary4gar 14y ago
        
      - scribu 14y ago
        Fair enough :)
      - boyter 14y ago
        Based on that I would like to hear how you would do so. Would make an interesting blog post or comment here.
cpt1138 14y ago
Ive heard that Java is DSL for turning large blocks of XML into stack traces. But it does it fast and reliably.
- anonymous 14y ago
  Hah. I'd put it more like: Java is a platform for building frameworks for implementing interpreters for XML-based DSLs.
malandrew 14y ago
While I can agree that scala/java/jvm deserve some credit for this (typing, concurrency support), I think stories like this do a disservice in that they underestimate the importance of building a system a second time with all the lessons learned the first time around. This is not unlike those stories where a developer writes a trivial program in a new language that is similar to a trivial program they wrote in another language 1+ years prior and compare the results. "I wrote program foo in 30 lines of code in language X, which is much better than the 120 lines of code it took me in language Y two years ago". It's natural for a developer to write a shorter program two years later since they have 2 more years under their belt. In fact I'd expect the same program to be much better two years later even if written in the same language.
- 46Bit 14y ago
  Very true. Twitter may have had a lot of scaling trouble, but that didn't necessarily mean moving everything onto the JVM was necessary to 'make it scale'. It was just sensible to pick a language that would prevent problems in future.
- ww520 14y ago
  But if the second design is the better design theory has any impact, Twitter would have done better to build the second design with Ruby.
nnq 14y ago
...I never get it why scaling is seen as such a "hot" problem and everybody seems inclined to make language and technology choices based on it ...I bet Twitter could have scaled well keeping Ruby and rewriting performance critical parts in C and maybe keep tweaking and tuning the Ruby interpreter, much along the way the Youtube team did with Python and C and now some Go I hear... I think their decision was more influenced by the experience of their team of programmers or by employable talent pool - they went the JVM|Java|Scala route because they had people with experience in high level languages and the JVM. If they happened to start with a team with "C hackers" background they would've gone the Ruby|C way and it would've worked as well. ...I think almost any language and technology can (be made to) scale, even to Twitter scale, at least if it's open-source and you have people with the required technology to hack around the internals and recode performance critical parts in lower level languages (basically C, C++ or Go nowadays)
- willvarfar 14y ago
  Did you skip the bit in the article where they mentioned the faster Ruby VM they built, before moving to JVM? http://engineering.twitter.com/2011/03/building-faster-ruby-garbage-collector.html http://engineering.twitter.com/2011/03/building-faster-ruby-...
  - nnq 14y ago
    no, I didn't skip it, and I can only guess that either they considered they wasted to many resources for such a "small" performance gain or didn't have people with enough skill in the right areas to keep going this direction. But no, I don't know all the details and steps of Twitter's switch and my experience is in the Python ecosystem but I think lots of things are similar...
    - Xylakant 14y ago
      Well, there are several problems with MRI-1.8.x/REE that come into play when you try to go for raw performance. The biggest problem is probably the "stop the world" GIL that occurs around the GC and AFAIK around all external C calls. The second biggest problem is that the lack of multi-core capabilities. Both can bite you pretty badly when you're aiming for raw performance of a single process. They're both not as bad when you're doing web-level stuff where it matters little whether you start one, two or 20 processes to handle your load.
SiVal 14y ago
What this says to me, a non-RoR user, is that it's harder to build websites with Java than with RoR, but if you pass a certain (very unusual) level of traffic, you'll wish you'd made the extra effort; otherwise, you'll be glad you didn't. (Extra effort isn't free.) Fine, but what I wish I understood was why RoR is so popular to begin with. The claim is always that Rails is so wonderful that it's worth learning Ruby just to be able to build web apps with Rails. Well, if so, then why isn't there a sequel to RoR: Python on Rails? "The benefit people care most about (Rails, not Ruby) using the language you already know (Python)." Since Python, Perl, and Ruby are so similar except in syntax, and people love the Rails part more than the Ruby part, and so many more people know and love Python, and new Python web app frameworks appear all the time...why isn't one of them Python on Rails after years of Pythonistas being forced to abandon Python and learn Ruby just to be able to use Rails? Is there some significant difference between Python and Ruby that explains this? What makes Rails so attractive and why isn't the same thing done with Python?
- hayksaakian 14y ago
  Its called Django.
- knowtheory 14y ago
  Python has a roughly equivalent MVC web framework called Django. Ruby and Python as languages are not radically different in kind, but their respective developer communities have had different focuses, and as a consequence the library of tools are not identical. Ruby on Rails became popular because people were dissatisfied with the way that web development was being done, and DHH is very good at marketing/propaganda. Ruby on Rails has had a substantive effect on the way that web development is done, and there have been numerous attempts in other languages (that didn't already have a framework like Django) to recreate the things that Ruby programmers enjoy with Rails (CakePHP and Grails come to mind immediately).
  - bryanh 14y ago
    This is spot on. To elaborate a bit, Rails tended to have a lot more generated code (the fabled "magic") that really sped up development of your standard CRUD apps. As far as I can tell, this was fairly novel in web development. Django didn't really have a focus on that, you spent a little (lot?) more time configuring. It's better now though. Of course, at the complete opposite of the spectrum, you have very minimalist stuff like Flask (Python) or Sinatra (Ruby) which is doesn't include a lot of bells and whistles. You'll have to import your own ORM, templating, etc...
bfac112 14y ago
This is a stupid article
neya 14y ago
This is why I insist people to work on a robust stack from ground up, so it will be less painful in the long run. This is not to say that you shouldn't build a prototype in Rails to get everything up and running as quickly as possible and worry about scaling later, but, it is just my opinion that if you invest the time and effort in working with a robust stack (for example Scala+Lift), your investment will pay you off in the long run. I've always admired rails for its flexibility and its enormous productivity boost, but all my serious applications are coded in Lift. I for one believe in "develop and forget", because I'd like to call myself a business guy than a programmer, though I'm deep into both. I like to spend more time expanding/marketing my business than worry about scaling it. But that's just my perspective. JVM is terribly under-estimated and I realized this when I got started with Lift+Scala. Scala is a very powerful language and requires a totally different mindset (=functional). And Lift is fairly complex for those wishing to get started with it and has poor documentation, despite being a 5-year old Framework. But once you understand it fully (somehow), there's no looking back. Lift provides so many things out of the box, especially related to security (unlike PLay!), so it's kind of a trade-off you have to choose between. Even if you compare all the benchmarks, most of the JVM-based languages like Scala outperform even something like GO! (Ok, that's not fair, since GO is fairly new) If you're interested in Scala, Coursera has a course on it by the creator of Scala himself (Martin Odersky).
- mnutt 14y ago
  > I like to spend more time expanding/marketing my business than worry about scaling it. If you're truly more interested in the business side, why don't you build it as quickly as possible in rails/django and then later if it warrants it you can hire some people to build it in lift/something else?
  - stock_toaster 14y ago
    In addition, startups tend to pivot a few times, and agility is arguably more important early on than raw performance (depending on product of course).
    - 14y ago
DigitalSea 14y ago
I wonder if Node.JS would meet the needs of Twitter? It's marketed as a language for real-time applications that require a lot of concurrent connections and whatnot, not nearly as complicated as Java/Scala either but would it be stable enough to power a site like Twitter? The inner nerd in me is going haywire with the possibilities. While most won't ever encounter the issues Twitter encountered using Ruby and Ruby on Rails these kinds of articles are very damaging for the Ruby language and Rails framework because even though I primarily still use PHP, Ruby & Rails are something I have a vested interest in as well and this will no doubt push potential newcomers away from the language.
- mytrile 14y ago
  Node.js is not a language. And if you think callback spaghetti code is better than clean and concise Scala code, something is wrong with you. And quote from Node.js creator: “I want programming computers to be like coloring with crayons and playing with duplo blocks. If my job was keeping Twitter up, of course I’d using a robust technology like the JVM.”
sicxu 14y ago
Scalability is an architecture issue, not a programming language issue. You can certainly build linearly scalable system in Ruby. It just costs more to scale a Ruby app vs. an equivalent Java app. In the end, the tradeoff is between engineering cost and operating cost. When you spend 100k to hire an engineer and your traffic can be handled by a few boxes. Engineering productivity is your primary focus. As your traffic grows, you need more and more servers, server efficiency becomes more and more important. At some point, the savings in server cost justifies the increased cost in engineering effort and you make a switch from Ruby to Java.
jakejake 14y ago
I would think that scaling the web servers to handle 15k requests per second would be relatively easy compared to scaling the database servers. I would have thought that you could just throw cheap hardware at that problem, whereas the database would be a considerably complex scaling issue.
- jacques_chester 14y ago
  Not to trivialise your question, because scaling to tens of thousands of database TPS transactions is difficult. But it is commercially solved. If you turn up with a fistful of money, Oracle, IBM, Sybase and a bunch of other companies would love to handle that for you.
  - jakejake 14y ago
    probably projecting my own problems onto that one, hehe.
camus 14y ago
And i say they should have use C++ directly, C++ is the only thing that scales ... what? oh sorry i meant Assembly. Nothing scales more than assembly, java is slow... why are we having that discussion again anyway ?
keymone 14y ago
i wonder if invokedynamic was introduced before they started moving their stack and staff to Java - would jruby save them in that case?

131 points by tawman 14y ago

eta_carinae 14y ago

The rumors of Java's death are, indeed, greatly exaggerated.

esusatyo 14y ago
They used Scala, not Java, on JVM.
- trimbo 14y ago
  They use both. "Last week, we launched a replacement for our Ruby-on-Rails front-end: a Java server we call Blender. We are pleased to announce that this change has produced a 3x drop in search latencies and will enable us to rapidly iterate on search features in the coming months." http://engineering.twitter.com/2011/04/twitter-search-is-now-3x-faster_1656.html http://engineering.twitter.com/2011/04/twitter-search-is-now...
  - mnutt 14y ago
    At the same time they did that, they replaced MySQL with a real-time version of Lucene. Almost every one of these "we switched from A to B and got a 3x speed increase" articles conflates a lot of different variables. The first version you build when you have no traffic and product/market fit is the most important thing. Performance is a low priority. Eventually it hits a bottleneck and you begin to look at performance. Perhaps there is another language out there that is faster than the one you're using. At this point nobody says "let's do an exact code translation from A to B". As you rewrite, you keep a constant eye on performance. It often involves ripping out abstractions and moving closer to the metal. The system you end up with usually looks nothing like the one you started out with, nor should it since it is the product of all of your experience scaling up to that point.
- mercurial 14y ago
  Actually, TFA states they use a mix of both.
46Bit 14y ago
Java isn't sexy any more. You use Java because your problems are, and so it doesn't matter to anyone.

elchief 14y ago

JVM is a wonderful thing.

camus 14y ago
Github and the thousands of Rails powered websites have no problem scaling, so i'm pretty sure neither Ruby nor Rails are a problem.
- thebluesky 14y ago
  With enough hardware you can make virtually anything scale, for a price...
  - camus 14y ago
    Well github is quite profitable , maybe that's twitter's problem... what service does twitter provides that people are willing to pay for ? None...
    - deleted 14y ago
      [deleted]
- elchief 14y ago
  500 million users is more a little more demanding than 2 million users.
- jacques_chester 14y ago
  Github is an easier problem: generate pages. Twitter has to deal with very high fanout and low cacheability.

Cowen 14y ago

> None of this will be welcome news to the army of fanatical Ruby developers who believe the language's syntax, its high developer productivity, and its overall philosophy far outweigh any performance disadvantage it might have compared to other languages. I'm a relative newbie to the Ruby world, but as best I can tell, the Ruby and Rails communities both accepted long ago that they weren't made for Twitter levels of traffic. Fact is, almost no one has Twitter levels of traffic besides Twitter. That's why Ruby and Rails are still so popular, because for ~99% of performance needs, they're more than capable and also extremely pleasant to work with.

robryan 14y ago
Yeah, that line feels like lazy reporting to me. Any involvement at any in the Ruby community over the past few years and they would know that very few Ruby developers are concerned about the performance issues past a certain scale. Seems like a no brainer that something that is a lot more strict and static will outperform something that has to deal with many more possibilities at runtime.
- mercurial 14y ago
  > Yeah, that line feels like lazy reporting to me. Now, now, it's The Register. The tabloid style is part of its identity.

erichocean 14y ago

Would have been nice if the title was "Twitter survives election after Ruby-to-Scala move", since that's what actually happened. (They do use a bit of regular Java, but the majority of the core code is Scala, and it's Scala that should be getting the top line credit here, not Java.)

taligent 14y ago
No. It is the JVM that should be getting the credit.
- otibom 14y ago
  Magnetised needles and steady hands.
- erichocean 14y ago
  If that were the case, they could have just run JRuby and avoided rewriting a bunch of code. The truth is, Scala is actually helpful with the architecture they came up with AND its great that it runs on the JVM.
  - suresk 14y ago
    I'm not necessarily disagreeing with you (static typed languages are going to be faster on the JVM), but JRuby - partially with the help of invokedynamic, added in Java 7 - has come a long way performance-wise since Twitter began their rewrite. I wonder if they would have done anything different, had JRuby been where it is now?
    - spullara 14y ago
      JRuby, even when using a lot of Java based libraries, doesn't get you the performance boost you need — it is just incremental over Ruby and even then only with some code.
  - thebluesky 14y ago
    ...except that JRuby is 3-100x slower than Java: http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=jruby&lang2=java http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...

kevinconroy 14y ago

The novice programmer says: "My language is better than yours." The wise programmer says: "Use the right tool for the job." Please HN, we're wiser than trending stories would suggest.

verroq 14y ago
The wise programmer rewrites his stack in another language to scale right?
- pirateking 14y ago
  The wise programmer does not write programs. Therefore, the wise programmer does not exist.
- philwelch 14y ago
  If your software is Twitter, then you're constantly rewriting the stack anyway just to keep up, and switching to different languages, frameworks, technologies, and tools component-by-component is often the best way to do that.
pg 14y ago
Do you think it's impossible for one language to be better than another? If so, it's impossible for a language designer to do a good job, because it's impossible to improve a programming language.
- deleted 14y ago
  [deleted]
- fizx 14y ago
  It seems to me that he is arguing that languages do not have a total ordering. As such, novices will argue that a language is strictly better than another, when a broader perspective will reveal that this is only true for some use cases. The "no free lunch" theorem would indicate to me that there is no ultimate language, merely languages that are better for common (to you) use cases.

pyrotechnick 14y ago

This move marks the beginning of the end of a four year long effort for Twitter to rid themselves of Ruby. History will remember the entire Ruby industry as a series of compounding failures. The de facto formalisation and specifications. The black-box behaviour of core development. The broken-linked, un-versioned docs. The rampant cargo-cult mentality. The arcane exceptions. The meta-frameworks. Gem hell. 1.9/2.0 Rails.

SoftwareMaven 14y ago
I'm not a Ruby developer, but I still can't help thinking, "WTF". No technology stack is perfect, but I've yet to meet a stack that was pure evil. There may be some cargo-cult personalities in the Ruby community, but, if there is, it's only because there is value in it. An infinitesimal number of sites have to deal with Twitter's scale problems. The rest can work on getting crap done instead of worrying about Maseratti problems.
- knowtheory 14y ago
  Allow me to be withering in my criticism of GP. Cargo-culting is what you accuse others of when they are learning and you do not like them. The appropriate way to deal with newbies who do not fully understand the consequences of the decisions they have made (perhaps even while they are advocating others join them), is to explain their rhetorical and technological shortcomings in a way that others can learn from. Accusing someone of "cargo-culting" is just unhelpful character assassination. Also, weak-sauce.
- pyrotechnick 14y ago
  argumentum ad populum
  - deleted 14y ago
    [deleted]
- thebluesky 14y ago
  > No technology stack is perfect, but I've yet to meet a stack that was pure evil. As someone who has seen Ruby cause major problems I'd have to say it's not too far off "pure evil"

timothya 14y ago

It's interesting that when it comes time to scale to serve enormous loads, you have to be willing to change fundamental parts of your stack which you've made a huge investment in. Ruby holds up well enough on the majority of the sites that use it, but when you have traffic the size that Twitter does, it's just not good enough. And it turns out that Java provides a nice tradeoff with high performance and high-level code. It's also interesting to see different companies approach this problem differently - Facebook famously recreated a way to run their PHP source code (by compiling it to C and then running it natively) instead of actually rewriting the source to a different language. I wonder if something similar would have been possible for Twitter, or if they weren't happy with how their existing code was structured in the first place which may have made the rewrite more attractive.

adamsmith 14y ago
Facebook also has many more features, which probably means more code, which probably means larger switching costs. Take anything: photos, privacy logic, news feed, etc., and Facebook has the more sophisticated feature set.
jshen 14y ago
It's not primarily about the size of traffic, but the ability or inability to cache. At work we serve a ton of traffic with MRI ruby and 3 small VMs. Most requests are served by varnish and never hit the ruby stack. Most people do a terrible job at caching (edit: I'm not saying that twitter is bad at caching).
- kayoone 14y ago
  Twitter is hugely write intensive and needs realtime data, so their caching needs are probably vastly different from yours.
  - jshen 14y ago
    That was my point. Their caching needs are vastly different from almost everyone!

gary4gar 14y ago

Twitter's better performance is because of a better architecture than Java language. Further, the performance is from JVM, so you can use any of the JVM language including JRuby to get similar effects. This articles seems more like link-bait to me.

jrockway 14y ago
I agree. I think I could write a scalable Twitter in mostly Ruby. And I've never written a line of Ruby in my life.
- deleted 14y ago
  [deleted]
- scribu 14y ago
  Have you worked on other sites with a similar scale to Twitter? If not, how would you know?
  - jrockway 14y ago
    I work at Google :)
    - shanelja 14y ago
      Well, there's pretty much no response he can give to that then. Unless of course, you work in the marketing department or clean the buildings?
    - gary4gar 14y ago
      
    - scribu 14y ago
      Fair enough :)
    - boyter 14y ago
      Based on that I would like to hear how you would do so. Would make an interesting blog post or comment here.

cpt1138 14y ago

Ive heard that Java is DSL for turning large blocks of XML into stack traces. But it does it fast and reliably.

anonymous 14y ago
Hah. I'd put it more like: Java is a platform for building frameworks for implementing interpreters for XML-based DSLs.

malandrew 14y ago

While I can agree that scala/java/jvm deserve some credit for this (typing, concurrency support), I think stories like this do a disservice in that they underestimate the importance of building a system a second time with all the lessons learned the first time around. This is not unlike those stories where a developer writes a trivial program in a new language that is similar to a trivial program they wrote in another language 1+ years prior and compare the results. "I wrote program foo in 30 lines of code in language X, which is much better than the 120 lines of code it took me in language Y two years ago". It's natural for a developer to write a shorter program two years later since they have 2 more years under their belt. In fact I'd expect the same program to be much better two years later even if written in the same language.

46Bit 14y ago
Very true. Twitter may have had a lot of scaling trouble, but that didn't necessarily mean moving everything onto the JVM was necessary to 'make it scale'. It was just sensible to pick a language that would prevent problems in future.
ww520 14y ago
But if the second design is the better design theory has any impact, Twitter would have done better to build the second design with Ruby.

nnq 14y ago

...I never get it why scaling is seen as such a "hot" problem and everybody seems inclined to make language and technology choices based on it ...I bet Twitter could have scaled well keeping Ruby and rewriting performance critical parts in C and maybe keep tweaking and tuning the Ruby interpreter, much along the way the Youtube team did with Python and C and now some Go I hear... I think their decision was more influenced by the experience of their team of programmers or by employable talent pool - they went the JVM|Java|Scala route because they had people with experience in high level languages and the JVM. If they happened to start with a team with "C hackers" background they would've gone the Ruby|C way and it would've worked as well. ...I think almost any language and technology can (be made to) scale, even to Twitter scale, at least if it's open-source and you have people with the required technology to hack around the internals and recode performance critical parts in lower level languages (basically C, C++ or Go nowadays)

willvarfar 14y ago
Did you skip the bit in the article where they mentioned the faster Ruby VM they built, before moving to JVM? http://engineering.twitter.com/2011/03/building-faster-ruby-garbage-collector.html http://engineering.twitter.com/2011/03/building-faster-ruby-...
- nnq 14y ago
  no, I didn't skip it, and I can only guess that either they considered they wasted to many resources for such a "small" performance gain or didn't have people with enough skill in the right areas to keep going this direction. But no, I don't know all the details and steps of Twitter's switch and my experience is in the Python ecosystem but I think lots of things are similar...
  - Xylakant 14y ago
    Well, there are several problems with MRI-1.8.x/REE that come into play when you try to go for raw performance. The biggest problem is probably the "stop the world" GIL that occurs around the GC and AFAIK around all external C calls. The second biggest problem is that the lack of multi-core capabilities. Both can bite you pretty badly when you're aiming for raw performance of a single process. They're both not as bad when you're doing web-level stuff where it matters little whether you start one, two or 20 processes to handle your load.

SiVal 14y ago

What this says to me, a non-RoR user, is that it's harder to build websites with Java than with RoR, but if you pass a certain (very unusual) level of traffic, you'll wish you'd made the extra effort; otherwise, you'll be glad you didn't. (Extra effort isn't free.) Fine, but what I wish I understood was why RoR is so popular to begin with. The claim is always that Rails is so wonderful that it's worth learning Ruby just to be able to build web apps with Rails. Well, if so, then why isn't there a sequel to RoR: Python on Rails? "The benefit people care most about (Rails, not Ruby) using the language you already know (Python)." Since Python, Perl, and Ruby are so similar except in syntax, and people love the Rails part more than the Ruby part, and so many more people know and love Python, and new Python web app frameworks appear all the time...why isn't one of them Python on Rails after years of Pythonistas being forced to abandon Python and learn Ruby just to be able to use Rails? Is there some significant difference between Python and Ruby that explains this? What makes Rails so attractive and why isn't the same thing done with Python?

hayksaakian 14y ago
Its called Django.
knowtheory 14y ago
Python has a roughly equivalent MVC web framework called Django. Ruby and Python as languages are not radically different in kind, but their respective developer communities have had different focuses, and as a consequence the library of tools are not identical. Ruby on Rails became popular because people were dissatisfied with the way that web development was being done, and DHH is very good at marketing/propaganda. Ruby on Rails has had a substantive effect on the way that web development is done, and there have been numerous attempts in other languages (that didn't already have a framework like Django) to recreate the things that Ruby programmers enjoy with Rails (CakePHP and Grails come to mind immediately).
- bryanh 14y ago
  This is spot on. To elaborate a bit, Rails tended to have a lot more generated code (the fabled "magic") that really sped up development of your standard CRUD apps. As far as I can tell, this was fairly novel in web development. Django didn't really have a focus on that, you spent a little (lot?) more time configuring. It's better now though. Of course, at the complete opposite of the spectrum, you have very minimalist stuff like Flask (Python) or Sinatra (Ruby) which is doesn't include a lot of bells and whistles. You'll have to import your own ORM, templating, etc...

bfac112 14y ago

This is a stupid article

neya 14y ago

This is why I insist people to work on a robust stack from ground up, so it will be less painful in the long run. This is not to say that you shouldn't build a prototype in Rails to get everything up and running as quickly as possible and worry about scaling later, but, it is just my opinion that if you invest the time and effort in working with a robust stack (for example Scala+Lift), your investment will pay you off in the long run. I've always admired rails for its flexibility and its enormous productivity boost, but all my serious applications are coded in Lift. I for one believe in "develop and forget", because I'd like to call myself a business guy than a programmer, though I'm deep into both. I like to spend more time expanding/marketing my business than worry about scaling it. But that's just my perspective. JVM is terribly under-estimated and I realized this when I got started with Lift+Scala. Scala is a very powerful language and requires a totally different mindset (=functional). And Lift is fairly complex for those wishing to get started with it and has poor documentation, despite being a 5-year old Framework. But once you understand it fully (somehow), there's no looking back. Lift provides so many things out of the box, especially related to security (unlike PLay!), so it's kind of a trade-off you have to choose between. Even if you compare all the benchmarks, most of the JVM-based languages like Scala outperform even something like GO! (Ok, that's not fair, since GO is fairly new) If you're interested in Scala, Coursera has a course on it by the creator of Scala himself (Martin Odersky).

mnutt 14y ago
> I like to spend more time expanding/marketing my business than worry about scaling it. If you're truly more interested in the business side, why don't you build it as quickly as possible in rails/django and then later if it warrants it you can hire some people to build it in lift/something else?
- stock_toaster 14y ago
  In addition, startups tend to pivot a few times, and agility is arguably more important early on than raw performance (depending on product of course).
  - 14y ago

DigitalSea 14y ago

I wonder if Node.JS would meet the needs of Twitter? It's marketed as a language for real-time applications that require a lot of concurrent connections and whatnot, not nearly as complicated as Java/Scala either but would it be stable enough to power a site like Twitter? The inner nerd in me is going haywire with the possibilities. While most won't ever encounter the issues Twitter encountered using Ruby and Ruby on Rails these kinds of articles are very damaging for the Ruby language and Rails framework because even though I primarily still use PHP, Ruby & Rails are something I have a vested interest in as well and this will no doubt push potential newcomers away from the language.

mytrile 14y ago
Node.js is not a language. And if you think callback spaghetti code is better than clean and concise Scala code, something is wrong with you. And quote from Node.js creator: “I want programming computers to be like coloring with crayons and playing with duplo blocks. If my job was keeping Twitter up, of course I’d using a robust technology like the JVM.”

sicxu 14y ago

Scalability is an architecture issue, not a programming language issue. You can certainly build linearly scalable system in Ruby. It just costs more to scale a Ruby app vs. an equivalent Java app. In the end, the tradeoff is between engineering cost and operating cost. When you spend 100k to hire an engineer and your traffic can be handled by a few boxes. Engineering productivity is your primary focus. As your traffic grows, you need more and more servers, server efficiency becomes more and more important. At some point, the savings in server cost justifies the increased cost in engineering effort and you make a switch from Ruby to Java.

jakejake 14y ago

I would think that scaling the web servers to handle 15k requests per second would be relatively easy compared to scaling the database servers. I would have thought that you could just throw cheap hardware at that problem, whereas the database would be a considerably complex scaling issue.

jacques_chester 14y ago
Not to trivialise your question, because scaling to tens of thousands of database TPS transactions is difficult. But it is commercially solved. If you turn up with a fistful of money, Oracle, IBM, Sybase and a bunch of other companies would love to handle that for you.
- jakejake 14y ago
  probably projecting my own problems onto that one, hehe.

camus 14y ago

And i say they should have use C++ directly, C++ is the only thing that scales ... what? oh sorry i meant Assembly. Nothing scales more than assembly, java is slow... why are we having that discussion again anyway ?

keymone 14y ago

i wonder if invokedynamic was introduced before they started moving their stack and staff to Java - would jruby save them in that case?