5 ms·

At 18m visitors, I don't think your choice of Django or Rails matters. DB tends to be the limit for even the normal applications, let alone something with that

by irahul 14y ago

At 18m visitors, I don't think your choice of Django or Rails matters. DB tends to be the limit for even the normal applications, let alone something with that kind of volume.

whalesalad 14y ago
There's a kabillion factors to consider though. With that many users, yes the exact implementation matters less and the ancillary systems and how they interact are more critical. But, building the platform, being able to iterate quickly and deliver robust functionality along the way is important as well. Not that Ruby hasn't, but Django and Python have really proved their worth with some of the heavy hitters recently. Instagram, Pinterest, FriendFeed, Disqus... It's an impressive roster.
- edwinnathaniel 14y ago
  ... and when we look at the um... "old" fart Java... their roster is even more impressive: LinkedIN, Google, NetFlix, Amazon. Just commenting ... :D
  - whalesalad 14y ago
    Touché ;) I mean really we're all just moving 1's and 0's around.
  - rayval 14y ago
    Regarding Google my understanding is that Java is only used in a few areas, and is far from the majority of their portfolio. Most their production systems are C++, while a lot of offline processing is Python. There is even a smattering of .NET (in Orkut). I think Java was used in Wave, but that is now defunct. Likely a few of these systems are being migrated to Go. Likewise with Amazon. My understanding, from chatting with some Amazon engineers at a conference, is that they have a pretty heterogeneous portfolio, even including some Perl and Oracle PL/SQL. Others who are better informed, please correct.
    - edwinnathaniel 14y ago
      GMail, Google Calendar, AdSense, AdWords, Android to name a few of the _actual_ heavy hitters from Google. Sure, not the majority, but they are all the bigger ones. http://google-opensource.blogspot.ca/2009/01/opengse-released.html http://google-opensource.blogspot.ca/2009/01/opengse-release... (I think Google Sites is also written in Java). Amazon is pretty much a big Java shop (don't forget the companies they acquired as well). Check out their jobs site.
    - lchengify 14y ago
      Java is pervasive at Google. Check out the Guava library and Google Guice for examples of how they adapted it for high-scalability web services. On top off edwinnathaniel's list, add Google Sites, Google Docs, and Mapreduce frameworks such as Flume Java.
    - swah 14y ago
      Transcript from Coders at Work interview with Joshua Block: Seibel: As a Java guy at Google, do you think it could be used more? Leaving aside the force of history and historical choices, if somehow you could wave a magic wand and replace all of C++ with Java, could that work? Bloch: Up to a point. Large parts of the system could be written that way, and over time, things are moving in that direction. But for the absolute core of the system - the inner loops of the index servers, for instance - very small gains in performance are worth an awful lot. When you have that many machines running the same piece of code, if you can make it even a few percent faster, then you've done something that has real benefits, financially and enviromentally. So there is some code that you want to write in assembly language, and what is C but glorified assembly language?
  - Styck 14y ago
    But you're not going to be able to build a Pintrest or a Instagram with 5 people in 2 years with Spring.
    - edwinnathaniel 14y ago
      You can. I feel sorry to those who can't :)
      - Styck 14y ago
        Of course you can, anything is possible, including convincing yourself Spring is as productive as Rails or Django :) Nothing against Java, it's the best tool for the job on many occasions but launching quickly and scaling fast are not it's forté.
        edwinnathaniel 14y ago
        When was the last time you use Spring? How about Spring MVC for front-end/controller and or JAX-RS for web API. I find Java to be much more modular and adaptable to different web paradigm: old school MVC, one page JS style (GWT or pure JS), asynchronous (Netty or Servlet 3.0), scalable web api that can spit both xml and json without extra coding (JAX-RS), component oriented (JSF). The integration between business logic and let say if you somehow need to use message queue are quite good via EJB 3.1 (very minimum coding) and JMS. Java took a beating a few years ago, fast forward to 2012, they have improved quite significantly. In the next two years, multi tenant architecture will be available in JavaEE 7 out of the box. Many people who keep complaining about Java typically are those who have wayy past experience.
        Fledeskum 14y ago
        The company I work for (which would qualify as an enterprise by any standards) decided to switch over to Django almost 3 years ago, rewriting our existing Java infrastructure on a project-to-project basis. The reasoning was primarily two-fold, we'd begun to have trouble recruiting good new talent interested in working with Spring/Java and we felt that for what we were doing, Java was not the strongest technology choice anymore (not like it was 10 years ago). Last year, we switched a 55k loc Spring MVC (with JAX-RS for the json api) over to Django (using tastypie for the api side and celery for offline tasks). The result ended up being little over 8k loc of Python. Closing a ticket has gone from taking on average around 5 days to a little over 4 hours. That was the third project we ported over, the others having had similar success. YMMV.
  - irahul 14y ago
    > ... and when we look at the um... "old" fart Java... their roster is even more impressive: LinkedIN, Google, NetFlix, Amazon. Sure it is. But I won't use Java for web stuff. I might use it for some background services, but for regular CRUD functionality, Java buys me nothing. I generally use Python(Flask)/Jinja2/Flask-SqlAlchemy. Flask is intuitive; Jinja2 is pleasant and fast; Flask-SQLAlchemy is concise with the option of exploring the raw power of SQLAlchemy. For regular use cases, nothing in Java beats this combination. I have looked at Play; it comes closer but it's still not there. Checkout this toy benchmark that was doing the rounds 2 months ago https://github.com/grahamking/Key-Value-Polyglot https://github.com/grahamking/Key-Value-Polyglot The naive python solution https://github.com/grahamking/Key-Value-Polyglot/blob/master/memg.py https://github.com/grahamking/Key-Value-Polyglot/blob/master... performs dog slow. Java has real threads, JIT; but guess what, the naive Java solution still is dog slow https://github.com/grahamking/Key-Value-Polyglot/blob/master/Memg.java https://github.com/grahamking/Key-Value-Polyglot/blob/master... My specialized python solutions are magnitudes of times faster than both naive Python and Java solutions: https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/memg_epoll.py https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/m... https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/memg_gevent_server.py https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/m... The naive solutions were taking around 20 seconds to do 500 writes followed by 500 reads. My changed solution does 5000 writes followed by 5000 reads in under 2 seconds. The 500 and 5000 aren't typos - the naive solutions was taking about 20 seconds for 1000 operations, whereas tailored solution was doing 10000 operations under 2 seconds. This is the discussion thread http://news.ycombinator.com/item?id=3733090 http://news.ycombinator.com/item?id=3733090 I get it that this is an IO bound problem, and naturally an epoll based solution will totally smoke a thread-per-request solution, but that's the point I am trying to make. If at the end of the day, intuitive solutions don't work and I have to do custom implementations, I am not getting much out of using Java. I might use it for something which is CPU bound about 70% of the time, but anything other than that, the pains far outweigh benefits.
    - edwinnathaniel 14y ago
      And the last time you use Java was....5 years ago? Better keep up with the latest frameworks and libraries. I know Rails and Django and I used cherrypy, cheetah, and sqlalchemy as well, and Java is on par with both modern frameworks if you know to choose the right frameworks. In fact, I do not have to deploy multiple processes one for Rails and one for sinatra or node.js. Just some app server or web container and I am ready to go. ... And you are asking me to check out a "toy benchmark"? I care more for the overall solutions from deployment, tools, etc. End to end baby...
tferris 14y ago
> DB tends to be the limit Yes, but Rails is another huge factor too (the ORM, Ruby itself, etc.)—you would get Rails scaled somehow to 18m visitor but it's damn hard.

by irahul 14y ago

At 18m visitors, I don't think your choice of Django or Rails matters. DB tends to be the limit for even the normal applications, let alone something with that kind of volume.

whalesalad 14y ago

There's a kabillion factors to consider though. With that many users, yes the exact implementation matters less and the ancillary systems and how they interact are more critical. But, building the platform, being able to iterate quickly and deliver robust functionality along the way is important as well. Not that Ruby hasn't, but Django and Python have really proved their worth with some of the heavy hitters recently. Instagram, Pinterest, FriendFeed, Disqus... It's an impressive roster.

edwinnathaniel 14y ago
... and when we look at the um... "old" fart Java... their roster is even more impressive: LinkedIN, Google, NetFlix, Amazon. Just commenting ... :D
- whalesalad 14y ago
  Touché ;) I mean really we're all just moving 1's and 0's around.
- rayval 14y ago
  Regarding Google my understanding is that Java is only used in a few areas, and is far from the majority of their portfolio. Most their production systems are C++, while a lot of offline processing is Python. There is even a smattering of .NET (in Orkut). I think Java was used in Wave, but that is now defunct. Likely a few of these systems are being migrated to Go. Likewise with Amazon. My understanding, from chatting with some Amazon engineers at a conference, is that they have a pretty heterogeneous portfolio, even including some Perl and Oracle PL/SQL. Others who are better informed, please correct.
  - edwinnathaniel 14y ago
    GMail, Google Calendar, AdSense, AdWords, Android to name a few of the _actual_ heavy hitters from Google. Sure, not the majority, but they are all the bigger ones. http://google-opensource.blogspot.ca/2009/01/opengse-released.html http://google-opensource.blogspot.ca/2009/01/opengse-release... (I think Google Sites is also written in Java). Amazon is pretty much a big Java shop (don't forget the companies they acquired as well). Check out their jobs site.
  - lchengify 14y ago
    Java is pervasive at Google. Check out the Guava library and Google Guice for examples of how they adapted it for high-scalability web services. On top off edwinnathaniel's list, add Google Sites, Google Docs, and Mapreduce frameworks such as Flume Java.
  - swah 14y ago
    Transcript from Coders at Work interview with Joshua Block: Seibel: As a Java guy at Google, do you think it could be used more? Leaving aside the force of history and historical choices, if somehow you could wave a magic wand and replace all of C++ with Java, could that work? Bloch: Up to a point. Large parts of the system could be written that way, and over time, things are moving in that direction. But for the absolute core of the system - the inner loops of the index servers, for instance - very small gains in performance are worth an awful lot. When you have that many machines running the same piece of code, if you can make it even a few percent faster, then you've done something that has real benefits, financially and enviromentally. So there is some code that you want to write in assembly language, and what is C but glorified assembly language?
- Styck 14y ago
  But you're not going to be able to build a Pintrest or a Instagram with 5 people in 2 years with Spring.
  - edwinnathaniel 14y ago
    You can. I feel sorry to those who can't :)
    - Styck 14y ago
      Of course you can, anything is possible, including convincing yourself Spring is as productive as Rails or Django :) Nothing against Java, it's the best tool for the job on many occasions but launching quickly and scaling fast are not it's forté.
      - edwinnathaniel 14y ago
        When was the last time you use Spring? How about Spring MVC for front-end/controller and or JAX-RS for web API. I find Java to be much more modular and adaptable to different web paradigm: old school MVC, one page JS style (GWT or pure JS), asynchronous (Netty or Servlet 3.0), scalable web api that can spit both xml and json without extra coding (JAX-RS), component oriented (JSF). The integration between business logic and let say if you somehow need to use message queue are quite good via EJB 3.1 (very minimum coding) and JMS. Java took a beating a few years ago, fast forward to 2012, they have improved quite significantly. In the next two years, multi tenant architecture will be available in JavaEE 7 out of the box. Many people who keep complaining about Java typically are those who have wayy past experience.
        Fledeskum 14y ago
        The company I work for (which would qualify as an enterprise by any standards) decided to switch over to Django almost 3 years ago, rewriting our existing Java infrastructure on a project-to-project basis. The reasoning was primarily two-fold, we'd begun to have trouble recruiting good new talent interested in working with Spring/Java and we felt that for what we were doing, Java was not the strongest technology choice anymore (not like it was 10 years ago). Last year, we switched a 55k loc Spring MVC (with JAX-RS for the json api) over to Django (using tastypie for the api side and celery for offline tasks). The result ended up being little over 8k loc of Python. Closing a ticket has gone from taking on average around 5 days to a little over 4 hours. That was the third project we ported over, the others having had similar success. YMMV.
- irahul 14y ago
  > ... and when we look at the um... "old" fart Java... their roster is even more impressive: LinkedIN, Google, NetFlix, Amazon. Sure it is. But I won't use Java for web stuff. I might use it for some background services, but for regular CRUD functionality, Java buys me nothing. I generally use Python(Flask)/Jinja2/Flask-SqlAlchemy. Flask is intuitive; Jinja2 is pleasant and fast; Flask-SQLAlchemy is concise with the option of exploring the raw power of SQLAlchemy. For regular use cases, nothing in Java beats this combination. I have looked at Play; it comes closer but it's still not there. Checkout this toy benchmark that was doing the rounds 2 months ago https://github.com/grahamking/Key-Value-Polyglot https://github.com/grahamking/Key-Value-Polyglot The naive python solution https://github.com/grahamking/Key-Value-Polyglot/blob/master/memg.py https://github.com/grahamking/Key-Value-Polyglot/blob/master... performs dog slow. Java has real threads, JIT; but guess what, the naive Java solution still is dog slow https://github.com/grahamking/Key-Value-Polyglot/blob/master/Memg.java https://github.com/grahamking/Key-Value-Polyglot/blob/master... My specialized python solutions are magnitudes of times faster than both naive Python and Java solutions: https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/memg_epoll.py https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/m... https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/memg_gevent_server.py https://github.com/rahulkmr/Key-Value-Polyglot/blob/master/m... The naive solutions were taking around 20 seconds to do 500 writes followed by 500 reads. My changed solution does 5000 writes followed by 5000 reads in under 2 seconds. The 500 and 5000 aren't typos - the naive solutions was taking about 20 seconds for 1000 operations, whereas tailored solution was doing 10000 operations under 2 seconds. This is the discussion thread http://news.ycombinator.com/item?id=3733090 http://news.ycombinator.com/item?id=3733090 I get it that this is an IO bound problem, and naturally an epoll based solution will totally smoke a thread-per-request solution, but that's the point I am trying to make. If at the end of the day, intuitive solutions don't work and I have to do custom implementations, I am not getting much out of using Java. I might use it for something which is CPU bound about 70% of the time, but anything other than that, the pains far outweigh benefits.
  - edwinnathaniel 14y ago
    And the last time you use Java was....5 years ago? Better keep up with the latest frameworks and libraries. I know Rails and Django and I used cherrypy, cheetah, and sqlalchemy as well, and Java is on par with both modern frameworks if you know to choose the right frameworks. In fact, I do not have to deploy multiple processes one for Rails and one for sinatra or node.js. Just some app server or web container and I am ready to go. ... And you are asking me to check out a "toy benchmark"? I care more for the overall solutions from deployment, tools, etc. End to end baby...

tferris 14y ago

> DB tends to be the limit Yes, but Rails is another huge factor too (the ORM, Ruby itself, etc.)—you would get Rails scaled somehow to 18m visitor but it's damn hard.