15 ms·

The switch: Python to Node.js

207 points by pquerna 15y ago

espeed 15y ago
The depth of libraries available in the Python or JVM ecosystems is going to dwarf what's available for Node. Were you able to find everything you need for Node?
- jahewson 15y ago
  OTOH JavaScript is the most popular language on GitHub
  - clojurerocks 15y ago
    Part of this though is because web apps are for better or worse increasingly being almost nothing but javascript. And it becomes tedious moving back and forth between javascript and another language. Even if that language is really good. So i think people are just going with javascript for "everything" and rewriting their libraries in it.
  - nivertech 15y ago
    HTML is the most popular language on GitHub ;)
  - ergo14 15y ago
    And how does this relate to the statement above? How many of this is DOM manipulation code for browser? i'll make a wild guess of 90%?
  - insin 15y ago
    I always wonder what percentage of that consists of included dependencies.
    - ceejayoz 15y ago
      A good half of it may just be "we threw jQuery in here instead of making you download it".
      - nkassis 15y ago
        How does it calculate the size of the code base for each language? Most people would be using jquery as a minified oneliner. How can that inflate the numbers so much?
        ceejayoz 15y ago
        I'd imagine counting by characters would make a lot more sense than counting lines.
        Tobu 15y ago
        GitHub's source code language detection is over there: https://github.com/github/linguist/tree/master/lib/linguist https://github.com/github/linguist/tree/master/lib/linguist It doesn't measure line length, which could be a good way to detect minified JavaScript. It uses file extensions and byte size. There is some exclusion of commonly bundled libraries, but it's a rough heuristic on file paths.
    - gurraman 15y ago
      Most of our "django projects" are listed as being written in js on GitHub.
- substack 15y ago
  Node is catching up pretty fast, although its modules might not be as mature yet: 102257 perl http://www.cpan.org/ 33270 java http://search.maven.org/#stats 31921 ruby https://rubygems.org/ 18068 python http://pypi.python.org/pypi 5732 node http://search.npmjs.org/
  - espeed 15y ago
    What's the quality like of most of those Ruby gems? I keep seeing people refer to the gem number, but how sophisticated are they? Last time I looked there wasn't much along the line of SciPy, NumPy, NLTK, or Matplotlib.
    - malandrew 15y ago
      I would venture to guess that the lack of such libraries is more the result of community focus. The rails community (and by extension ruby) is generally more product focused, whereas the python community (and by extension django) are more science focused. I intentionally ordered the language and the framework differently for each because most people who use ruby/rails, got into that community for the sake of the framework, whereas most of the people who use python/django, got into that community for the sake of the language. Finally, python has a much greater adoption among in the academic world, so it's natural to expect that its community will have built more academic libraries like those you mentioned. Node.js and Javascript are seeing adoption among those interested in real-time applications, evented systems and applications that bridge the chasm between the client and the server, so it is natural to expect that more libraries focused on problems within those domains. One thing I like that I've seen from the node.js community is that the fact that they are developing with the same language on both the client and server, that they consider them one in the same. The only thing separating the two is latency and connection reliability. The latency issue is psychologically not much different than having a bias for doing things in memory and avoiding disk IO server-side. The lack of connection robustness is likely to evolve into solutions that mirror the problems that the erlang/OTP community has spent a lot of time solving. The ecosystem around languages are heavily influenced by the strengths and weaknesses of the communities that adopt them, and the community that adopts them is largely the result of which problems that language is well suited for either by language design or historical coincidence (e.g. Javascript is in the browser)
- clojurerocks 15y ago
  Of course it depends what libraries youre looking or. If youre looking for libraries for build the "basics" of a web app then there are plenty of libraries available. If yore doing other things though and looking for libraries that dont exist maybe its because youre using the wrong technology.
  - malandrew 15y ago
    Not necessarily. Sometimes the lack of libraries isn't solely because it is the wrong technology, but because it is the wrong community. Language and community are pretty intertwined. LISP for example is a great technology for a lot of things, but library-wise it suffers because within that community there is a tendency for people to "roll their own" solutions. Java has tons of libraries, but I'm not sure most people would say it's the right technology for a lot of things. It's generally the right technology for projects that involve lots of developers with mixed levels of ability (before some Java person down votes me for this, consider that Gosling himself made this point back in the mid-90s when he invented the language)
- pquerna 15y ago
  No, we were not able to find everything we needed out of the box. We had to write many things. There is a list of our dependencies at the bottom of this post: http://journal.paul.querna.org/articles/2011/12/17/technology-cloud-monitoring/ http://journal.paul.querna.org/articles/2011/12/17/technolog... Having said that, the only ones I consider moderately painful is when we need to deal with node-thrift, which on the Python or JVM is well maintained by existing groups.
baghali 15y ago
Did you evaluate EventMachine? If yes would you please share your findings.
- pimeys 15y ago
  I've used fibers-enabled EventMachine in one part of our application. It's pretty nice, because you don't have to do callbacks and the Ruby fibers are pretty nice actually. We're using it for handling jobs from Resque, which reads and writes from our database and do a ping to a 3rd party server. But. The amount of bugs in the em-enabled libraries is just amazing. For example em-activerecord just didn't work so well under huge load (dropping SQL-connections), so I had to write my own mini ORM for the workers. Also Resque blocks by default, so of course doing a non-blocking and non-forking version was a priority. The worst thing here are the error messages and backtraces. There are none (DeadFiberException for failing assert) and with hacks you can get out a bit more information if the job crashes. Now thinking this again, it would be a good idea to write it again with Node.js. Just because it's still more mature compared to EventMachine, and all of it's libraries are reactor core friendly by default.
  - mattgreenrocks 15y ago
    It's a shame that fibered Ruby feels like such a ghetto. I'd really like to see it be a viable concurrency choice, especially for apps that call a lot of third party services.
  - toisanji 15y ago
    I'm using em in production also and I am seeing quite few bugs too. I wanted to ask you some eventmachine related questions, how can I get ahold of you, I didn't see your info in your profile.
    - pimeys 15y ago
      My nick combined with Google's free email service will give you my contact.
  - sandGorgon 15y ago
lordlarm 15y ago
I've come up with the same conclusion regarding the Django ORM, as they have. An excerpt: I believe our single biggest mistake from a technical side was not reigning in our use Django ORM earlier in our applications life. We had Twisted services running huge Django ORM operations inside of the Twisted thread pool. It was very easy to get going, but as our services grew, not only was this not very performant, and it was extremely hard to debug Django has in many ways become the industry standard in regards to fast development - but it, as so many other ORMs, does not scale and produces horrible SQL-statements. Also for many applications I find Django to be a huge overkill - and in more recent times I've focused a lot more energy using microframeworks such as Flask and Bottle.
- deleted 15y ago
  [deleted]
- ergo14 15y ago
  "Django has in many ways become the industry standard in regards to fast development - but it, as so many other ORMs, does not scale and produces horrible SQL-statements." I disagree - many ORM's in python world? Try using sqlalchemy it will put django's ORM to shame, it will build fast good looking queries most of time, and if you have a rare case of something not good enough - you can easly tweak it. Can't say anything about Storm but saying that "industry standard" django ORM is OK because "others have same flaws" is just distorting reality to fit this thesis ;-) Ugly truth is django ORM is not ok and it bites many people when they are at the point where it's real pain to fix this - the more developers are aware of this the better.
  - jpk 15y ago
    I'm in the process of building an api in Django (GeoDjango, actually, with PostGIS). What are the gripes with the ORM and how could it be better? (Not that I'm disagreeing; I'm curious. I'm not experienced enough to know what makes an ORM okay or not.)
    - clojurerocks 15y ago
viraptor 15y ago
I'd really like to see that spreadsheet, since there are many python frameworks other than twisted available. There are eventlets/greenlets, there's gevent, there's diesel and monocle for minimal things, etc. Makes you wonder what actually made them change the whole technology... Than again, they could probably even reuse pyjamas to still write node.js code in python... OT: "We should of used SQLAlchemy. We should of built ..." I thought this expression was Bristol-specific. Apparently it's taking US too.
- tikhonj 15y ago
  I don't know if you missed the link or if it was added later, but it's here: https://docs.google.com/spreadsheet/ccc?key=0AvBGESHWxhk2dHJ2Q0lWRFF3dkxLZmFiMVVGRElQaEE#gid=0 https://docs.google.com/spreadsheet/ccc?key=0AvBGESHWxhk2dHJ... It seems they looked at Gevent, Go, C++ and the JVM. As usual, no love for Haskell :(, not that they're entirely to blame there.
  - reinhardt 15y ago
    Too bad that the comparison doesn't make much sense: Node/Twisted/Gevent (language specific frameworks) vs Go/C++ (languages) vs JVM (platform).
    - dextorious 15y ago
      It's not like (language specific framework + language) is that different from (language + built-in features + standard library).
  - viraptor 15y ago
    Thanks, either I'm going crazy, or this wasn't a link before...
  - lucian1900 15y ago
    It's very obviously skewed to get Node.JS as a winner.
- LeafStorm 15y ago
andrewcooke 15y ago
so what makes javascript callbacks easier to understand than twisted's deferreds? aren't they basically the same thing? and how does "we picked Node.js" logically follow from "It is obvious that the JVM platform is one of the best ways to build large distributed systems right now"?
- dextorious 15y ago
  There probably a middle step "we also wanted to play with the latest shiny technology", that's not mentioned.
  - clojurerocks 15y ago
    Id have to disagree with that myself. While node and mongodb are both "shiny" they are because they solve a problem that people have and offer solutions to problems that are very appealing. Including as i mentioned i blieve in another comment here that using node allows you to only have ot work with one language. Which is great. So its true that python has similiar technologies such as tornado and twisted. But if node works then why not just stick to one technology to have to maintain rather then multiple ones. Also node has a great community that is increasing. Many python libraries dont have this. Meaning really documentation. For example tornado has very little documentation.
    - dextorious 15y ago
      "Id have to disagree with that myself. While node and mongodb are both "shiny" they are because they solve a problem that people have and offer solutions to problems that are very appealing. " A lot of people use them for solving problems they DON'T have. Lots of people use both Mongo and Node.js at ill-fated attempts at premature scaling (similar to premature optimization).
      - clojurerocks 15y ago
        Again i will disagree with you on that. One of the benefits of node and why its becomingly increasingly popular is that it allows ease of development with the benefits of scale built in. So you dont have to worry about a steep learning curve for a simple application that only a few people will ever use. Or that youre framework is easy to develop with but what happens if it becomes popular. What then. This is even more the case with todays applicaitons which incrasingly are "api" based with the web framework portion being used just to serve the ajax code and for urls and whatnot. With node you get all of it together. Also if node used some other language that wasnt a standard this might also be true. But again it uses javascript or coffeescript which is becoming more and more used all f the time. Even in mobile applications and gaming now with canvas. Obviously if you know python or ruby or php extremely well and can build whatever you need quickly with that then theres no reason not to. I was new to python so it wasnt a major switch for me.
amix 15y ago
I blame them and not the technology stack they used. It's idiotic to mix blocking technologies with non-blocking technologies as blocking the IO loop is fatal for performance as you won't be able to process any events while the IO loop is blocked. And this person still has not learned from their mistake, they think that using SQLAlchemy would have solved their problem. If they wanted to make Twisted work they should have picked a database that can be used in a non-blocking way. The reason why I like node.js is that everything is built in a non-blocking way. Twisted has the same philosophy until people think they can get away using a blocking library inside the IO loop.
- dextorious 15y ago
  SO, basically you just learned about non-blocking and think node.js is a golden bullet. It's not. And it's exactly the same situation as anything they could have used in Python, as soon as they add actual code to get work done (as opposed to hand over work and wait for a callback). """The reason why I like node.js is that everything is built in a non-blocking way. Twisted has the same philosophy until people think they can get away using a blocking library inside the IO loop.""" That's not what they were doing, though. At the end of the non-blocking fiesta you want to actually return results to a waiting connection -- actual results, rendered templates etc. That will block and take time -- especially in node.js which is single threaded. So they just traded Python's threads/processes/lightweight threads for node.js processes.
  - taf2 15y ago
    Single threaded, but small enough processes that multiple processes in node makes up for it. http://nodejs.org/docs/latest/api/child_processes.html http://nodejs.org/docs/latest/api/child_processes.html build a process pool and queue jobs to background processes, it's the same arch, you'd use if you had threads so what's your point?
  - amix 15y ago
    No, I did not learn about non-blocking approach recently. I have built solutions in Java and in node that scaled to over 300.000 open connections pocessing billions of messages. Besides that you are missing my point, which basically is to not mix blocking with non-blocking code.
clojurerocks 15y ago
After developing a v1.0 of a project with django i ended up also moving to node.js as well as mongodb as opposed to postgresql. I found myself increasingly wrestling with django. I really like python and the django community and think django is good for certain types of applications. But at least for what im building it wasnt the right solution. Im also using node.js for another project which is also currently built with django. It has different requirements and i might use python on the backend whereas the other project will most likely just be node.js and mongodb.
peregrine 15y ago
I'm curious as to why Go wasn't a final contender? It started off with Go being a crowd favorite and ended with it nowhere. I assume it has to do with the young package repository but what's the reasoning you used?
- pquerna 15y ago
  Go's core packages are good. Go's not-in-core packages are a mixed bag. Lots of "one off" experiment projects. Which is fine, thats where it is in its life. Node.js was in the same place packages wise 18 months ago. Additionally, at the time we were looking at this, Go was still doing releases every 2 weeks -- this was months before the Go 1.0 plan was even announced: http://blog.golang.org/2011/10/preview-of-go-version-1.html http://blog.golang.org/2011/10/preview-of-go-version-1.html
  - peregrine 15y ago
    Thanks! I am playing around with Go and you are right the packages are a mixed bag. Not to mention using a search engine is basically impossible.
sneak 15y ago
Python is a programming language. Node.js is not. What you meant to say is that you switched from Python (presumably running on top of a Real Web Server) to javascript running on Node. The argument could be made that they are both interpreters, but please, get it right - you are writing javascript, which is a pretty shitty language in which to develop apps these days. The only reason we use it is because it has near-universal browser support - and then only for UIs. Also, not to parrot, but it's true - node is cancer. It's for fad chasers who have no idea how real servers manage to serve volume. Fortunately, most people inexperienced enough to choose node are inexperienced enough to never need to scale, so it'll work out okay for them - for implementation-specific values of 'okay'. :)
- karterk 15y ago
  > you are writing javascript, which is a pretty shitty language in which to develop apps these days. Woah. I do both Python and JS, and guess what - there are a fair number of people who hate Python's significant whitespace. And to them, it's also a "hacky" language (with all kinds of __foo__ stuff). So, shitty-ness is in the eyes of the beholder. > It's for fad chasers who have no idea how real servers manage to serve volume. There are real world apps like Trello, which is on Node, and are scaling well. Get your facts right, or stop trolling.
  - hello_moto 15y ago
    I'm not going to argue too much between JS and Python. But whitespace as debate topic over corner cases in JS language? really? And also Trello is just launched and suddenly it scales off miles away? really?
    - jQueryIsAwesome 15y ago
      The have pagerank 6 and they "just launch"? ... that is weird.
      - hello_moto 15y ago
hello_moto 15y ago
If they believe in Node.JS, good for them. I assume they have the right people to write good and modular javascript code. If you're a good front-end JS developers but never ever wrote a minimum reusable JS library, you've got a lot to learn before diving to JS. As long as the code is waaay more modular than this: https://github.com/marijnh/CodeMirror2/blob/master/lib/codemirror.js https://github.com/marijnh/CodeMirror2/blob/master/lib/codem... They should be alright in terms of adding new features. And if Node.JS hits scalability issue? well tough luck I suppose. I see that JVM is a very serious contender but as already noted: License issue. And let me add this: I'm betting my ass that there's the "I don't want to write Java" whisperers as well. Now now, don't BS me. I know how the so-called "engineers" think of Java these days. Sure, you could use Clojure, Scala, or JRuby. But seeing the competitor is Node.JS, pretty easy choice don't you think? My experience writing JS code for Node.JS based platform (ExpressJS, etc) is that there's a bigger chance that you'll write more code. Code to make the code more modular (sounds weird isn't it?). Code to make sure you work-around the warts of JS. Each line of code should be highly scrutinized due to JS warts. This thread is going to be a typical fun nerds-fight. I guarantee you. I'm going to grab a popcorn and watch nerds doing keyboard slamming.
wulczer 15y ago
Sounds like a weird reason to switch the language and framework used for your product. Blocking libraries block just the same in Node and in Twisted. Why not just stop using the Django ORM or more generally, stop writing code that blocks the main thread for long periods of time? The problem of the main thread waiting on a response from the database is easily solved without rewriting all your code, which incidentally will have the same issues, since there's no new magic there. If your software is a giant, ugly hairball because you wrote it in a hurry (I know I did this many times!) don't blame it on the language or framework. What I'm wondering is: Python/Twisted/Django rank equally or similarly on all the points from the list, except for Team Experience, which has to be much higher with the old stack than with a completely new technology and set of tools. This makes me think that there was a "play with shiny things" point after all.
- andrewvc 15y ago
  Well, I agree and I don't agree. I'm personally not a huge fan of node, and I prefer to write non-blocking code in Aleph/Netty (which is closer to twisted in terms of potential to block due to blocking java libs) myself. But, I can understand on a large team not wanting to deal with the potential for blocking libs. That said, if performance is that important, it should be part of your test suite, and should catch that.
- rictic 15y ago
  It's been a while since I've written nodejs in anger so I may be out of date, but as a rule libraries in Node don't block, they present async interfaces and do what they have to in order to keep from blocking. Python has no such conventions, making many libraries into dangerous minefields for the Twisted developer.
kyledrake 15y ago
If you want to use a reactor pattern and you don’t want to use JavaScript, you can use EventMachine on Ruby. Twisted is the same thing for Python. I’m sorry to hear you were having problems with Twisted, but EventMachine is absolutely rock solid. I have gotten a lot of work done with it. Now, let’s say you don’t want to do callbacks, but still take advantage of the Reactor pattern in EM. I wrote a patch for Sinatra that uses Fibers to wrap callbacks, hence allowing you to continue to program synchronously as you used to, and it’s called Sinatra Synchrony (http://kyledrake.net/sinatra-synchrony http://kyledrake.net/sinatra-synchrony). But in order to do non-blocking IO, you don’t need to use a Reactor pattern, because Ruby internally does not block on IO (the GIL does not affect this). And of course you get solutions like JRuby (built on the JVM), which provide for threads (and Rubinius 2 coming soon). My APIs written with any and all of these concurrency options can get thousands of hits per second. It’s quite scalable, but still provides all the rich libraries, reusable code, and testing support that makes my APIs high quality, which to me (and my users) is more important than making them fast. With this approach, my APIs are both high quality and fast. And I have never experienced a single crash of any worker processes in production. I would know, because my process monitor is smart enough to observe the workers in my deploy code and informs me, while adding another worker to replace the fallen one. Did I mention I can do zero-downtime hot deploys? Again, all implemented in Ruby. I’m not trying to brag here, but there’s this weird current of arguments that these existing solutions don’t scale. It’s a myth, I don’t know where it comes from, and I’m worried that if I don’t speak out against these arguments, I’m going to wake up in a giant pile of half-baked Javascript spaghetti written by people that didn’t even understand the real problem in the first place (blocking IO people, research it!).

207 points by pquerna 15y ago

The depth of libraries available in the Python or JVM ecosystems is going to dwarf what's available for Node. Were you able to find everything you need for Node?

jahewson 15y ago
OTOH JavaScript is the most popular language on GitHub
- clojurerocks 15y ago
  Part of this though is because web apps are for better or worse increasingly being almost nothing but javascript. And it becomes tedious moving back and forth between javascript and another language. Even if that language is really good. So i think people are just going with javascript for "everything" and rewriting their libraries in it.
- nivertech 15y ago
  HTML is the most popular language on GitHub ;)
- ergo14 15y ago
  And how does this relate to the statement above? How many of this is DOM manipulation code for browser? i'll make a wild guess of 90%?
- insin 15y ago
  I always wonder what percentage of that consists of included dependencies.
  - ceejayoz 15y ago
    A good half of it may just be "we threw jQuery in here instead of making you download it".
    - nkassis 15y ago
      How does it calculate the size of the code base for each language? Most people would be using jquery as a minified oneliner. How can that inflate the numbers so much?
      - ceejayoz 15y ago
        I'd imagine counting by characters would make a lot more sense than counting lines.
      - Tobu 15y ago
        GitHub's source code language detection is over there: https://github.com/github/linguist/tree/master/lib/linguist https://github.com/github/linguist/tree/master/lib/linguist It doesn't measure line length, which could be a good way to detect minified JavaScript. It uses file extensions and byte size. There is some exclusion of commonly bundled libraries, but it's a rough heuristic on file paths.
  - gurraman 15y ago
    Most of our "django projects" are listed as being written in js on GitHub.
substack 15y ago
Node is catching up pretty fast, although its modules might not be as mature yet: 102257 perl http://www.cpan.org/ 33270 java http://search.maven.org/#stats 31921 ruby https://rubygems.org/ 18068 python http://pypi.python.org/pypi 5732 node http://search.npmjs.org/
- espeed 15y ago
  What's the quality like of most of those Ruby gems? I keep seeing people refer to the gem number, but how sophisticated are they? Last time I looked there wasn't much along the line of SciPy, NumPy, NLTK, or Matplotlib.
  - malandrew 15y ago
    I would venture to guess that the lack of such libraries is more the result of community focus. The rails community (and by extension ruby) is generally more product focused, whereas the python community (and by extension django) are more science focused. I intentionally ordered the language and the framework differently for each because most people who use ruby/rails, got into that community for the sake of the framework, whereas most of the people who use python/django, got into that community for the sake of the language. Finally, python has a much greater adoption among in the academic world, so it's natural to expect that its community will have built more academic libraries like those you mentioned. Node.js and Javascript are seeing adoption among those interested in real-time applications, evented systems and applications that bridge the chasm between the client and the server, so it is natural to expect that more libraries focused on problems within those domains. One thing I like that I've seen from the node.js community is that the fact that they are developing with the same language on both the client and server, that they consider them one in the same. The only thing separating the two is latency and connection reliability. The latency issue is psychologically not much different than having a bias for doing things in memory and avoiding disk IO server-side. The lack of connection robustness is likely to evolve into solutions that mirror the problems that the erlang/OTP community has spent a lot of time solving. The ecosystem around languages are heavily influenced by the strengths and weaknesses of the communities that adopt them, and the community that adopts them is largely the result of which problems that language is well suited for either by language design or historical coincidence (e.g. Javascript is in the browser)
clojurerocks 15y ago
Of course it depends what libraries youre looking or. If youre looking for libraries for build the "basics" of a web app then there are plenty of libraries available. If yore doing other things though and looking for libraries that dont exist maybe its because youre using the wrong technology.
- malandrew 15y ago
  Not necessarily. Sometimes the lack of libraries isn't solely because it is the wrong technology, but because it is the wrong community. Language and community are pretty intertwined. LISP for example is a great technology for a lot of things, but library-wise it suffers because within that community there is a tendency for people to "roll their own" solutions. Java has tons of libraries, but I'm not sure most people would say it's the right technology for a lot of things. It's generally the right technology for projects that involve lots of developers with mixed levels of ability (before some Java person down votes me for this, consider that Gosling himself made this point back in the mid-90s when he invented the language)
pquerna 15y ago
No, we were not able to find everything we needed out of the box. We had to write many things. There is a list of our dependencies at the bottom of this post: http://journal.paul.querna.org/articles/2011/12/17/technology-cloud-monitoring/ http://journal.paul.querna.org/articles/2011/12/17/technolog... Having said that, the only ones I consider moderately painful is when we need to deal with node-thrift, which on the Python or JVM is well maintained by existing groups.

baghali 15y ago

Did you evaluate EventMachine? If yes would you please share your findings.

pimeys 15y ago
I've used fibers-enabled EventMachine in one part of our application. It's pretty nice, because you don't have to do callbacks and the Ruby fibers are pretty nice actually. We're using it for handling jobs from Resque, which reads and writes from our database and do a ping to a 3rd party server. But. The amount of bugs in the em-enabled libraries is just amazing. For example em-activerecord just didn't work so well under huge load (dropping SQL-connections), so I had to write my own mini ORM for the workers. Also Resque blocks by default, so of course doing a non-blocking and non-forking version was a priority. The worst thing here are the error messages and backtraces. There are none (DeadFiberException for failing assert) and with hacks you can get out a bit more information if the job crashes. Now thinking this again, it would be a good idea to write it again with Node.js. Just because it's still more mature compared to EventMachine, and all of it's libraries are reactor core friendly by default.
- mattgreenrocks 15y ago
  It's a shame that fibered Ruby feels like such a ghetto. I'd really like to see it be a viable concurrency choice, especially for apps that call a lot of third party services.
- toisanji 15y ago
  I'm using em in production also and I am seeing quite few bugs too. I wanted to ask you some eventmachine related questions, how can I get ahold of you, I didn't see your info in your profile.
  - pimeys 15y ago
    My nick combined with Google's free email service will give you my contact.
- sandGorgon 15y ago

lordlarm 15y ago

I've come up with the same conclusion regarding the Django ORM, as they have. An excerpt: I believe our single biggest mistake from a technical side was not reigning in our use Django ORM earlier in our applications life. We had Twisted services running huge Django ORM operations inside of the Twisted thread pool. It was very easy to get going, but as our services grew, not only was this not very performant, and it was extremely hard to debug Django has in many ways become the industry standard in regards to fast development - but it, as so many other ORMs, does not scale and produces horrible SQL-statements. Also for many applications I find Django to be a huge overkill - and in more recent times I've focused a lot more energy using microframeworks such as Flask and Bottle.

deleted 15y ago
[deleted]
ergo14 15y ago
"Django has in many ways become the industry standard in regards to fast development - but it, as so many other ORMs, does not scale and produces horrible SQL-statements." I disagree - many ORM's in python world? Try using sqlalchemy it will put django's ORM to shame, it will build fast good looking queries most of time, and if you have a rare case of something not good enough - you can easly tweak it. Can't say anything about Storm but saying that "industry standard" django ORM is OK because "others have same flaws" is just distorting reality to fit this thesis ;-) Ugly truth is django ORM is not ok and it bites many people when they are at the point where it's real pain to fix this - the more developers are aware of this the better.
- jpk 15y ago
  I'm in the process of building an api in Django (GeoDjango, actually, with PostGIS). What are the gripes with the ORM and how could it be better? (Not that I'm disagreeing; I'm curious. I'm not experienced enough to know what makes an ORM okay or not.)
  - clojurerocks 15y ago

viraptor 15y ago

I'd really like to see that spreadsheet, since there are many python frameworks other than twisted available. There are eventlets/greenlets, there's gevent, there's diesel and monocle for minimal things, etc. Makes you wonder what actually made them change the whole technology... Than again, they could probably even reuse pyjamas to still write node.js code in python... OT: "We should of used SQLAlchemy. We should of built ..." I thought this expression was Bristol-specific. Apparently it's taking US too.

tikhonj 15y ago
I don't know if you missed the link or if it was added later, but it's here: https://docs.google.com/spreadsheet/ccc?key=0AvBGESHWxhk2dHJ2Q0lWRFF3dkxLZmFiMVVGRElQaEE#gid=0 https://docs.google.com/spreadsheet/ccc?key=0AvBGESHWxhk2dHJ... It seems they looked at Gevent, Go, C++ and the JVM. As usual, no love for Haskell :(, not that they're entirely to blame there.
- reinhardt 15y ago
  Too bad that the comparison doesn't make much sense: Node/Twisted/Gevent (language specific frameworks) vs Go/C++ (languages) vs JVM (platform).
  - dextorious 15y ago
    It's not like (language specific framework + language) is that different from (language + built-in features + standard library).
- viraptor 15y ago
  Thanks, either I'm going crazy, or this wasn't a link before...
- lucian1900 15y ago
  It's very obviously skewed to get Node.JS as a winner.
LeafStorm 15y ago

andrewcooke 15y ago

so what makes javascript callbacks easier to understand than twisted's deferreds? aren't they basically the same thing? and how does "we picked Node.js" logically follow from "It is obvious that the JVM platform is one of the best ways to build large distributed systems right now"?

dextorious 15y ago
There probably a middle step "we also wanted to play with the latest shiny technology", that's not mentioned.
- clojurerocks 15y ago
  Id have to disagree with that myself. While node and mongodb are both "shiny" they are because they solve a problem that people have and offer solutions to problems that are very appealing. Including as i mentioned i blieve in another comment here that using node allows you to only have ot work with one language. Which is great. So its true that python has similiar technologies such as tornado and twisted. But if node works then why not just stick to one technology to have to maintain rather then multiple ones. Also node has a great community that is increasing. Many python libraries dont have this. Meaning really documentation. For example tornado has very little documentation.
  - dextorious 15y ago
    "Id have to disagree with that myself. While node and mongodb are both "shiny" they are because they solve a problem that people have and offer solutions to problems that are very appealing. " A lot of people use them for solving problems they DON'T have. Lots of people use both Mongo and Node.js at ill-fated attempts at premature scaling (similar to premature optimization).
    - clojurerocks 15y ago
      Again i will disagree with you on that. One of the benefits of node and why its becomingly increasingly popular is that it allows ease of development with the benefits of scale built in. So you dont have to worry about a steep learning curve for a simple application that only a few people will ever use. Or that youre framework is easy to develop with but what happens if it becomes popular. What then. This is even more the case with todays applicaitons which incrasingly are "api" based with the web framework portion being used just to serve the ajax code and for urls and whatnot. With node you get all of it together. Also if node used some other language that wasnt a standard this might also be true. But again it uses javascript or coffeescript which is becoming more and more used all f the time. Even in mobile applications and gaming now with canvas. Obviously if you know python or ruby or php extremely well and can build whatever you need quickly with that then theres no reason not to. I was new to python so it wasnt a major switch for me.

amix 15y ago

I blame them and not the technology stack they used. It's idiotic to mix blocking technologies with non-blocking technologies as blocking the IO loop is fatal for performance as you won't be able to process any events while the IO loop is blocked. And this person still has not learned from their mistake, they think that using SQLAlchemy would have solved their problem. If they wanted to make Twisted work they should have picked a database that can be used in a non-blocking way. The reason why I like node.js is that everything is built in a non-blocking way. Twisted has the same philosophy until people think they can get away using a blocking library inside the IO loop.

dextorious 15y ago
SO, basically you just learned about non-blocking and think node.js is a golden bullet. It's not. And it's exactly the same situation as anything they could have used in Python, as soon as they add actual code to get work done (as opposed to hand over work and wait for a callback). """The reason why I like node.js is that everything is built in a non-blocking way. Twisted has the same philosophy until people think they can get away using a blocking library inside the IO loop.""" That's not what they were doing, though. At the end of the non-blocking fiesta you want to actually return results to a waiting connection -- actual results, rendered templates etc. That will block and take time -- especially in node.js which is single threaded. So they just traded Python's threads/processes/lightweight threads for node.js processes.
- taf2 15y ago
  Single threaded, but small enough processes that multiple processes in node makes up for it. http://nodejs.org/docs/latest/api/child_processes.html http://nodejs.org/docs/latest/api/child_processes.html build a process pool and queue jobs to background processes, it's the same arch, you'd use if you had threads so what's your point?
- amix 15y ago
  No, I did not learn about non-blocking approach recently. I have built solutions in Java and in node that scaled to over 300.000 open connections pocessing billions of messages. Besides that you are missing my point, which basically is to not mix blocking with non-blocking code.

clojurerocks 15y ago

After developing a v1.0 of a project with django i ended up also moving to node.js as well as mongodb as opposed to postgresql. I found myself increasingly wrestling with django. I really like python and the django community and think django is good for certain types of applications. But at least for what im building it wasnt the right solution. Im also using node.js for another project which is also currently built with django. It has different requirements and i might use python on the backend whereas the other project will most likely just be node.js and mongodb.

peregrine 15y ago

I'm curious as to why Go wasn't a final contender? It started off with Go being a crowd favorite and ended with it nowhere. I assume it has to do with the young package repository but what's the reasoning you used?

pquerna 15y ago
Go's core packages are good. Go's not-in-core packages are a mixed bag. Lots of "one off" experiment projects. Which is fine, thats where it is in its life. Node.js was in the same place packages wise 18 months ago. Additionally, at the time we were looking at this, Go was still doing releases every 2 weeks -- this was months before the Go 1.0 plan was even announced: http://blog.golang.org/2011/10/preview-of-go-version-1.html http://blog.golang.org/2011/10/preview-of-go-version-1.html
- peregrine 15y ago
  Thanks! I am playing around with Go and you are right the packages are a mixed bag. Not to mention using a search engine is basically impossible.

sneak 15y ago

Python is a programming language. Node.js is not. What you meant to say is that you switched from Python (presumably running on top of a Real Web Server) to javascript running on Node. The argument could be made that they are both interpreters, but please, get it right - you are writing javascript, which is a pretty shitty language in which to develop apps these days. The only reason we use it is because it has near-universal browser support - and then only for UIs. Also, not to parrot, but it's true - node is cancer. It's for fad chasers who have no idea how real servers manage to serve volume. Fortunately, most people inexperienced enough to choose node are inexperienced enough to never need to scale, so it'll work out okay for them - for implementation-specific values of 'okay'. :)

karterk 15y ago
> you are writing javascript, which is a pretty shitty language in which to develop apps these days. Woah. I do both Python and JS, and guess what - there are a fair number of people who hate Python's significant whitespace. And to them, it's also a "hacky" language (with all kinds of __foo__ stuff). So, shitty-ness is in the eyes of the beholder. > It's for fad chasers who have no idea how real servers manage to serve volume. There are real world apps like Trello, which is on Node, and are scaling well. Get your facts right, or stop trolling.
- hello_moto 15y ago
  I'm not going to argue too much between JS and Python. But whitespace as debate topic over corner cases in JS language? really? And also Trello is just launched and suddenly it scales off miles away? really?
  - jQueryIsAwesome 15y ago
    The have pagerank 6 and they "just launch"? ... that is weird.
    - hello_moto 15y ago

hello_moto 15y ago

If they believe in Node.JS, good for them. I assume they have the right people to write good and modular javascript code. If you're a good front-end JS developers but never ever wrote a minimum reusable JS library, you've got a lot to learn before diving to JS. As long as the code is waaay more modular than this: https://github.com/marijnh/CodeMirror2/blob/master/lib/codemirror.js https://github.com/marijnh/CodeMirror2/blob/master/lib/codem... They should be alright in terms of adding new features. And if Node.JS hits scalability issue? well tough luck I suppose. I see that JVM is a very serious contender but as already noted: License issue. And let me add this: I'm betting my ass that there's the "I don't want to write Java" whisperers as well. Now now, don't BS me. I know how the so-called "engineers" think of Java these days. Sure, you could use Clojure, Scala, or JRuby. But seeing the competitor is Node.JS, pretty easy choice don't you think? My experience writing JS code for Node.JS based platform (ExpressJS, etc) is that there's a bigger chance that you'll write more code. Code to make the code more modular (sounds weird isn't it?). Code to make sure you work-around the warts of JS. Each line of code should be highly scrutinized due to JS warts. This thread is going to be a typical fun nerds-fight. I guarantee you. I'm going to grab a popcorn and watch nerds doing keyboard slamming.

wulczer 15y ago

Sounds like a weird reason to switch the language and framework used for your product. Blocking libraries block just the same in Node and in Twisted. Why not just stop using the Django ORM or more generally, stop writing code that blocks the main thread for long periods of time? The problem of the main thread waiting on a response from the database is easily solved without rewriting all your code, which incidentally will have the same issues, since there's no new magic there. If your software is a giant, ugly hairball because you wrote it in a hurry (I know I did this many times!) don't blame it on the language or framework. What I'm wondering is: Python/Twisted/Django rank equally or similarly on all the points from the list, except for Team Experience, which has to be much higher with the old stack than with a completely new technology and set of tools. This makes me think that there was a "play with shiny things" point after all.

andrewvc 15y ago
Well, I agree and I don't agree. I'm personally not a huge fan of node, and I prefer to write non-blocking code in Aleph/Netty (which is closer to twisted in terms of potential to block due to blocking java libs) myself. But, I can understand on a large team not wanting to deal with the potential for blocking libs. That said, if performance is that important, it should be part of your test suite, and should catch that.
rictic 15y ago
It's been a while since I've written nodejs in anger so I may be out of date, but as a rule libraries in Node don't block, they present async interfaces and do what they have to in order to keep from blocking. Python has no such conventions, making many libraries into dangerous minefields for the Twisted developer.

kyledrake 15y ago

If you want to use a reactor pattern and you don’t want to use JavaScript, you can use EventMachine on Ruby. Twisted is the same thing for Python. I’m sorry to hear you were having problems with Twisted, but EventMachine is absolutely rock solid. I have gotten a lot of work done with it. Now, let’s say you don’t want to do callbacks, but still take advantage of the Reactor pattern in EM. I wrote a patch for Sinatra that uses Fibers to wrap callbacks, hence allowing you to continue to program synchronously as you used to, and it’s called Sinatra Synchrony (http://kyledrake.net/sinatra-synchrony http://kyledrake.net/sinatra-synchrony). But in order to do non-blocking IO, you don’t need to use a Reactor pattern, because Ruby internally does not block on IO (the GIL does not affect this). And of course you get solutions like JRuby (built on the JVM), which provide for threads (and Rubinius 2 coming soon). My APIs written with any and all of these concurrency options can get thousands of hits per second. It’s quite scalable, but still provides all the rich libraries, reusable code, and testing support that makes my APIs high quality, which to me (and my users) is more important than making them fast. With this approach, my APIs are both high quality and fast. And I have never experienced a single crash of any worker processes in production. I would know, because my process monitor is smart enough to observe the workers in my deploy code and informs me, while adding another worker to replace the fallen one. Did I mention I can do zero-downtime hot deploys? Again, all implemented in Ruby. I’m not trying to brag here, but there’s this weird current of arguments that these existing solutions don’t scale. It’s a myth, I don’t know where it comes from, and I’m worried that if I don’t speak out against these arguments, I’m going to wake up in a giant pile of half-baked Javascript spaghetti written by people that didn’t even understand the real problem in the first place (blocking IO people, research it!).