6 ms·

Finagle works fine. Thank you. Love it. But it does not take off. Why? What all the JVM Node.js clones are missing and what Node.js sets apart are async librar

Finagle works fine. Thank you. Love it. But it does not take off. Why? What all the JVM Node.js clones are missing and what Node.js sets apart are async libraries. There is no async (MySQL) JDBC driver for starters. If your IO drivers are not async, your async container is not very useful in real life.

kyleburton 14y ago
This. The biggest difference is in the two cultures: blocking is anathema to the Node.js community - they will literally reject libraries or code that blocks because it destroys the entire model; the JVM community does not value non-blocking code - most of the core (JDBC, Networking in general, File system operations) is all written in a blocking style - the JVM community accepts this with the implicit assumption that threads will help assuage those issues. Python, Ruby and Perl all have the same cultural tolerance for blocking code. The Node.js community has a complete lack of tolerance for blocking code. I work with the JVM every day (Clojure) and wish it was different wrt the common use of non-blocking code, but it's going to be a long road to get there on the JVM. Kyle
- Locke1689 14y ago
  What are you talking about? libevent invented everything Node.js uses and Python's Twisted had and has everything Node.js could dream of. Node.js is just a reinvention of old technologies in Javascript. Edit: Removed flame about Javascript because I don't want to have this debate again.
  - kyleburton 14y ago
    I am talking about the cultures surrounding these languages and frameworks. Node's community rejects blocking libraries. Java's does not. I've used the non-blocking frameworks in Perl (POE and my own), C (select, and some of the poll variants), Ruby (event machine) and they are fine if you can avoid blocking libraries -- in these communities it is generally acceptable to write blocking libraries. I don't see it as a technical hurdle, I see it as a cultural one.
    - batista 14y ago
      I am talking about the cultures surrounding these languages and frameworks. Node's community rejects blocking libraries. Java's does not. So what? You get tons of Java libs to use, a majority of blocking ones and lots of non blocking on one hand, or you restrict yourself to the fewer non-blocking libs of varying quality available for Node. With Java you can also turn blocking libs to non blocking with a wrapper and threads, whereas with Node.js if it's blocking you're screwing, because the js engine is single-threaded.
      - ricardobeat 14y ago
        Node is nearing 10k published libraries. Is there a comprehensive site listing Java libraries? It's trivial to offload blocking operations to other processes in node too, it's just not the preferred option.
        moe 14y ago
        Node is nearing 10k published libraries. And ruby has 38237 gems. 99,9% of which are garbage. Library-count is a terrible metric.
        juiceandjuice 14y ago
        Have you ever programmed in Java? That's a strange question to ask if you have. http://mvnrepository.com/ http://mvnrepository.com/ Whatever libraries node has is a drop in the bucket compared to java.
        ricardobeat 14y ago
        I haven't, thats why I asked. Node is just 2 years old. You'll get 90% of garbage on any open package manager.
    - soc88 14y ago
      You make it sound like "rejecting blocking libraries" was some noble principle. Due to the severe limitations of JavaScript, there is no other choice.
- pron 14y ago
  Java Executors combined with Guava's ListenableFutures easily turn any blocking operation to an asynchronous one. Netty's entire model is asynchronous, and Java 7 now has AsynchronousChannels for IO which, I assume, Netty will make use of. All in all, the JVM has a much more solid and performant foundation than anything Node can provide. The whole difference will come down to a programming style preference. I am not entirely sure why Vert.x adopted the Node style rather than the proven servlet container, as I'm sure both styles provide comparable performance. I guess each may shine under different loads/usage patterns (my guess is that Vert.x/Node can squeeze more performance from a single thread, but servlets are more scalable).
  - Uchikoma 14y ago
    There is no async MySDQL JDBC driver. If you encapsulate it in an async layer, you need to keep a thread for the connection.
    - pron 14y ago
      Is that supposed to be bad? The programming style will be the same. If threads are done right, and the JVM can manage their affinity well (especially on NUMA architectures), it's best to use them and pass a relatively small amount of data between them, then they can provide much better performance than accessing the same large piece of RAM from many threads (that's what happens if you simply replicate a single event-loop thread with asynchronous IO).
    - tensor 14y ago
      Maybe I am missing something, but how can you possibly have an async SQL driver without threads like this? This sounds like a case of your Node.js database driver hiding the exact same behaviour described here within C code.
      - d503 14y ago
        PostgresSQL's libpq supports nonblocking asynchronous operation, and node-postgres takes advantage of that. http://www.postgresql.org/docs/9.1/static/libpq-async.html http://www.postgresql.org/docs/9.1/static/libpq-async.html https://github.com/brianc/node-postgres/blob/master/src/binding.cc https://github.com/brianc/node-postgres/blob/master/src/bind... (notice the Connect method on line 325 of binding.cc) At some level a client-server database driver isn't all that different from any other network client; you send a request over a socket and wait for a result. There's no reason you have to block while waiting. Moreover some databases (like Postgres) let you receive asynchronous notifications signaled by transactions on other connections; that's how trigger-based replication systems like Bucardo do their thing. http://www.postgresql.org/docs/9.1/static/sql-notify.html http://www.postgresql.org/docs/9.1/static/sql-notify.html
- Uchikoma 14y ago
  We run a major site on Java, had some thread trouble years ago in the very beginning, works very well now when tuned. Threads work. BUT: I assume people will move to backend services with REST and combining REST backend results to a page. This increases IO a lot and will kill your latency and default thread models when you do sync code. You'd need to use async IO, composeable futures to manage latency and thread count. And if you do async backend REST, why not do async JDBC etc. But there are no libraries.
- batista 14y ago
  >This. The biggest difference is in the two cultures: blocking is anathema to the Node.js community - they will literally reject libraries or code that blocks because it destroys the entire model Really? So they reject any kind of library that does anything except call a callback? Because everything else, from calculating 2+2 to creating a template blocks. And it doesn't matter when it happens, when it happens it blocks.
andrewvc 14y ago
So, I googled 'finagle java' and all the results were terrible. There was no concise description with a short example. Nothing led me to believe 'node.js' clone in anything I saw in the first few results. In fact, the first result was this: http://engineering.twitter.com/2011/08/finagle-protocol-agnostic-rpc-system.html http://engineering.twitter.com/2011/08/finagle-protocol-agno... "Protocol-agnostic RPC System" does not sound like a node clone at all.
- Uchikoma 14y ago
  (as a side note, what do you think REST PUT calls over HTTP on Node are?) Finagle is about composing async IO calls, e.g. here http://twitter.github.com/scala_school/finagle.html http://twitter.github.com/scala_school/finagle.html This works much better in Scala, as Java is not very good at composing Futures. It needs a lot of boiler plate code as it lacks noise free closures.
batista 14y ago
>What all the JVM Node.js clones are missing and what Node.js sets apart are async libraries. There is no async (MySQL) JDBC driver for starters. Would you call Node.js's MySQL drivers "production quality"?
- ricardobeat 14y ago
  Being non-existent and being production quality or not are very different things. BTW there are hundreds of node apps in production.
  - batista 14y ago
    >Being non-existent and being production quality or not are very different things. As far production is concerned, they might as well be the same thing. >BTW there are hundreds of node apps in production. Do they use some of Node's MySQL drivers though? For they (or their lack thereof) was the topic of this subthread.
    - ricardobeat 14y ago
      The topic was the non-existence of drivers in other platforms, you brought quality up. Node itself isn't 1.0 yet. I don't see how not having anything is better than having something in development. transloadit.com, from the module mantainers, has been using it for an year+. I don't have any magic insight into what modules sites use, but judging from the activity in their repo it's quite popular (~1000 watchers, 100+ forks): https://github.com/felixge/node-mysql https://github.com/felixge/node-mysql
sehugg 14y ago
Your async container still can be useful, since you can have worker threads executing blocking operations. This works fine as long as your JDBC queries are fast enough to feed the event loop without creating tons of threads. What async containers like Netty (and by proxy, Vert.x) solve is high-latency cases like long polling and file upload/download. Netty for one does have support for zero-copy file transfers: http://docs.jboss.org/netty/3.2/xref/org/jboss/netty/example/http/file/HttpStaticFileServerHandler.html#80 http://docs.jboss.org/netty/3.2/xref/org/jboss/netty/example... In my experience, these are a small percentage of cases so the majority of the app can use blocking I/O -- even long-polling code, as long as you are using the async framework for the client connection.
scubaguy 14y ago
Finagle looks like it is a bit lower level framework compared to vert.x. While Finagle aims to be a framework that supports services using multiple protocols, vert.x appears to be much more tailored for HTTP web services.
- gcampbell 14y ago
  finagle-http (https://github.com/twitter/finagle/tree/master/finagle-http https://github.com/twitter/finagle/tree/master/finagle-http) provides pretty much everything you need to build an HTTP web service.
  - scubaguy 14y ago
    But take a look at the Finagle example from Heroku and compare it to the example from vert.x. There's a lot of boilerplate in the Finagle version because Finagle is a general purpose async service framework, which was my entire point.
spullara 14y ago
Twitter is sponsoring a Summer of Code developer to create an async library for MySQL: http://engineering.twitter.com/2012/05/summer-of-code-at-twitter.html http://engineering.twitter.com/2012/05/summer-of-code-at-twi...
- Uchikoma 14y ago
  This can't be voted high enough.
tlrobinson 14y ago
There was no MySQL driver for Node when it was first created. You have to start somewhere.

kyleburton 14y ago

This. The biggest difference is in the two cultures: blocking is anathema to the Node.js community - they will literally reject libraries or code that blocks because it destroys the entire model; the JVM community does not value non-blocking code - most of the core (JDBC, Networking in general, File system operations) is all written in a blocking style - the JVM community accepts this with the implicit assumption that threads will help assuage those issues. Python, Ruby and Perl all have the same cultural tolerance for blocking code. The Node.js community has a complete lack of tolerance for blocking code. I work with the JVM every day (Clojure) and wish it was different wrt the common use of non-blocking code, but it's going to be a long road to get there on the JVM. Kyle

Locke1689 14y ago
What are you talking about? libevent invented everything Node.js uses and Python's Twisted had and has everything Node.js could dream of. Node.js is just a reinvention of old technologies in Javascript. Edit: Removed flame about Javascript because I don't want to have this debate again.
- kyleburton 14y ago
  I am talking about the cultures surrounding these languages and frameworks. Node's community rejects blocking libraries. Java's does not. I've used the non-blocking frameworks in Perl (POE and my own), C (select, and some of the poll variants), Ruby (event machine) and they are fine if you can avoid blocking libraries -- in these communities it is generally acceptable to write blocking libraries. I don't see it as a technical hurdle, I see it as a cultural one.
  - batista 14y ago
    I am talking about the cultures surrounding these languages and frameworks. Node's community rejects blocking libraries. Java's does not. So what? You get tons of Java libs to use, a majority of blocking ones and lots of non blocking on one hand, or you restrict yourself to the fewer non-blocking libs of varying quality available for Node. With Java you can also turn blocking libs to non blocking with a wrapper and threads, whereas with Node.js if it's blocking you're screwing, because the js engine is single-threaded.
    - ricardobeat 14y ago
      Node is nearing 10k published libraries. Is there a comprehensive site listing Java libraries? It's trivial to offload blocking operations to other processes in node too, it's just not the preferred option.
      - moe 14y ago
        Node is nearing 10k published libraries. And ruby has 38237 gems. 99,9% of which are garbage. Library-count is a terrible metric.
      - juiceandjuice 14y ago
        Have you ever programmed in Java? That's a strange question to ask if you have. http://mvnrepository.com/ http://mvnrepository.com/ Whatever libraries node has is a drop in the bucket compared to java.
        ricardobeat 14y ago
        I haven't, thats why I asked. Node is just 2 years old. You'll get 90% of garbage on any open package manager.
  - soc88 14y ago
    You make it sound like "rejecting blocking libraries" was some noble principle. Due to the severe limitations of JavaScript, there is no other choice.
pron 14y ago
Java Executors combined with Guava's ListenableFutures easily turn any blocking operation to an asynchronous one. Netty's entire model is asynchronous, and Java 7 now has AsynchronousChannels for IO which, I assume, Netty will make use of. All in all, the JVM has a much more solid and performant foundation than anything Node can provide. The whole difference will come down to a programming style preference. I am not entirely sure why Vert.x adopted the Node style rather than the proven servlet container, as I'm sure both styles provide comparable performance. I guess each may shine under different loads/usage patterns (my guess is that Vert.x/Node can squeeze more performance from a single thread, but servlets are more scalable).
- Uchikoma 14y ago
  There is no async MySDQL JDBC driver. If you encapsulate it in an async layer, you need to keep a thread for the connection.
  - pron 14y ago
    Is that supposed to be bad? The programming style will be the same. If threads are done right, and the JVM can manage their affinity well (especially on NUMA architectures), it's best to use them and pass a relatively small amount of data between them, then they can provide much better performance than accessing the same large piece of RAM from many threads (that's what happens if you simply replicate a single event-loop thread with asynchronous IO).
  - tensor 14y ago
    Maybe I am missing something, but how can you possibly have an async SQL driver without threads like this? This sounds like a case of your Node.js database driver hiding the exact same behaviour described here within C code.
    - d503 14y ago
      PostgresSQL's libpq supports nonblocking asynchronous operation, and node-postgres takes advantage of that. http://www.postgresql.org/docs/9.1/static/libpq-async.html http://www.postgresql.org/docs/9.1/static/libpq-async.html https://github.com/brianc/node-postgres/blob/master/src/binding.cc https://github.com/brianc/node-postgres/blob/master/src/bind... (notice the Connect method on line 325 of binding.cc) At some level a client-server database driver isn't all that different from any other network client; you send a request over a socket and wait for a result. There's no reason you have to block while waiting. Moreover some databases (like Postgres) let you receive asynchronous notifications signaled by transactions on other connections; that's how trigger-based replication systems like Bucardo do their thing. http://www.postgresql.org/docs/9.1/static/sql-notify.html http://www.postgresql.org/docs/9.1/static/sql-notify.html
Uchikoma 14y ago
We run a major site on Java, had some thread trouble years ago in the very beginning, works very well now when tuned. Threads work. BUT: I assume people will move to backend services with REST and combining REST backend results to a page. This increases IO a lot and will kill your latency and default thread models when you do sync code. You'd need to use async IO, composeable futures to manage latency and thread count. And if you do async backend REST, why not do async JDBC etc. But there are no libraries.
batista 14y ago
>This. The biggest difference is in the two cultures: blocking is anathema to the Node.js community - they will literally reject libraries or code that blocks because it destroys the entire model Really? So they reject any kind of library that does anything except call a callback? Because everything else, from calculating 2+2 to creating a template blocks. And it doesn't matter when it happens, when it happens it blocks.

andrewvc 14y ago

So, I googled 'finagle java' and all the results were terrible. There was no concise description with a short example. Nothing led me to believe 'node.js' clone in anything I saw in the first few results. In fact, the first result was this: http://engineering.twitter.com/2011/08/finagle-protocol-agnostic-rpc-system.html http://engineering.twitter.com/2011/08/finagle-protocol-agno... "Protocol-agnostic RPC System" does not sound like a node clone at all.

Uchikoma 14y ago
(as a side note, what do you think REST PUT calls over HTTP on Node are?) Finagle is about composing async IO calls, e.g. here http://twitter.github.com/scala_school/finagle.html http://twitter.github.com/scala_school/finagle.html This works much better in Scala, as Java is not very good at composing Futures. It needs a lot of boiler plate code as it lacks noise free closures.

batista 14y ago

>What all the JVM Node.js clones are missing and what Node.js sets apart are async libraries. There is no async (MySQL) JDBC driver for starters. Would you call Node.js's MySQL drivers "production quality"?

ricardobeat 14y ago
Being non-existent and being production quality or not are very different things. BTW there are hundreds of node apps in production.
- batista 14y ago
  >Being non-existent and being production quality or not are very different things. As far production is concerned, they might as well be the same thing. >BTW there are hundreds of node apps in production. Do they use some of Node's MySQL drivers though? For they (or their lack thereof) was the topic of this subthread.
  - ricardobeat 14y ago
    The topic was the non-existence of drivers in other platforms, you brought quality up. Node itself isn't 1.0 yet. I don't see how not having anything is better than having something in development. transloadit.com, from the module mantainers, has been using it for an year+. I don't have any magic insight into what modules sites use, but judging from the activity in their repo it's quite popular (~1000 watchers, 100+ forks): https://github.com/felixge/node-mysql https://github.com/felixge/node-mysql

sehugg 14y ago

Your async container still can be useful, since you can have worker threads executing blocking operations. This works fine as long as your JDBC queries are fast enough to feed the event loop without creating tons of threads. What async containers like Netty (and by proxy, Vert.x) solve is high-latency cases like long polling and file upload/download. Netty for one does have support for zero-copy file transfers: http://docs.jboss.org/netty/3.2/xref/org/jboss/netty/example/http/file/HttpStaticFileServerHandler.html#80 http://docs.jboss.org/netty/3.2/xref/org/jboss/netty/example... In my experience, these are a small percentage of cases so the majority of the app can use blocking I/O -- even long-polling code, as long as you are using the async framework for the client connection.

scubaguy 14y ago

Finagle looks like it is a bit lower level framework compared to vert.x. While Finagle aims to be a framework that supports services using multiple protocols, vert.x appears to be much more tailored for HTTP web services.

gcampbell 14y ago
finagle-http (https://github.com/twitter/finagle/tree/master/finagle-http https://github.com/twitter/finagle/tree/master/finagle-http) provides pretty much everything you need to build an HTTP web service.
- scubaguy 14y ago
  But take a look at the Finagle example from Heroku and compare it to the example from vert.x. There's a lot of boilerplate in the Finagle version because Finagle is a general purpose async service framework, which was my entire point.

spullara 14y ago

Twitter is sponsoring a Summer of Code developer to create an async library for MySQL: http://engineering.twitter.com/2012/05/summer-of-code-at-twitter.html http://engineering.twitter.com/2012/05/summer-of-code-at-twi...

Uchikoma 14y ago
This can't be voted high enough.

tlrobinson 14y ago

There was no MySQL driver for Node when it was first created. You have to start somewhere.