14 ms·

About PayPal's Node vs Java “fight”

puppetmaster3 13y ago
Summary: Most frameworks sux.
- romanovcode 13y ago
  Or PayPal programmers.
- zabraxias 13y ago
  Wrong. PayPal refuses to use open source frameworks because of an internal culture (and self preservation interest) of some very old "architects" who really only care about being in charge of the next internal framework. The best part is they later get to showcase how much better the new one is compared to the old framework, even though it's multiples slower than the slowest open-source counterpart.
  - cgh 13y ago
    Spring is open source. From the sounds of things, they replaced Spring with a closed in-house framework.
    - zabraxias 13y ago
      "Based on Spring" is not Spring. I worked there for a bit, they called it Sparta, and its appeal to the top tech brass was they could restrict which objects are available for use in the library they provided for you. The subtle message behind it was, we can hire anyone as long as we limit what they can do.
      - cgh 13y ago
        Thanks, I stand corrected. Sounds heinous.
      - mattgreenrocks 13y ago
        That's atrocious. One thing I'll never understand is how companies get so hungry for developers that they subconsciously admit to hiring amateurs: "well, Joe can't do too much damage!"
  - ricardobeat 13y ago
    Their node.js "framework" is really just a couple modules sitting on top of express, and they are all open-source.
auvi 13y ago
TL;DR from the post: """ My take-home for you, the persistent reader who has read all my ramblings in this post... Don’t build your app on top of a pile of crap in-house framework. """
- kcmarshall 13y ago
  Which suggests that take-home #0 should be: Don't build a pile of crap in-house framework. Sniffing that out before too late is the tricky part...
_random_ 13y ago
That is a refreshingly sober point of view. Blaming frameworks is easy. Speaking of technical debt is hard.
Xorlev 13y ago
Well-reasoned. I've always found framework overhead and especially tech debt to be the largest contributors to performance issues. Especially when said frameworks have lots of ways to do things and they're not all the best way to do it. Modern JVM development starts to look a lot like Sinatra -- you don't get a lot baked in, but you probably didn't need it anyways.
- pron 13y ago
  Yes. JAX-RS (the standard implemented by Jersey), which is part of the often maligned Java EE, is a (surprisingly?) well designed REST "microframework". Java EE in general has come a long way from the monster it used to be, and is becoming quite pleasant.
  - edwinnathaniel 13y ago
    You can use Servlet + Jersey. It's just one dependency away and minimal web.xml config (or spring-jersey...whichever). It's not necessarily tied to Java EE albeit Java EE app-server _must_ implement JAX-RS properly.
    - pron 13y ago
      Both Servlet and JAX-RS are pretty central pieces of Java EE. But Java EE is now really a collection of modules that you can pick and choose.
- curun1r 13y ago
  The biggest contributor to performance issues these days, from what I've seen, is synchronous code. The JVM can be blindingly fast at running your code, but when it hits a point where you make a database call or web services call that's implemented synchronously, it sits there, waiting, doing nothing. No amount of VM optimizations can increase of the speed of idling code. This is why Node apps often out-perform Java apps despite running on a VM that's significantly slower. Node is pervasively asynchronous...it's baked into the DNA of the platform and almost all libraries follow that philosophy. And as organizations move to a micro-services architecture where one external call can result in multiple internal calls between systems, the need for reactive, asynchronous programming becomes all the more glaring. That's where this critique, for me, fell down. It's spent too much time comparing the JVM to V8 and not enough time talking about how the applications being run were designed. Because CPUs are fast enough these days that any of the VMs will give you decent performance for workloads that aren't CPU-bound as long as you, the application designer, do the right thing. It's just that some environments, Node in particular, make it easier to do the right thing. This will get easier in Java with the introduction of lambdas in Java 8 and the inevitable asynchronous community frameworks and libraries, but right now it's still harder to write reactive applications in Java than it is in Node.
smoyer 13y ago
Management won't take the blame but ... who allowed the architects/engineers to build two systems that failed the same way?
- wpietri 13y ago
  I agree that management is likely a problem here, but disagree with the framing. If you're in a structure where management's job is to allow/deny, then I think the code is already doomed. Software development is work that can't be effectively micromanaged. In software, I think management's job is to support and challenge. All the people coding should be professionals, thoughtful and aware of issues like tech debt, second-system effects, reinventing the wheel, and so on. Management's job isn't to be more professional than the pros; it's to support the pros and make sure they're living up to (and sometimes advancing) their professional standards. I'm sure there are plenty of people who work with these frameworks that know that they're crap. Management didn't fail by allowing these to get built. They failed by creating power relationships that allowed the people who couldn't see the problems to control the building of the frameworks, and to suppress criticism and improvement.
sehrope 13y ago
Regarding the JVM performance: > Under minimum load the best page rendering time was 233ms Regarding the node.js performance: > Under minimum load the best page rendering time was 249ms What the heck are they doing that takes 250ms that could possibly be app code related? That's a lot of CPU usage if its actually doing CPU-bound work. Since these are web apps that's obviously not the case. The request time is going to be mostly waiting for external resources (DB, message queue, etc). Without knowing how much time is spent on those external requests these numbers are meaningless. Language choice is no where near as important as your DB-access patterns (and more generally DB performance, caching, etc). If every request you process involves serially accessing a 50-100ms external resource four or five times then it will always take 250ms (though you could increase parallelism of multiple requests if done right).
- cgh 13y ago
  The fact that both implementations are ridiculously slow tells me to look for commonalities. In this case, how much do you want to bet that there are some truly scary, gigantic and all-but-incomprehensible stored procedures underlying everything?
  - Xorlev 13y ago
    No bet. :)
- camus2 13y ago
  > Language choice is no where near as important as your DB-access patterns (and more generally DB performance, caching, etc). i'm pretty sure the paypal node app does not directly access the db , the node app talks to a private api.
  - Helianthus 13y ago
    That doesn't really change the point, it just wraps it with a layer of abstraction.
    - 13y ago
Joeri 13y ago
Maybe we should assume the best instead of the worst. We could assume that paypal has capable developers who are knowledgeable about both the jvm and node.js, and who have concluded that their specific featureset can best be built using node. That is not a value judgment of the jvm vs node in general, as their situation is theirs and theirs alone.
spullara 13y ago
Two poorly written apps compared with blame for each placed on the underlying virtual machine rather than the part between the chair and the keyboard. I'd much rather compare VMs and systems with something a little more structured and open like the TechEmpower benchmarks. They still have their flaws and simplifications, but at least the source for all apps is open and you can see and contribute to them.
srean 13y ago
This was a good read. Minor nitpick, given the author's stress of not comparing apples to oranges I think the motivating anecdote on protein folding struck a dissonant cord. A fair comparison would have been a profile guided optimization with comparably aggressive compile optimization flags. For these types of problems the profile characteristic is fairly static, so there is hardly ever a need to pay for the warm up time. Java can indeed be very fast, and in my opinion JVM is the one of the most optimized VMs that we have now. That said Java code beating a C or a C++ code on a CPU bound task raised an eyebrow. In my experience Java usually gets to the 80% speed of C++ quite easily but at a cost of 2.5 to 3.5 the memory. For these types of applications memory tends to be an expensive resource. For our number crunching server, 80% of its cost is sunk in the RAM.
- stephenconnolly 13y ago
  I debated including that example, but at the end the point I was making is that the JVM can be faster than native code in some cases... this case had static allocation of data and the calculations could benefit from MMX instructions when available, plus there was a logical unrolling that I think it was able to infer from code path analysis
  - rossjudson 13y ago
    And superword optimization, when possible...
- pron 13y ago
  My experience is similar: Java will get you 80% of super-optimized-compiled C++ speed for the cost of more memory (although you may choose to give Java less memory and pay in speed), and will get much closer to C++ speed with some work (off-heap memory, etc.). However, Java has the upper hand in two scenarios: 1) long-running applications that are developed by a large team – those usually make heavy use of virtual inheritance for the sake of good engineering, and the execution profile is not static, so the program can greatly benefit from JIT optimization; and 2) fairly complex multithreaded code – Java's incredibly useful and well-implemented blocking and lock-free data structures, as well as an incredible work-stealing scheduler (all expertly programmed by Doug Lea) make great use of a general purpose GC, plus Java usually gets them about 5 years before C++ (if C++ gets them at all).
abritishguy 13y ago
Damn, paypal really suck at making fast sites.
deleted 13y ago
[deleted]
tunesmith 13y ago
I wonder if a lot of headway could have been made simply by doing some constraint analysis on the Spring framework and figuring out how to improve the performance. I have a hard time believing that Spring itself was completely topped out. It might have been an architectural choice, or some bloat that could have been taken out incrementally. Instead they seem to be going the route of "the unicorns will solve it". The overall cost of that approach seems higher - finding unicorns is rare. Finding unicorns that are very good at all areas is even harder. I know there are a lot of unicorns on this board, but there sure aren't a lot of unicorn resumes on a hirer's desk. I think the value of Spring has always been not that it's super-high-performance out of the box, but that it does a lot for you, and that as you learn more and more of it, you can develop solutions fairly quickly. It's built to make it easier on the developer, not the servers - but it is still easy to improve performance by focusing on constraints. I think it's easy to blame what's between the keyboard and the chair, but the availability of said unicorns is the one ingredient of the system that is not in an employer's control. What is in control is sponsoring more education, and focusing on simplicity and constraint analysis.
ebiester 13y ago
I fantasize about rebuilding the java ecosystem. Create "JPM", don't allow anything from the old Maven system in it. Create a new "hip" minimalist framework that ignores everything about java and its notions of OOP. Basic dependency management handled NPM-style. Developers encourage to put everything in the same package for their own stuff, and everything is package level. Getters and setters and encapsulation and java's poor implementation of OOP ignored. Heck, fork Spark, split out the individual pieces as modules, and start there. All "JPM" packages expected to be under 100 lines, a la NPM. Frameworks are discouraged. Take what you need a la carte. I'd argue that java is a better back end language than javascript in and of itself, but that the community norms have led to the pain we see. Have an extreme preference for 1.8 code (IE, the new java.time system) and away from anything bloated. Rethink for the common use cases, rather than trying to support everything under the sun. Spring still exists for that, after all. :) The problem with java isn't that it's a BLUB language anymore, but that it's a BLUB ecosystem. I'm not saying the language is perfect, but Java 8 as good as ruby or javascript for web development if the ecosystem is ignored.
- fat0wl 13y ago
  Hear hear. When I read about JVM performance issues I usually immediately think "team of 20+ complacent corporate devs who don't care if this succeeds or not because they can get a new job insanely easily at a moment's notice". The issue with these other frameworks/languages is that even when modern frameworks are used there are usually some Java backend services (or another "powerful-outside-of-the-web-also" language) if the app does anything substantially complex. A ton of the hip web tech is basically for building front-end and cobbling together relatively inefficient backends. I was an RoR guy myself but moved to Java & I see the correct hipster tech here but just lacking unanimity. I bet if the RoR community decided "Clojure hell yea" or "Scala hell yea" and contributed lots of plug-ins to those ecosystems, they would generally have similar development times with a lot of performance gain for free. I'm thinking about trying Play framework on Heroku just to see what the monthly bills & New Relic readouts look like in comparison.
  - cloverich 13y ago
    Here's[1] a comparison of the two from an RoR guy who used Play! (with Scala) for 16 months. I found the ensuing discussion pretty informative with minimal FUD / bashing. [1]: https://groups.google.com/forum/#!searchin/play-framework/play$20vs$20ruby$20on$20rails/play-framework/F72DYs2MPsA/f_ol4AZplz0J https://groups.google.com/forum/#!searchin/play-framework/pl...
lampe3 13y ago
I stopped reading at the part with the "Type information". Or better i should have stopped... Another post about(and the from paypal too) about my language X vs language Y... It's the same flameware like Facebook started with its we can't get HTML5 as fast as native... then some people who know what they where doing did it... No don't use java No don't use javascript For god sake learn one language or two and learn it very well! i mean very well! and you will write good code and performant code... EDIT: Why do i get down voted? Jesus HN gets worse and worse...
- dangrossman 13y ago
  You're getting downvoted because this isn't a "language X vs language Y" article at all. Its point is that the language choice doesn't matter when you're building on top of crap, and jumping from one language to another probably won't solve platform problems. It's an article about the politics of being a software developer in enterprise where management is entrenched in legacy codebases, not about language features.
  - lampe3 13y ago
    What? the first part is exactly that... for example: "So which is the faster virtual machine, V8 or JVM? My belief is that when you are coding in the most basic building blocks of either virtual machine (i.e JavaScript vs JVM Byte Code) that the JVM will win out every time. If you start to compare higher up, you may end up comparing apples with oranges and see false comparisons. For example, if we consider this comparison of V8 vs JVM performance at mathematical calculations. This blog post tells us that if we relax our specifications we can calculate things faster. Here is the specification that V8 must use for Math.pow and here is the specification that the JVM must use for Math.pow notice that the JavaScript specification allows for an “implementation-dependent approximation” (of unspecified accuracy) while the JVM version has the addition that" and at the second part too: "... a limit of Node.js itself when used with the backing APIs that they have to call through to." It's just hidden better... to me it reads like a java fan boy is crying why a company switched to language X from his language Y...
    - 13y ago
ExpiredLink 13y ago
> Spring brings a lot of functionality to the table. Likely far too much functionality. Most people are moving away from the monolithic application and moving towards smaller more lightweight frameworks… It's high time to debunk Spring. In the name of 'Dependency Injection' many Java developers still create complex, monolithic and - first of all - untestable applications with this framework. In fact, Spring adds another layer of complexity on top of Java EE without real benefit. It fosters Anti-Patterns like 'Dependency Injection' that give you the illusion of modularity but instead create dependencies within the application ('Dependency Injection' needs to be taken literally).
- mtrimpe 13y ago
  I'm pretty good with Spring and have even given trainings on it and I'd argue that Spring's main problem is that it needs to be seen in the right context: a poor man's DSL for declarative system configuration (it is last 10 years old after all.) That it creates untestable applications seems like completely nonsensical FUD however since the entire point of Spring is to keep code easily testable. Spring is an almost direct response to the lack of testability of the ServiceLocator anti-pattern. I'm not saying you can't make untestable code in Spring, but if you do the problem is most likely more with the people than with Spring.
  - penland 13y ago
    I've done a lot of Spring side work and your first paragraph is the best explanation I've ever read for the framework. I also agree that if you can't test Spring code you are doing something wrong. I get the frustration with having to create test harnesses for some things, and the SpringJ4Unit stuff oftentimes feels a bit too magical, but I've never seen well written Spring code that wasn't easily testable at the both the class and module functionality layer.
    - ExpiredLink 13y ago
      With Spring you can easily test your empty mocked-out shells. Great!
- 13y ago
iamleppert 13y ago
He forgets to mention one critical fact: developer productivity. Most applications (sans his terrible example of protein folding) are not that performance critical. Most of the code you write as a developer will do-away with any platform or language advantages itself. It is not productive to write web applications in Java. Everyone hates it, which is why we have ruby, php, and now javascript (node). I'd much rather trade a little performance for a scripting language anyday.
- cloverich 13y ago
  I wonder if someone has written up comparisons on their productivity between the languages / frameworks. My previous job had me on the python stack (Pyramid / Mako / Sqlalchemy), which I loved, but have been on Java / Play! framework for the last 10 months. The one thing that has consistently surprised me is how productive I feel writing Java. Mainly, I notice I make far fewer mistakes / typos while coding; the IDE does a great deal of the work for me, and as the application grows, having static types has (for me) improved the readability of code I haven't looked at for a while. No doubt a large part of any language choice is subjective, but I'm starting to believe that developer productivity has more to do with the tools and frameworks than the language itself. As these continue to improve in Java-land, I'm willing to bet that the disparity in productivity will continue to decrease.
  - goostavos 13y ago
    >The one thing that has consistently surprised me is how productive I feel writing Java. I know exactly how you feel. I recently moved from a Java project to a Python one and felt a little amiss without the power that an IDE gives a statically typed language. The tooling for Python flat out sucks. And it pains me to say, as I love the language to death, but I do feel more productive in other languages. Whether or not that's true is another thing entirely, but it does feel like a lot of time is wasted calling `dir` on objects to figure out what the hell the name was of the function you need. As of about a week ago I switched from my text editor + console to Eclipse + PyDev. It's not perfect -- intellisense functionality is really hit or miss, but it does make me feel a lot more productive. The background linting is pretty good at catching basic errors as you type them, which is really nice. It's also does an OK job at renaming things -- which is generally a nightmare to do on a big project in Python. So, point being, while I miss a lot of the other tooling that an IDE affords a statically typed language, PyDev gets you, like, 80% of the way there. Worth checking out if you move back to Python.
coolsunglasses 13y ago
Lots of people trying to unblub blub. I'd venture a guess that mimicking Node.js isn't going to fix the deeper problems with Java.
gprasanth 13y ago
I was reminded that self deceit is the worst of its kind on reading the article. seeing a benchmark and deciding what to write your next application in kinda like that. you browse until you find the benchmark that aligns with what works out best for the lazy you and not works out best for the faithful app. this was just an observation of mine and not necessarily true or applicable to you.
neakor 13y ago
Application based comparisons are usually flawed, because the writer will never and cannot release enough info to reveal the real problem. Just because your bad-performing application runs faster on one platform doesn't necessarily mean that platform is superior to the old one. It simply means whatever problems you have in your bad-performing application manifests less in the new platform. In the end, fix your application!
kylequest 13y ago
What a shocker... a Java guy says that you shouldn't abandon Java :-) Lots of wild speculations... especially about the reasons PayPal picked node.js/JavaScript instead of a Ruby stack. He says that Ruby wasn't chosen because you can run it on JVM ignoring the fact that you can run JavaScript on JVM too using DynJS (he actually mentions DynJS earlier in his blog post).
pkolaczk 13y ago
So far I tried compiling some of my numeric Scala code to JS with Scala.JS and unfortunately running that code in Chrome is much, much slower (~10-20 times) than running the original version on Java 7. :( Not sure if it is because Scala.JS creates poor code, or just JS is still not as fast as they claim.
- virtualwhys 13y ago
  Current version of Scala.JS is an alpha release, don't expect miracles.
jroseattle 13y ago
This is more about PayPal's dirty laundry than an comparative evaluation of technologies. If a collective set of tools are used improperly, comparisons beyond the context of those walls means very little.
api 13y ago
TL;DR: * Micro-benchmarks don't provide a good picture of real system performance and can be skewed to make almost anything look better/worse than anything else. * Much of Java's "slowness," clunkiness, large amounts of boilerplate code, etc. is the fault of the Java ecosystem and libraries rather than the language itself or the JVM. * The inverse is true for Node: the quality of its ecosystem helps make up for language and runtime shortcomings.
nswanberg 13y ago
This reads like someone reading a 1995 article about a company switching from writing Netscape Enterprise Server code in C++ on a Windows machine to Apache code in perl on a Sun server, and then writing a defense of the x86 processor against SPARC. I might have missed something but I don't recall the original PayPal article arguing that the switch was motivated by moving from a slow virtual machine to a faster one. But just as switching from C++ to perl would trade some computing power for productivity in getting a site built, I don't see it as controversial for an organization to say they achieved the same productivity gains going from java to javascript.
eddflrs 13y ago
Does anyone know how vertx[1] fits into this picture? It gives you the JVM, JavaScript, and non-blocking I/O. [1]: http://vertx.io/ http://vertx.io/
- kylequest 13y ago
  Nodyn is an effort to run node.js apps on top of Vert.x. JavaScript alone is not enough to make Vert.x popular for node.js people. You need to be able to leverage the node.js module echosystem. Hopefully Nodyn will do that, so you can use your favorite node modules on Vert.x. It's still a work in progress though.
  - eddflrs 13y ago
    Cool, Nodyn looks interesting. Thanks for the info.
bhauer 13y ago
I've had the misfortune of using the PayPal Java SDK to integrate PayPal credit card and Express payment into a web application. Based on that experience, I find it especially difficult to digest any opinion that PayPal has on Java. They may be experts on handling money, processing credit card payments, fraud prevention, and the like. (Or maybe not, given their reputation.) But their opinion on Java is about as meaningful to me as Microsoft's opinion on Java. Or more on point, a Rails developer's opinion on Java. Both are characteristically stale. And not "whew, that bread is really ripe, throw it out" stale. Rather, "what we have here is a new form of fungal life" stale. I've worked with a fair number of Java APIs. I would put the quality of their Java SDK in the bottom 10%. It has the stench of really old Java. Remember how the first Java APIs reeked of C? The fact that the PayPal Java SDK still looked like hastily ported C code in 2011 when I last worked with it makes me worry that their entire manner of using Java is stuck in the early 2000s. That's being generous. It's the kind of Java you'd expect to see written by a company that still thinks about "applets" and "beans" and tries to come up with clever coffee double-entendres for their package names. My theory: PayPal is big but they know they are vulnerable. They've acquired BrainTree and that helps. I suspect they want to improve their technical chops and felt a need to migrate away from Java in order to break free of their organizational quagmire. PayPal is from 1998 and if their Java SDK is any indicator, they probably have a Subversion commit log for their core systems going all the way back. This sort of move is the kind of thing I could see myself doing if I were in a decision-making position at PayPal. Attempting to invigorate sensible modern Java practices within that organization would seem hopeless. I would have to start anew with an approach that is free of poisonous legacy. As others have pointed out, the fact that a rewrite yielded so little performance gain is decidedly troubling. At least they got a fun and well-linked blog entry out of it. If I had just gone through that effort and reaped such a modest improvement, I'd be crestfallen.
- ControllerLowDC 13y ago
  Precisely. I spent some time at PayPal and, one thing I think is being lost in all of these discussions is exactly what was "fixed." All your points on introducing new practices to an older, establish (in the company) framework and technology are spot on. I see this as more a way to shake things up and improve things under the guise of new tech. Also, this is in the top-most bit of their stack. All of the underlying services are the same. That's where the majority of the latency exists and a lot of it comes from old-school PayPal. But you can't shift a company by trying to tackle a deeply integrated bit of architecture first. The easiest place to demonstrate that change is feasible and actually quite beneficial (to a company that has been change-averse in the past) is your app tier. Get them on your side with an easy (easier?) win, then tackle the harder problems.
puppetmaster3 13y ago
I think it comes to this: Java > JavaScript. But Java programmer < JavaScript programmer, due to culture. (ex: Spring, Maven, EJB light). If Java programmer is true polyglot, then whatever tool you are using wins. That's why C programmers are the best.
tn13 13y ago
In our first lecture at graduate college, the prof told us how to read research papers. Rule 1 : Louder the guest shouts of his honor, faster you should count your spoons. Similarly if someone tells you about 50% savings just because you switched to some other technology you should be extremely skeptical. Besides those people who engage in `Node vs X` debate every now and then should spend some time educating themselves about compilers and interpreters to understand what makes some programming languages faster than others in certain circumstances.
peter_mcrae 13y ago
First post on HN. I'm not entirely sure how the PayPal Node vs. Java message has been packaged, but I think there is a lot of confusion. From my understanding (via blogs and conferences), Node was used to replace Java on the front-end. Meaning, the Node front-end is still calling Java + other stacks upstream -- which likely explains the performance benchmarks (the front-end wasn't the bottle neck). That said, even with similar performance benchmarks, I can see how Node might result in higher developer productivity for the front-end components of PayPal -- hence the enthusiasm they are showing. Lot's of negativity and blame on this thread, but I think it's because people are making assumptions.
exabrial 13y ago
Incredible insight... Here's the TLDR: Everything is faster when you engineer it a second time. Rewriting in a different language is not an apples to apples comparison.
anotherfadjs 13y ago
Lets not forget about the shitload of unsolved security vulnerabilities in the v8 engine used by node.js http://www.cvedetails.com/vulnerability-list/vendor_id-1224/product_id-17734/Google-V8.html http://www.cvedetails.com/vulnerability-list/vendor_id-1224/...

Summary: Most frameworks sux.

romanovcode 13y ago
Or PayPal programmers.
zabraxias 13y ago
Wrong. PayPal refuses to use open source frameworks because of an internal culture (and self preservation interest) of some very old "architects" who really only care about being in charge of the next internal framework. The best part is they later get to showcase how much better the new one is compared to the old framework, even though it's multiples slower than the slowest open-source counterpart.
- cgh 13y ago
  Spring is open source. From the sounds of things, they replaced Spring with a closed in-house framework.
  - zabraxias 13y ago
    "Based on Spring" is not Spring. I worked there for a bit, they called it Sparta, and its appeal to the top tech brass was they could restrict which objects are available for use in the library they provided for you. The subtle message behind it was, we can hire anyone as long as we limit what they can do.
    - cgh 13y ago
      Thanks, I stand corrected. Sounds heinous.
    - mattgreenrocks 13y ago
      That's atrocious. One thing I'll never understand is how companies get so hungry for developers that they subconsciously admit to hiring amateurs: "well, Joe can't do too much damage!"
- ricardobeat 13y ago
  Their node.js "framework" is really just a couple modules sitting on top of express, and they are all open-source.

auvi 13y ago

TL;DR from the post: """ My take-home for you, the persistent reader who has read all my ramblings in this post... Don’t build your app on top of a pile of crap in-house framework. """

kcmarshall 13y ago
Which suggests that take-home #0 should be: Don't build a pile of crap in-house framework. Sniffing that out before too late is the tricky part...

_random_ 13y ago

That is a refreshingly sober point of view. Blaming frameworks is easy. Speaking of technical debt is hard.

Xorlev 13y ago

Well-reasoned. I've always found framework overhead and especially tech debt to be the largest contributors to performance issues. Especially when said frameworks have lots of ways to do things and they're not all the best way to do it. Modern JVM development starts to look a lot like Sinatra -- you don't get a lot baked in, but you probably didn't need it anyways.

pron 13y ago
Yes. JAX-RS (the standard implemented by Jersey), which is part of the often maligned Java EE, is a (surprisingly?) well designed REST "microframework". Java EE in general has come a long way from the monster it used to be, and is becoming quite pleasant.
- edwinnathaniel 13y ago
  You can use Servlet + Jersey. It's just one dependency away and minimal web.xml config (or spring-jersey...whichever). It's not necessarily tied to Java EE albeit Java EE app-server _must_ implement JAX-RS properly.
  - pron 13y ago
    Both Servlet and JAX-RS are pretty central pieces of Java EE. But Java EE is now really a collection of modules that you can pick and choose.
curun1r 13y ago
The biggest contributor to performance issues these days, from what I've seen, is synchronous code. The JVM can be blindingly fast at running your code, but when it hits a point where you make a database call or web services call that's implemented synchronously, it sits there, waiting, doing nothing. No amount of VM optimizations can increase of the speed of idling code. This is why Node apps often out-perform Java apps despite running on a VM that's significantly slower. Node is pervasively asynchronous...it's baked into the DNA of the platform and almost all libraries follow that philosophy. And as organizations move to a micro-services architecture where one external call can result in multiple internal calls between systems, the need for reactive, asynchronous programming becomes all the more glaring. That's where this critique, for me, fell down. It's spent too much time comparing the JVM to V8 and not enough time talking about how the applications being run were designed. Because CPUs are fast enough these days that any of the VMs will give you decent performance for workloads that aren't CPU-bound as long as you, the application designer, do the right thing. It's just that some environments, Node in particular, make it easier to do the right thing. This will get easier in Java with the introduction of lambdas in Java 8 and the inevitable asynchronous community frameworks and libraries, but right now it's still harder to write reactive applications in Java than it is in Node.

smoyer 13y ago

Management won't take the blame but ... who allowed the architects/engineers to build two systems that failed the same way?

wpietri 13y ago
I agree that management is likely a problem here, but disagree with the framing. If you're in a structure where management's job is to allow/deny, then I think the code is already doomed. Software development is work that can't be effectively micromanaged. In software, I think management's job is to support and challenge. All the people coding should be professionals, thoughtful and aware of issues like tech debt, second-system effects, reinventing the wheel, and so on. Management's job isn't to be more professional than the pros; it's to support the pros and make sure they're living up to (and sometimes advancing) their professional standards. I'm sure there are plenty of people who work with these frameworks that know that they're crap. Management didn't fail by allowing these to get built. They failed by creating power relationships that allowed the people who couldn't see the problems to control the building of the frameworks, and to suppress criticism and improvement.

sehrope 13y ago

Regarding the JVM performance: > Under minimum load the best page rendering time was 233ms Regarding the node.js performance: > Under minimum load the best page rendering time was 249ms What the heck are they doing that takes 250ms that could possibly be app code related? That's a lot of CPU usage if its actually doing CPU-bound work. Since these are web apps that's obviously not the case. The request time is going to be mostly waiting for external resources (DB, message queue, etc). Without knowing how much time is spent on those external requests these numbers are meaningless. Language choice is no where near as important as your DB-access patterns (and more generally DB performance, caching, etc). If every request you process involves serially accessing a 50-100ms external resource four or five times then it will always take 250ms (though you could increase parallelism of multiple requests if done right).

cgh 13y ago
The fact that both implementations are ridiculously slow tells me to look for commonalities. In this case, how much do you want to bet that there are some truly scary, gigantic and all-but-incomprehensible stored procedures underlying everything?
- Xorlev 13y ago
  No bet. :)
camus2 13y ago
> Language choice is no where near as important as your DB-access patterns (and more generally DB performance, caching, etc). i'm pretty sure the paypal node app does not directly access the db , the node app talks to a private api.
- Helianthus 13y ago
  That doesn't really change the point, it just wraps it with a layer of abstraction.
  - 13y ago

Joeri 13y ago

Maybe we should assume the best instead of the worst. We could assume that paypal has capable developers who are knowledgeable about both the jvm and node.js, and who have concluded that their specific featureset can best be built using node. That is not a value judgment of the jvm vs node in general, as their situation is theirs and theirs alone.

spullara 13y ago

Two poorly written apps compared with blame for each placed on the underlying virtual machine rather than the part between the chair and the keyboard. I'd much rather compare VMs and systems with something a little more structured and open like the TechEmpower benchmarks. They still have their flaws and simplifications, but at least the source for all apps is open and you can see and contribute to them.

srean 13y ago

This was a good read. Minor nitpick, given the author's stress of not comparing apples to oranges I think the motivating anecdote on protein folding struck a dissonant cord. A fair comparison would have been a profile guided optimization with comparably aggressive compile optimization flags. For these types of problems the profile characteristic is fairly static, so there is hardly ever a need to pay for the warm up time. Java can indeed be very fast, and in my opinion JVM is the one of the most optimized VMs that we have now. That said Java code beating a C or a C++ code on a CPU bound task raised an eyebrow. In my experience Java usually gets to the 80% speed of C++ quite easily but at a cost of 2.5 to 3.5 the memory. For these types of applications memory tends to be an expensive resource. For our number crunching server, 80% of its cost is sunk in the RAM.

stephenconnolly 13y ago
I debated including that example, but at the end the point I was making is that the JVM can be faster than native code in some cases... this case had static allocation of data and the calculations could benefit from MMX instructions when available, plus there was a logical unrolling that I think it was able to infer from code path analysis
- rossjudson 13y ago
  And superword optimization, when possible...
pron 13y ago
My experience is similar: Java will get you 80% of super-optimized-compiled C++ speed for the cost of more memory (although you may choose to give Java less memory and pay in speed), and will get much closer to C++ speed with some work (off-heap memory, etc.). However, Java has the upper hand in two scenarios: 1) long-running applications that are developed by a large team – those usually make heavy use of virtual inheritance for the sake of good engineering, and the execution profile is not static, so the program can greatly benefit from JIT optimization; and 2) fairly complex multithreaded code – Java's incredibly useful and well-implemented blocking and lock-free data structures, as well as an incredible work-stealing scheduler (all expertly programmed by Doug Lea) make great use of a general purpose GC, plus Java usually gets them about 5 years before C++ (if C++ gets them at all).

abritishguy 13y ago

Damn, paypal really suck at making fast sites.

deleted 13y ago

[deleted]

tunesmith 13y ago

I wonder if a lot of headway could have been made simply by doing some constraint analysis on the Spring framework and figuring out how to improve the performance. I have a hard time believing that Spring itself was completely topped out. It might have been an architectural choice, or some bloat that could have been taken out incrementally. Instead they seem to be going the route of "the unicorns will solve it". The overall cost of that approach seems higher - finding unicorns is rare. Finding unicorns that are very good at all areas is even harder. I know there are a lot of unicorns on this board, but there sure aren't a lot of unicorn resumes on a hirer's desk. I think the value of Spring has always been not that it's super-high-performance out of the box, but that it does a lot for you, and that as you learn more and more of it, you can develop solutions fairly quickly. It's built to make it easier on the developer, not the servers - but it is still easy to improve performance by focusing on constraints. I think it's easy to blame what's between the keyboard and the chair, but the availability of said unicorns is the one ingredient of the system that is not in an employer's control. What is in control is sponsoring more education, and focusing on simplicity and constraint analysis.

ebiester 13y ago

I fantasize about rebuilding the java ecosystem. Create "JPM", don't allow anything from the old Maven system in it. Create a new "hip" minimalist framework that ignores everything about java and its notions of OOP. Basic dependency management handled NPM-style. Developers encourage to put everything in the same package for their own stuff, and everything is package level. Getters and setters and encapsulation and java's poor implementation of OOP ignored. Heck, fork Spark, split out the individual pieces as modules, and start there. All "JPM" packages expected to be under 100 lines, a la NPM. Frameworks are discouraged. Take what you need a la carte. I'd argue that java is a better back end language than javascript in and of itself, but that the community norms have led to the pain we see. Have an extreme preference for 1.8 code (IE, the new java.time system) and away from anything bloated. Rethink for the common use cases, rather than trying to support everything under the sun. Spring still exists for that, after all. :) The problem with java isn't that it's a BLUB language anymore, but that it's a BLUB ecosystem. I'm not saying the language is perfect, but Java 8 as good as ruby or javascript for web development if the ecosystem is ignored.

fat0wl 13y ago
Hear hear. When I read about JVM performance issues I usually immediately think "team of 20+ complacent corporate devs who don't care if this succeeds or not because they can get a new job insanely easily at a moment's notice". The issue with these other frameworks/languages is that even when modern frameworks are used there are usually some Java backend services (or another "powerful-outside-of-the-web-also" language) if the app does anything substantially complex. A ton of the hip web tech is basically for building front-end and cobbling together relatively inefficient backends. I was an RoR guy myself but moved to Java & I see the correct hipster tech here but just lacking unanimity. I bet if the RoR community decided "Clojure hell yea" or "Scala hell yea" and contributed lots of plug-ins to those ecosystems, they would generally have similar development times with a lot of performance gain for free. I'm thinking about trying Play framework on Heroku just to see what the monthly bills & New Relic readouts look like in comparison.
- cloverich 13y ago
  Here's[1] a comparison of the two from an RoR guy who used Play! (with Scala) for 16 months. I found the ensuing discussion pretty informative with minimal FUD / bashing. [1]: https://groups.google.com/forum/#!searchin/play-framework/play$20vs$20ruby$20on$20rails/play-framework/F72DYs2MPsA/f_ol4AZplz0J https://groups.google.com/forum/#!searchin/play-framework/pl...

lampe3 13y ago

I stopped reading at the part with the "Type information". Or better i should have stopped... Another post about(and the from paypal too) about my language X vs language Y... It's the same flameware like Facebook started with its we can't get HTML5 as fast as native... then some people who know what they where doing did it... No don't use java No don't use javascript For god sake learn one language or two and learn it very well! i mean very well! and you will write good code and performant code... EDIT: Why do i get down voted? Jesus HN gets worse and worse...

dangrossman 13y ago
You're getting downvoted because this isn't a "language X vs language Y" article at all. Its point is that the language choice doesn't matter when you're building on top of crap, and jumping from one language to another probably won't solve platform problems. It's an article about the politics of being a software developer in enterprise where management is entrenched in legacy codebases, not about language features.
- lampe3 13y ago
  What? the first part is exactly that... for example: "So which is the faster virtual machine, V8 or JVM? My belief is that when you are coding in the most basic building blocks of either virtual machine (i.e JavaScript vs JVM Byte Code) that the JVM will win out every time. If you start to compare higher up, you may end up comparing apples with oranges and see false comparisons. For example, if we consider this comparison of V8 vs JVM performance at mathematical calculations. This blog post tells us that if we relax our specifications we can calculate things faster. Here is the specification that V8 must use for Math.pow and here is the specification that the JVM must use for Math.pow notice that the JavaScript specification allows for an “implementation-dependent approximation” (of unspecified accuracy) while the JVM version has the addition that" and at the second part too: "... a limit of Node.js itself when used with the backing APIs that they have to call through to." It's just hidden better... to me it reads like a java fan boy is crying why a company switched to language X from his language Y...
  - 13y ago

ExpiredLink 13y ago

> Spring brings a lot of functionality to the table. Likely far too much functionality. Most people are moving away from the monolithic application and moving towards smaller more lightweight frameworks… It's high time to debunk Spring. In the name of 'Dependency Injection' many Java developers still create complex, monolithic and - first of all - untestable applications with this framework. In fact, Spring adds another layer of complexity on top of Java EE without real benefit. It fosters Anti-Patterns like 'Dependency Injection' that give you the illusion of modularity but instead create dependencies within the application ('Dependency Injection' needs to be taken literally).

mtrimpe 13y ago
I'm pretty good with Spring and have even given trainings on it and I'd argue that Spring's main problem is that it needs to be seen in the right context: a poor man's DSL for declarative system configuration (it is last 10 years old after all.) That it creates untestable applications seems like completely nonsensical FUD however since the entire point of Spring is to keep code easily testable. Spring is an almost direct response to the lack of testability of the ServiceLocator anti-pattern. I'm not saying you can't make untestable code in Spring, but if you do the problem is most likely more with the people than with Spring.
- penland 13y ago
  I've done a lot of Spring side work and your first paragraph is the best explanation I've ever read for the framework. I also agree that if you can't test Spring code you are doing something wrong. I get the frustration with having to create test harnesses for some things, and the SpringJ4Unit stuff oftentimes feels a bit too magical, but I've never seen well written Spring code that wasn't easily testable at the both the class and module functionality layer.
  - ExpiredLink 13y ago
    With Spring you can easily test your empty mocked-out shells. Great!
13y ago

iamleppert 13y ago

He forgets to mention one critical fact: developer productivity. Most applications (sans his terrible example of protein folding) are not that performance critical. Most of the code you write as a developer will do-away with any platform or language advantages itself. It is not productive to write web applications in Java. Everyone hates it, which is why we have ruby, php, and now javascript (node). I'd much rather trade a little performance for a scripting language anyday.

cloverich 13y ago
I wonder if someone has written up comparisons on their productivity between the languages / frameworks. My previous job had me on the python stack (Pyramid / Mako / Sqlalchemy), which I loved, but have been on Java / Play! framework for the last 10 months. The one thing that has consistently surprised me is how productive I feel writing Java. Mainly, I notice I make far fewer mistakes / typos while coding; the IDE does a great deal of the work for me, and as the application grows, having static types has (for me) improved the readability of code I haven't looked at for a while. No doubt a large part of any language choice is subjective, but I'm starting to believe that developer productivity has more to do with the tools and frameworks than the language itself. As these continue to improve in Java-land, I'm willing to bet that the disparity in productivity will continue to decrease.
- goostavos 13y ago
  >The one thing that has consistently surprised me is how productive I feel writing Java. I know exactly how you feel. I recently moved from a Java project to a Python one and felt a little amiss without the power that an IDE gives a statically typed language. The tooling for Python flat out sucks. And it pains me to say, as I love the language to death, but I do feel more productive in other languages. Whether or not that's true is another thing entirely, but it does feel like a lot of time is wasted calling `dir` on objects to figure out what the hell the name was of the function you need. As of about a week ago I switched from my text editor + console to Eclipse + PyDev. It's not perfect -- intellisense functionality is really hit or miss, but it does make me feel a lot more productive. The background linting is pretty good at catching basic errors as you type them, which is really nice. It's also does an OK job at renaming things -- which is generally a nightmare to do on a big project in Python. So, point being, while I miss a lot of the other tooling that an IDE affords a statically typed language, PyDev gets you, like, 80% of the way there. Worth checking out if you move back to Python.

coolsunglasses 13y ago

Lots of people trying to unblub blub. I'd venture a guess that mimicking Node.js isn't going to fix the deeper problems with Java.

gprasanth 13y ago

I was reminded that self deceit is the worst of its kind on reading the article. seeing a benchmark and deciding what to write your next application in kinda like that. you browse until you find the benchmark that aligns with what works out best for the lazy you and not works out best for the faithful app. this was just an observation of mine and not necessarily true or applicable to you.

neakor 13y ago

Application based comparisons are usually flawed, because the writer will never and cannot release enough info to reveal the real problem. Just because your bad-performing application runs faster on one platform doesn't necessarily mean that platform is superior to the old one. It simply means whatever problems you have in your bad-performing application manifests less in the new platform. In the end, fix your application!

kylequest 13y ago

What a shocker... a Java guy says that you shouldn't abandon Java :-) Lots of wild speculations... especially about the reasons PayPal picked node.js/JavaScript instead of a Ruby stack. He says that Ruby wasn't chosen because you can run it on JVM ignoring the fact that you can run JavaScript on JVM too using DynJS (he actually mentions DynJS earlier in his blog post).

pkolaczk 13y ago

So far I tried compiling some of my numeric Scala code to JS with Scala.JS and unfortunately running that code in Chrome is much, much slower (~10-20 times) than running the original version on Java 7. :( Not sure if it is because Scala.JS creates poor code, or just JS is still not as fast as they claim.

virtualwhys 13y ago
Current version of Scala.JS is an alpha release, don't expect miracles.

jroseattle 13y ago

This is more about PayPal's dirty laundry than an comparative evaluation of technologies. If a collective set of tools are used improperly, comparisons beyond the context of those walls means very little.

api 13y ago

TL;DR: * Micro-benchmarks don't provide a good picture of real system performance and can be skewed to make almost anything look better/worse than anything else. * Much of Java's "slowness," clunkiness, large amounts of boilerplate code, etc. is the fault of the Java ecosystem and libraries rather than the language itself or the JVM. * The inverse is true for Node: the quality of its ecosystem helps make up for language and runtime shortcomings.

nswanberg 13y ago

This reads like someone reading a 1995 article about a company switching from writing Netscape Enterprise Server code in C++ on a Windows machine to Apache code in perl on a Sun server, and then writing a defense of the x86 processor against SPARC. I might have missed something but I don't recall the original PayPal article arguing that the switch was motivated by moving from a slow virtual machine to a faster one. But just as switching from C++ to perl would trade some computing power for productivity in getting a site built, I don't see it as controversial for an organization to say they achieved the same productivity gains going from java to javascript.

eddflrs 13y ago

Does anyone know how vertx[1] fits into this picture? It gives you the JVM, JavaScript, and non-blocking I/O. [1]: http://vertx.io/ http://vertx.io/

kylequest 13y ago
Nodyn is an effort to run node.js apps on top of Vert.x. JavaScript alone is not enough to make Vert.x popular for node.js people. You need to be able to leverage the node.js module echosystem. Hopefully Nodyn will do that, so you can use your favorite node modules on Vert.x. It's still a work in progress though.
- eddflrs 13y ago
  Cool, Nodyn looks interesting. Thanks for the info.

bhauer 13y ago

I've had the misfortune of using the PayPal Java SDK to integrate PayPal credit card and Express payment into a web application. Based on that experience, I find it especially difficult to digest any opinion that PayPal has on Java. They may be experts on handling money, processing credit card payments, fraud prevention, and the like. (Or maybe not, given their reputation.) But their opinion on Java is about as meaningful to me as Microsoft's opinion on Java. Or more on point, a Rails developer's opinion on Java. Both are characteristically stale. And not "whew, that bread is really ripe, throw it out" stale. Rather, "what we have here is a new form of fungal life" stale. I've worked with a fair number of Java APIs. I would put the quality of their Java SDK in the bottom 10%. It has the stench of really old Java. Remember how the first Java APIs reeked of C? The fact that the PayPal Java SDK still looked like hastily ported C code in 2011 when I last worked with it makes me worry that their entire manner of using Java is stuck in the early 2000s. That's being generous. It's the kind of Java you'd expect to see written by a company that still thinks about "applets" and "beans" and tries to come up with clever coffee double-entendres for their package names. My theory: PayPal is big but they know they are vulnerable. They've acquired BrainTree and that helps. I suspect they want to improve their technical chops and felt a need to migrate away from Java in order to break free of their organizational quagmire. PayPal is from 1998 and if their Java SDK is any indicator, they probably have a Subversion commit log for their core systems going all the way back. This sort of move is the kind of thing I could see myself doing if I were in a decision-making position at PayPal. Attempting to invigorate sensible modern Java practices within that organization would seem hopeless. I would have to start anew with an approach that is free of poisonous legacy. As others have pointed out, the fact that a rewrite yielded so little performance gain is decidedly troubling. At least they got a fun and well-linked blog entry out of it. If I had just gone through that effort and reaped such a modest improvement, I'd be crestfallen.

ControllerLowDC 13y ago
Precisely. I spent some time at PayPal and, one thing I think is being lost in all of these discussions is exactly what was "fixed." All your points on introducing new practices to an older, establish (in the company) framework and technology are spot on. I see this as more a way to shake things up and improve things under the guise of new tech. Also, this is in the top-most bit of their stack. All of the underlying services are the same. That's where the majority of the latency exists and a lot of it comes from old-school PayPal. But you can't shift a company by trying to tackle a deeply integrated bit of architecture first. The easiest place to demonstrate that change is feasible and actually quite beneficial (to a company that has been change-averse in the past) is your app tier. Get them on your side with an easy (easier?) win, then tackle the harder problems.

puppetmaster3 13y ago

I think it comes to this: Java > JavaScript. But Java programmer < JavaScript programmer, due to culture. (ex: Spring, Maven, EJB light). If Java programmer is true polyglot, then whatever tool you are using wins. That's why C programmers are the best.

tn13 13y ago

In our first lecture at graduate college, the prof told us how to read research papers. Rule 1 : Louder the guest shouts of his honor, faster you should count your spoons. Similarly if someone tells you about 50% savings just because you switched to some other technology you should be extremely skeptical. Besides those people who engage in `Node vs X` debate every now and then should spend some time educating themselves about compilers and interpreters to understand what makes some programming languages faster than others in certain circumstances.

peter_mcrae 13y ago

First post on HN. I'm not entirely sure how the PayPal Node vs. Java message has been packaged, but I think there is a lot of confusion. From my understanding (via blogs and conferences), Node was used to replace Java on the front-end. Meaning, the Node front-end is still calling Java + other stacks upstream -- which likely explains the performance benchmarks (the front-end wasn't the bottle neck). That said, even with similar performance benchmarks, I can see how Node might result in higher developer productivity for the front-end components of PayPal -- hence the enthusiasm they are showing. Lot's of negativity and blame on this thread, but I think it's because people are making assumptions.

exabrial 13y ago

Incredible insight... Here's the TLDR: Everything is faster when you engineer it a second time. Rewriting in a different language is not an apples to apples comparison.

anotherfadjs 13y ago

Lets not forget about the shitload of unsolved security vulnerabilities in the v8 engine used by node.js http://www.cvedetails.com/vulnerability-list/vendor_id-1224/product_id-17734/Google-V8.html http://www.cvedetails.com/vulnerability-list/vendor_id-1224/...