4 ms·

The point of the article that resonates with me is how frequently a technology that is poorly fit with a problem domain is selected because of conventional wisd

by bhauer 13y ago

The point of the article that resonates with me is how frequently a technology that is poorly fit with a problem domain is selected because of conventional wisdom rather than data. Related, it is remarkable how we developers routinely cite Knuth's advice about premature optimization to justify our decision when the shoe fits, and then turn around and flatly ignore the advice when it doesn't fit. Selecting Hadoop before you have a specific and concrete need for it--or see that need approaching rapidly on the horizon--is in my experience often and surprisingly coupled with a disdain for other performance characteristics (because Knuth!). The developers prematurely selecting Hadoop as their data management platform will routinely be the same developers who believe it's reasonable for a web application with modest functionality to require dozens of application nodes to service concurrent request load measured in the mere thousands. The sad thing here being that application platforms and frameworks are not all that dissimilar; today, selecting something with higher performance in the application tier is relatively low-friction from a learning and effort perspective. But it's often not done. Meanwhile, selecting Hadoop on the data tier is a substantially different paradigm versus the alternative (as the article points out), so you have some debt incoming once you make that decision. And yet, this is done often enough for many of us to recognize the problem. In my experience, for a modest web application, it's better to focus resources and effort on avoiding common and stupid mistakes that lead to performance and scale pain. Selecting Hadoop too early doesn't really do a whole lot to move the performance needle for a modest web application. Trouble is, many web businesses are blind to the fact that they are a modest concern and not the next Facebook.

john_fushi 13y ago
"Related, it is remarkable how we developers routinely cite Knuth's advice about premature optimization to justify our decision when the shoe fits, and then turn around and flatly ignore the advice when it doesn't fit." It is especially bad when people uses Knuth's advice to forgo the conception part where you think about what technologies you will use and why.
superuser2 13y ago
> the same developers who believe it's reasonable for a web application with modest functionality to require dozens of application nodes to service concurrent request load measured in the mere thousands I'm assuming this is a dig at Rails and Django? What are you suggesting instead?
- bhauer 13y ago
  Just about anything on the JVM (Dropwizard, Play, Finagle, Scalatra, Rest-Express, Rest-Easy, Compojure, Unfiltered, Jersey, Vert.x, Spark), Go (Gorilla, Beego, Revel), Lua (Lapis), Haskell, Erlang...
  - superuser2 13y ago
    Are any of these environments actually proven in more than a handful of real-world production environments? I'd certainly like to get into alternatives but my impression was that none of these battle-tested the way Rails and Django are.
    - bhauer 13y ago
      I assume you mean battle-tested from a security standpoint. I suppose you'd need to make that judgment call yourself. In my opinion, many of the options I listed have seen sufficient production usage that I consider them roughly as secure as any mainstream framework. For example, Play and Scalatra are used at Netflix, BBC, LinkedIn, etc. Many expressly avoid notoriously dangerous (and frankly alarming) deserialize-user-input-and-execute patterns, which is in my experience the most common form of framework vulnerability. As for performance, scalability, and reliability under pressure, that is precisely the point I was making earlier: higher-performance platforms combined with frameworks designed for performance and scalability are targeting grace under pressure moreso than legacy frameworks. Higher performance gives you greater peace of mind with availability, responsiveness, and user experience.
    - jake_morrison 13y ago
      There is a reason that the largest Internet companies are still running their infrastructure on Java and C++. It scales and operations people know how to manage it. Rails is the classic example of optimizing for developer ease of use instead of performance. Which probably makes sense for startups, but scaling is a pain in the ass. Openresty is great for infrastructure level solutions, e.g. request routing and authentication. It is used by some of the largest Internet companies in China. The benchmarks at http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ are a reminder of how poorly some popular frameworks perform. And how poorly cloud performs vs relatively cheap dedicated hardware.
VladRussian2 13y ago
>Selecting Hadoop before you have a specific and concrete need for it--or see that need approaching rapidly on the horizon the great achievement of Hadoop isn't performance, throughput, etc... (and in many cases it can actually be worse than alternatives). The great achievement is that it is an easy (i.e. on the orders of magnitude) accessible (i.e. easy installable, configurable, scalable, supportable, programmable) cluster for "mere mortal" enterprise. Before Hadoop, any clusterable software (where even laughable by Hadoop standards 2 nodes were already priced at the cluster level) would cost arm and leg and 4 or 8 nodes would be considered a huge cluster. Hadoop gives a chance to an enterprise (not to CERN :) to taste an unconstrained (loose use of term) distributed computing, and the next generation of software is already coming - Impala, etc... And obligatory - "nobody gets fired for using Hadoop".

by bhauer 13y ago

john_fushi 13y ago

"Related, it is remarkable how we developers routinely cite Knuth's advice about premature optimization to justify our decision when the shoe fits, and then turn around and flatly ignore the advice when it doesn't fit." It is especially bad when people uses Knuth's advice to forgo the conception part where you think about what technologies you will use and why.

superuser2 13y ago

> the same developers who believe it's reasonable for a web application with modest functionality to require dozens of application nodes to service concurrent request load measured in the mere thousands I'm assuming this is a dig at Rails and Django? What are you suggesting instead?

bhauer 13y ago
Just about anything on the JVM (Dropwizard, Play, Finagle, Scalatra, Rest-Express, Rest-Easy, Compojure, Unfiltered, Jersey, Vert.x, Spark), Go (Gorilla, Beego, Revel), Lua (Lapis), Haskell, Erlang...
- superuser2 13y ago
  Are any of these environments actually proven in more than a handful of real-world production environments? I'd certainly like to get into alternatives but my impression was that none of these battle-tested the way Rails and Django are.
  - bhauer 13y ago
    I assume you mean battle-tested from a security standpoint. I suppose you'd need to make that judgment call yourself. In my opinion, many of the options I listed have seen sufficient production usage that I consider them roughly as secure as any mainstream framework. For example, Play and Scalatra are used at Netflix, BBC, LinkedIn, etc. Many expressly avoid notoriously dangerous (and frankly alarming) deserialize-user-input-and-execute patterns, which is in my experience the most common form of framework vulnerability. As for performance, scalability, and reliability under pressure, that is precisely the point I was making earlier: higher-performance platforms combined with frameworks designed for performance and scalability are targeting grace under pressure moreso than legacy frameworks. Higher performance gives you greater peace of mind with availability, responsiveness, and user experience.
  - jake_morrison 13y ago
    There is a reason that the largest Internet companies are still running their infrastructure on Java and C++. It scales and operations people know how to manage it. Rails is the classic example of optimizing for developer ease of use instead of performance. Which probably makes sense for startups, but scaling is a pain in the ass. Openresty is great for infrastructure level solutions, e.g. request routing and authentication. It is used by some of the largest Internet companies in China. The benchmarks at http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ are a reminder of how poorly some popular frameworks perform. And how poorly cloud performs vs relatively cheap dedicated hardware.

VladRussian2 13y ago

>Selecting Hadoop before you have a specific and concrete need for it--or see that need approaching rapidly on the horizon the great achievement of Hadoop isn't performance, throughput, etc... (and in many cases it can actually be worse than alternatives). The great achievement is that it is an easy (i.e. on the orders of magnitude) accessible (i.e. easy installable, configurable, scalable, supportable, programmable) cluster for "mere mortal" enterprise. Before Hadoop, any clusterable software (where even laughable by Hadoop standards 2 nodes were already priced at the cluster level) would cost arm and leg and 4 or 8 nodes would be considered a huge cluster. Hadoop gives a chance to an enterprise (not to CERN :) to taste an unconstrained (loose use of term) distributed computing, and the next generation of software is already coming - Impala, etc... And obligatory - "nobody gets fired for using Hadoop".