5 ms·

> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease o

> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipelines. Sure, there's a discipline to building pipelined tooling well, just as with any other kind of software. Your problems seem to stem from a lack of disciplined software engineering rather than the tools, or maybe just an environment that encouraged high technical debt. The kicker? We were using pipeline-based tooling ... running on a Hadoop cluster. Honestly, I'm a bit surprised to see such an apparent mindshare split (judging by some recent HN posts) between performant single-system approaches and techniques used in-cluster. The point that "be sure your data is really, truly big data" is obviously well made, and still bears repetition. Yet the logical follow-on is that these technique are even more applicable to cluster usage. Why would anyone throw away multiple orders of magnitude performance going to a cluster-based approach?

threeseed 12y ago
This "orders of magnitude" less performance going to a Hadoop cluster approach is nonsense. There are plenty of options for Hadoop that make it dramatically faster than the naive example in the article. Spark ? MR-Redis ? Storm ?
virmundi 12y ago
Something to keep in mind is that while a single app might be best served on a single machine piping data, multiple apps working the same data set probably wouldn't scale. Hadoop for all its faults does provide a nice, relativily simple programing platform to support multiple data processes. Edited for phone swipe mistakes.
developer1 12y ago
The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.
- freehunter 12y ago
  Yeah but 80% of the people writing Java and think they're good aren't as well. And plenty of companies support Java. The answer isn't "don't use it", it's "train your programmers in the languages they use".
  - kansface 12y ago
    The sorts of bugs people experience with Java mostly result in a crashed/stalled/hung process. Bash bugs erase your entire file system. The thing about Bash is that it is trivially easy to make these sorts of mistakes- the language just isn't suitable to general purpose scripting.
    - yourad_io 12y ago
      > Bash bugs erase your entire file system. EMPTY="" rm -rf $EMPTY/ Is this the kind of bug you're referring to? > The thing about Bash is that it is trivially easy to make these sorts of mistakes I fail to see how any other scripting language would have a different effect when you told it to do: system("rm -rf "+""+"/") > the language just isn't suitable to general purpose scripting. Yes, it is. Bash is deeper than it looks, but not by much. Learn how to handle errors and you'll be fine.
    - collyw 12y ago
      It shouldn't be able to erase your filesystem unless you are running as root or doing something equally stupid. That's pretty much common sense stuff for anyone that isn't a beginner.
      - gambiting 12y ago
        Yeah the "common sense stuff for anyone that isn't a beginner" argument is repeated ad nauseam, and even the largest companies make this mistake in their largest products. Take Valve - they should know how to write good code, right? And yet, last week an article was on top of HN, outlining how they put: "rm -rf '$STEAMROOT'/*" in their code, used to remove the library. But hey, no one checked if $STEAMROOT is not empty, so when it was for one user, Steam deleted all of his personal files, whole /home and /media directories next time it started. I'm not saying that command line tools shouldn't be used,but sometimes they are just too powerful for some users,and stupid mistakes like this happen.
Roboprog 12y ago
Unix/POSIX backgrounds are pretty common among the Hacker News crowd. Not so in "Enterprise" development. (Beam me up Scottie, there's no intelligent life here, only risk avoidance) Enterprise development is predominated by 2 or 3 trusted operating systems: Windows (/ .NET), and the JVM. POSIX systems are only useful in-so-far-as they are a cheaper (or sometimes more reliable) place to host Java virtual machines. Enterprise dev groups generally have very limited exposure to, and a lot of fear of, things like Borne shell, AWK, Perl, Python. These languages don't have Visual Studio or Eclipse to hold your hand while you make far reaching refactorings like renaming a variable. Sure, you and I would crawl log/data files trivially with a few piped commands, but that's a rare skill in most shops, at least since the turn of the century. Ugh, that sounds cliche, but it's hard not to feel that way after being drowned in "Java or nothing" for so long at work. http://tvtropes.org/pmwiki/pmwiki.php/Main/ElegantWeaponForAMoreCivilizedAge http://tvtropes.org/pmwiki/pmwiki.php/Main/ElegantWeaponForA...
- snambi 12y ago
  I agree with @roboprog. Most software shops employ engineers who don't have exposure into UNIX tools. Only few hardcore engineers have exposure or interest learning UNIX tools. For majority of engineers it is just a job. They simply use the same tool for everything. And they tend to use the tools that seem to get them into well paying jobs. If hadoop can get them good paying jobs, they would like us "hadoop" for something in their current job, even if that job can performed by a set of CLI utils. I have seen 100s of resume builder projects in the my past experience.
- jdmichal 12y ago
  Windows has PowerShell, which can be even more powerful than Unix shell depending on the data and processing required.
  - Roboprog 12y ago
    Does anybody without a unix/POSIX background even bother tinkering with PowerShell? Yes, it's a cool idea, as well, especially if your source data is in MS-Office, but I've not seen it put to much use.

threeseed 12y ago

This "orders of magnitude" less performance going to a Hadoop cluster approach is nonsense. There are plenty of options for Hadoop that make it dramatically faster than the naive example in the article. Spark ? MR-Redis ? Storm ?

virmundi 12y ago

Something to keep in mind is that while a single app might be best served on a single machine piping data, multiple apps working the same data set probably wouldn't scale. Hadoop for all its faults does provide a nice, relativily simple programing platform to support multiple data processes. Edited for phone swipe mistakes.

developer1 12y ago

The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.

freehunter 12y ago
Yeah but 80% of the people writing Java and think they're good aren't as well. And plenty of companies support Java. The answer isn't "don't use it", it's "train your programmers in the languages they use".
- kansface 12y ago
  The sorts of bugs people experience with Java mostly result in a crashed/stalled/hung process. Bash bugs erase your entire file system. The thing about Bash is that it is trivially easy to make these sorts of mistakes- the language just isn't suitable to general purpose scripting.
  - yourad_io 12y ago
    > Bash bugs erase your entire file system. EMPTY="" rm -rf $EMPTY/ Is this the kind of bug you're referring to? > The thing about Bash is that it is trivially easy to make these sorts of mistakes I fail to see how any other scripting language would have a different effect when you told it to do: system("rm -rf "+""+"/") > the language just isn't suitable to general purpose scripting. Yes, it is. Bash is deeper than it looks, but not by much. Learn how to handle errors and you'll be fine.
  - collyw 12y ago
    It shouldn't be able to erase your filesystem unless you are running as root or doing something equally stupid. That's pretty much common sense stuff for anyone that isn't a beginner.
    - gambiting 12y ago
      Yeah the "common sense stuff for anyone that isn't a beginner" argument is repeated ad nauseam, and even the largest companies make this mistake in their largest products. Take Valve - they should know how to write good code, right? And yet, last week an article was on top of HN, outlining how they put: "rm -rf '$STEAMROOT'/*" in their code, used to remove the library. But hey, no one checked if $STEAMROOT is not empty, so when it was for one user, Steam deleted all of his personal files, whole /home and /media directories next time it started. I'm not saying that command line tools shouldn't be used,but sometimes they are just too powerful for some users,and stupid mistakes like this happen.

Roboprog 12y ago

Unix/POSIX backgrounds are pretty common among the Hacker News crowd. Not so in "Enterprise" development. (Beam me up Scottie, there's no intelligent life here, only risk avoidance) Enterprise development is predominated by 2 or 3 trusted operating systems: Windows (/ .NET), and the JVM. POSIX systems are only useful in-so-far-as they are a cheaper (or sometimes more reliable) place to host Java virtual machines. Enterprise dev groups generally have very limited exposure to, and a lot of fear of, things like Borne shell, AWK, Perl, Python. These languages don't have Visual Studio or Eclipse to hold your hand while you make far reaching refactorings like renaming a variable. Sure, you and I would crawl log/data files trivially with a few piped commands, but that's a rare skill in most shops, at least since the turn of the century. Ugh, that sounds cliche, but it's hard not to feel that way after being drowned in "Java or nothing" for so long at work. http://tvtropes.org/pmwiki/pmwiki.php/Main/ElegantWeaponForAMoreCivilizedAge http://tvtropes.org/pmwiki/pmwiki.php/Main/ElegantWeaponForA...

snambi 12y ago
I agree with @roboprog. Most software shops employ engineers who don't have exposure into UNIX tools. Only few hardcore engineers have exposure or interest learning UNIX tools. For majority of engineers it is just a job. They simply use the same tool for everything. And they tend to use the tools that seem to get them into well paying jobs. If hadoop can get them good paying jobs, they would like us "hadoop" for something in their current job, even if that job can performed by a set of CLI utils. I have seen 100s of resume builder projects in the my past experience.
jdmichal 12y ago
Windows has PowerShell, which can be even more powerful than Unix shell depending on the data and processing required.
- Roboprog 12y ago
  Does anybody without a unix/POSIX background even bother tinkering with PowerShell? Yes, it's a cool idea, as well, especially if your source data is in MS-Office, but I've not seen it put to much use.