31 ms·

Command-line tools can be faster than a Hadoop cluster (2014)

362 points by 0xmohit 10y ago

edude03 10y ago
Arguable if you can keep everything on one box it will almost always be faster (and cheaper!) than any soft of distributed system. That said, scalability is generally more important than speed because once a task can be distributed you can add performance by adding hardware. As well, depending on your use case you can often get fault tolerance "for free".
- TickleSteve 10y ago
  No. Show that you need scalability first. Chances are you don't. When you do, scale the smallest possible part of your system that is the bottleneck, not the whole thing.
  - lmm 10y ago
    An established, standardised, existing platform is often more maintainable than a custom solution, even if that platform includes a bit more scalability than you actually need.
    - HelloNurse 10y ago
      But straightforward use of cat, grep, xargs and gawk is much less "custom" than depending on specific versions of exotic tools and committing servers for use as part of the current "platform". If I want to run a simple pipeline of UNIX tools on a random system, the worst case prerequisite scenario is reclaiming disk space and installing less common tools (e.g. mawk).
      - lmm 10y ago
        > But straightforward use of cat, grep, xargs and gawk is much less "custom" than depending on specific versions of exotic tools and committing servers for use as part of the current "platform". There are often subtle incompatibilities between the versions of those tools found on different systems (e.g. GNU vs BSD). Worse, there's no test suite or way to declare the dependency, and you may not even notice until the job fails halfway through. Whereas on a Spark job built with maven or similar the dependencies are at least explicit and very easy to reproduce. "Exotic" is relative. At my current job I would expect a greater proportion of my colleagues to know Spark than gawk. Unix is a platform too - and an underspecified one with poor release/version discipline at that. In an organization where Unix is standard and widely understood and Spark is not, use Unix; where the reverse applies, it's often more maintainable to use Spark, even if you don't need distribution. But my original point was: if you need a little more scalability than Unix offers then it may well be worth going all the way to Spark (even though it's far more scalability than you likely need) rather than hacking up some custom job-specific solution that scales just as far as you need, just because Spark is standard, documented and all that.
        qwertyuiop924 10y ago
        You can look for the GNU utils, you know. And you can do explicit dependancy declaration by grepping the version output to see if it's GNU or bsd, and the version number. It's not convenient, and it could be done better, but it is by no means impossible. Also, how do your colleagues not know AWK? In a primarily UNIX world, that's something that everybody should know. Besides, you can learn the basics in about 15 minutes.
        lmm 10y ago
        > You can look for the GNU utils, you know. And you can do explicit dependancy declaration by grepping the version output to see if it's GNU or bsd, and the version number. > It's not convenient, and it could be done better, but it is by no means impossible. In principle it may be possible, but in practice it's vaporware at best. There's no standard, established way to do this, which means there's no way that other people could maintain - the closest is probably autoconf, but I never saw anyone ship an aclocal.m4 with their shell script. Since unix utilities tend to be installed system-wide, it's not really practical to develop or test (not that the unix platform is at all test-friendly to start with) against old target versions (docker may eventually get there, but it's currently immature) - if your development machine has version 2.6 of some utility, you'll probably end up accidentally using 2.6-only features without noticing. > Also, how do your colleagues not know AWK? In a primarily UNIX world, that's something that everybody should know. Besides, you can learn the basics in about 15 minutes. My organization isn't primarily-unix, and people don't use awk often enough to learn it. I'm sure one could pick up the basics fairly quickly, but that's true of Spark too.
        qwertyuiop924 10y ago
        Okay, I guess. For utilities, it's usually safe to assume that whoever is running it is running a comparable version - the most commonly used options are decades old at this point, and it's relatively unlikely you'll use the new stuff. As for checking for GNU tooling, a grep against <tool> -v does the trick. This can also get you the version number. You can probably even write a command to do it. It's nonstandard and suboptimal, but it is, once again, possible.
- zzzcpan 10y ago
  There is nothing preventing distributed systems to be faster than one box for this kind of thing. But they don't always bother to pursue efficiency on that level, because things are very different once you have a lot of boxes and something that used to look important for a couple of boxes doesn't anymore.
  - kuschku 10y ago
    Yes, there is, you have a lot of overhead in any case for the same tools.
    - zzzcpan 10y ago
      You don't have the same tools. You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system. EDIT: For something that beats a single box easily I envision an interpreter with JIT running on each node in a distributed system and on the same process that stores data, having pretty much no overhead to access and process it.
      - qwertyuiop924 10y ago
        >You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system. Yeah, but Manta's mapreduce does something close, and it seems to work okay.
  - arielb1 10y ago
    Fancy highly-scalable distributed algorithms have that annoying tendency of starting at 10x slower than the most naïve single-machine algorithm.
- toast0 10y ago
  My hadoop experience is dated (circa 2011), do the work nodes still poll the scheduler to see if they have work to do? If so, that's still a giant impediment to speed for smaller tasks. Especially if poll times are in the range of minutes. If hadoop put effort into making small tasks time efficient, I think your argument has merit, if there's a reasonable chance of actually needing to scale, or to pick up ancillary benefits (fault tolerance, access to other data that needs to be processed with hadoop etc)
amelius 10y ago
But what would happen if those exact same command-line tools were used inside a Hadoop node? What would be the optimum number of processors then?
- TickleSteve 10y ago
  you would just be adding management-overhead. More software != more efficient software.
  - amelius 10y ago
    But faster because parallel.
- beagle3 10y ago
  That depends on the tradeoff between management/transfer overhead and actually doing work. Always in the "word count" style examples, but quite often in real life, the "get the data into the process" takes more time than actually processing it. When you need to distribute, you need to distribute. However, the point where "you need to distribute" is about 100x more data than the time most hadoop users do, and the overhead costs are far from negligible - in fact, they dominate everything until you get to 100x more data.
sgt101 10y ago
Ok, now do it for >2tb. Our prod hadoop dataset is now > 130tb, try that!
- Twirrim 10y ago
  Note the magic words were "can be faster", not "are faster". If you'd read the entire article you'd even have picked up that he's explicitly calling out use of hadoop for data that easily fits in memory, not large data sets.
- pessimizer 10y ago
  > Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools. If you have a huge amount of data or really need distributed processing, then tools like Hadoop may be required, but more often than not these days I see Hadoop used where a traditional relational database or other solutions would be far better in terms of performance, cost of implementation, and ongoing maintenance.
- teh 10y ago
  What kind of data do you have? Is this mostly text, or more like compressed time series?
  - cottonseed 10y ago
    We're analyzing genome sequence data on that scale: https://github.com/hail-is/hail https://github.com/hail-is/hail
  - sgt101 10y ago
    It's long rows of lots of numbers that have to be crunched with each other in quite straightforward ways.
- daemonk 10y ago
  I am a bioinformatician. 130tb of raw reads or processed data? Are you trying to build a general purpose platform for all *-seq or focusing on something specific (genotyping)?
ben_bai 10y ago
This seems to be the Manta [0] way. Letting you run your beloved Unix command pipeline on your Object Store files. [0] https://www.joyent.com/manta https://www.joyent.com/manta But the youtube videos with Bryan Cantrill are even better at explaining.
mej10 10y ago
> 1.75GB > 3.46GB These will fit in memory on modest hardware. No reason to use Hadoop. The title could be: "Using tools suitable for a problem can be 235x faster than tools unsuitable for the problem"
- TickleSteve 10y ago
  This is exactly the point he was making. People have a desire to use the 'big' tools instead of trying to solve the real problem. People both underestimate the power of their desktop machine and the 'old' tools and overestimate the size of their task.
  - cloudjacker 10y ago
    I love it when clients think they need a server workstation Specs be damned! I need to start selling boxes
  - sotojuan 10y ago
    Occasionally designers seem to seek credit merely for possessing a new technology, rather than using it to make better designs. Computers and their affiliated apparatus can do powerful things graphically, in part by turning out the hundreds of plots necessary for good data analysis. But at least a few computer graphics only evoke the response "Isn’t it remarkable that the computer can be programmed to draw like that?" instead of "My, what interesting data". - Edward Tufte Applies to more than just design.
  - cheez 10y ago
    Yeah, someone was telling me they need big data for a million rows. I laughed and said SQLite handles that...
    - matt_wulfeck 10y ago
      I would not want to be the one on-call for a million row SQLite database!
sfifs 10y ago
This is just plain click bait. Obviously if a dataset is small enough to possibly fit in memory, it will be much faster to run on a single computer.
- unethical_ban 10y ago
  I guess the author should have called it out more explicitly for some, but I think that's the point. I've seen the testimony dozens of times on HN, and I've heard it from a friend who manages Hadoop at a bank, and I've seen it with people building scaled ELK stacks for log analysis: People are too eager to scale out when things can be done locally, given moderate datasets.
  - Anderkent 10y ago
    Though sometimes hadoop makes sense even if local computation is faster. For example you might just be using hadoop for data replication.
    - gopalv 10y ago
      > For example you might just be using hadoop for data replication. Good point. The reason someone who holds data for 7+ years uses hadoop is not because it is faster. The processing aspect of the system is only tangential to the failure tolerance when you consider the age of the data set. HDFS does waste a significant amount of IO merely reading through cold data and validating the checksums, so that it safe against HDD bit-rot (dfs.datanode.scan.period.hours). The general argument about failure tolerance is off-site backups, but the backups tend to have availability problems (i.e machine failed, the Postgres backup restore takes 7 hours). The system is built for constant failure of hardware, connectivity and in some parts, the software itself (java over C++) - because those are unavoidable concerns for a distributed multi-machine system. The requirement that it be fast takes a backseat to the availability, reliability and scalability - an unreliable, but fast system is only useful for a researcher digging through data at random, not a daily data pipeline where failures cascade up.
- blakeyrat 10y ago
chajath 10y ago
It's all about picking the right tool for the job. I think shell scripting is a great prototyping tool and often a good place to start. As the problem gets more complex and bigger, eventually it will warrant a full scale development.
- astrobe_ 10y ago
  I think people overlook the fact that the author made an even more strong point by using shell scripting, which is relatively inefficient compared to using a compiled language. I guess it would hit the I/O cap without even going parallel.
qwertyuiop924 10y ago
...And when you do have 140 TB of chess data, you can move to Manta, and you get to keep your data processing pipeline almost exactly the same. Upwards scalability! I don't know how the performance would stack against Hadoop, but it'd work.
- 0xmohit 10y ago
  Manta storage service teaser: https://www.youtube.com/watch?v=d2KQ2SQLQgg https://www.youtube.com/watch?v=d2KQ2SQLQgg
- elcritch 10y ago
  Actually, just posted essentially the same thing, before reading your comment. I'm wondering as well how the performance would/will scale. It likely depends on how the data is scattered / replicated, but presumable they've worked out decent schedulers for the system. If not, it is open source! Lovin it.
  - qwertyuiop924 10y ago
    Well, all the code running against the data would already have the paralellization advantages of a shell script, as described in this article. It would additionally probably be running accross multiple nodes, meaning that the IO speeds increase the number of records that can be processe)d simultaneously. The disadvantage is that that data has to be streamed over a network to the reducer node, which could add a good chunk of latency, depending on how fast that is (if you can do some reduction during the map, it would help, but it's possible that Manta spawns one process and virtualized nods per object (and indeed, this is likely), meaning this is impossible), and how many virtual nodes are running on the same physical hardware (but then you're running into the same boundaries you hit on a laptop, just on a much beefier system), as the network latency is near zero if the reducer and the mapper nodes are on the same physical system. But if you're processing terrabytes, the network latency is probably barely factoring into your considerations, given how much time you're saving by processing data in parallel in the first place.
    - 10y ago
yelnatz 10y ago
(2014) Here's the previous comments from the submission a couple years ago: https://news.ycombinator.com/item?id=8908462 https://news.ycombinator.com/item?id=8908462
eggy 10y ago
Date on article: Sat 25 January 2014 I am not a Big Data expert, but does that change any of the comments below with reference to large datasets and memory available? I use J and Jd for fun with great speed on my meager datasets, but others have used it on billion row queries [1]. Along with q/kdb+, it was faster than Spark/Shark last I checked, however, I see Spark has made some advances recently I have not checked into. J is interpreted and can be run from the console, from a Qt interface/IDE, or in a browser with JHS. [1] http://www.jsoftware.com/jdhelp/overview.html http://www.jsoftware.com/jdhelp/overview.html
- justinsaccount 10y ago
  There isn't exactly a direct relationship between the size of the data set and the amount of memory required to process it. It depends on the specific reporting you are doing. In the case of this article, the output is 4 numbers: games, white, black, draw Processing 10 items takes the same amount of memory as processing 10 billion items. If the data set in this case was 50TB instead of a few GB, it would benefit from running the processing pipeline across many machines to increase the IO performance. You could still process everything on a single machine, it would just take longer. Some other examples of large data sets+reports that don't require a large amount of memory to process: * input: petabytes of web log data. output: count by day by country code * input: petabytes of web crawl data. output: html tags and their frequencies * input: petabytes of network flow data. output: inbound connection attempts by port Reports that require no grouping (like this chess example) or group things into buckets with a defined size (ports that are in a range of 1-65535) are easy to process on a single machine with simple data structures. Now, as soon as you start reporting over more dimensions things become harder to process on a single machine, or at least, harder to process using simple data structures. * input: petabytes of web crawl data. output: page rank * input: petabytes of network flow data. output: count of connection attempts by source address and destination port I kinda forget what point I was trying to make.. I guess.. Big data != Big report. I generated a report the other day from a few TB of log data, but the report was basically for day in /data/* ; do #YYYY-MM-DD echo $day $(zcat $day/*.gz | fgrep -c interestingValue) done
vlahmot 10y ago
I feel like every time something like this comes up people completely skip over the benefit of having as much of your data processing jobs in one ecosystem as possible. Many of our jobs operate on low TBs and growing but even if the data for a job is super small I'll write it in Hadoop (Spark these days) so that the build, deployment, and schedluing of the job is handled for free by our curent system. Sure spending more time listing files on S3 at startup than running the job is a waste but far less than the man hours to build and maintain a custom data transformation. The main benefit of these tools is not the scale or processing speed though. The main benefits are the fault tolerance, failure recovery, elasticity, and the massive ecosystem of aggregations, data types and external integrations provided by the community.
- ende 10y ago
  This is a very good point. I think many people are so caught up in bashing the hype train around big data and data science that they just casually dismiss these incredibly valid points. It's not necessarily about how big your data is right now, but how big your data will be in the very near future. Retooling then is often a lot more difficult than just tooling properly up front, so even if some Spark operations might seem to add unnecessary overhead right now the lower transition cost down the road is often worth it.
  - jacques_chester 10y ago
    Day 2 problems don't come to mind during days 0 and 1.
    - shitgoose 10y ago
      Sometimes they don't come at all.
      - shitgoose 10y ago
        That was quote from Austin Powers. And a reference to the fact that Day 2 requirements almost never come.
deadgrey19 10y ago
This idea was the subject of a paper at a major systems conference. The paper is called "Scalability! But at what cost?" - It goes well beyond this simple example above to explore how most major systems papers produce results that can be beaten by a single laptop. Here's the paper and the blog post describing it. http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
- aub3bhat 10y ago
  The paper is frankly stupid and a great example of difference between practice and academia. it looks good because they are using a snapshot of Twitter network from 2010. In reality the work flow is complex, e.g. the follower graph gets updates every hour. 10 different teams have their different requirements as to how to set up the graph and computations. These computations need to be run at different (hourly, weekly, daily) granularity. 100 downstream jobs are also dependent on them and need to start as soon as previous job finishes. The output of the jobs gets imported/indexed in database which is then pushed to production systems and/or used by analysts who might update and retry/rerun computations. Unlike a bunch of out of touch researchers the key concern isn't how "fast" calculations finish, but several others such as ability to reuse, fault tolerance, multi user support etc. I can outrun a Boeing 777 on my bike in a 3 meter race but no would care. The single laptop example is essentially that.
  - pjmlp 10y ago
    How many companies out there playing with big data are at least half of the size of Twitter?
    - aub3bhat 10y ago
      You don't need to be "half the size of Twitter". What does that even mean, in headcount, in TB stored, half of the snapshot they used? The benefits of using a distributed/hadoop style approach to managing your data assets becomes evident as soon as you have more than 5 employees who access such systems. Unless your workload is highly specific, e.g. in Deep Learning, it makes total sense to use a single machine with as many GPUs as possible. Let me clarify that I used the exact snapshot, in 2012 (here is post that was even cited by few papers [0]) , However I knew that reality of using this data was far complex, and even though you can write "faster" programs on your laptop (I used GraphLab) than a cluster (I had access to 50 nodes Cornell cluster), it didn't mean much. [0] https://scholar.google.com/citations?view_op=view_citation&hl=en&user=udmAQC4AAAAJ&citation_for_view=udmAQC4AAAAJ:W7OEmFMy1HYC https://scholar.google.com/citations?view_op=view_citation&h...
cs702 10y ago
Most "big data" problems are really "small data" by the standards of modern hardware: * Desktop PCs with up to 6TB of RAM and many dozens of cores have been available for over a year.[1] * Hard drives with 100TB capacity in a 3.5-inch form factor were recently announced.[2] CORRECTION: THE FIGURE IS 60TB, NOT 100TB. See MagnumOpus's comment below. In a haste, I searched Google and mistakengly linked to an April Fool's story. Now I feel like a fool, of course. Still, the point is valid. * Four Nvidia Titan X GPUs can give you up to 44 Teraflops of 32-bit FP computing power in a single desktop.[3] Despite this, the number of people who have unnecessarily spent money and/or complicated their lives with tools like Hadoop is pretty large, particularly in "enterprise" environments. A lot of "big data" problems can be handled by a single souped-up machine that fits under your desk. [1] http://www.alphr.com/news/enterprise/387196/intel-xeon-e7-v2-servers-support-6tb-of-ram http://www.alphr.com/news/enterprise/387196/intel-xeon-e7-v2... [2] http://www.storagenewsletter.com/rubriques/hard-disk-drives/incredible-record-of-100tb-into-3-5-inch-hdd-with-hamrhelium/ http://www.storagenewsletter.com/rubriques/hard-disk-drives/... [3] https://news.ycombinator.com/item?id=12141334 https://news.ycombinator.com/item?id=12141334
- rdtsc 10y ago
  Well basically as soon as big data hit the news everyone stopped doing data and started doing big data. Same thing with data science. Opened a Jupyter notebook, loaded 100 rows, and displayed a graph - "I am a data scientist".
- MagnumOpus 10y ago
  > Hard drives with 100TB capacity in a 3.5-inch form factor were recently announced That is an April Fools story. (Of course you can still get a Synology DS2411+/DX1211 24-bay NAS combo for a few thousand bucks, but it will take up a lot of space under your desk and keep your legs toasty...)
  - onion2k 10y ago
    Putting a 100TB of storage on your network isn't hard though. There are off-the-shelf NAS servers that big - eg http://www.ebuyer.com/671448-qnap-ts-ec2480u-rp-144tb-24-x-6tb-hitach-ultrastar-24-bay-4u-nas-ts-ec2480u-rp-144tb-us http://www.ebuyer.com/671448-qnap-ts-ec2480u-rp-144tb-24-x-6...
aub3bhat 10y ago
This article is a great litmus test for checking if someone has experience working at scale (Multi Terabytes, Multiple analysts, Multiple job types) or not. Anyone who has had that experience will instantly describe why this article is wrong. It's akin to saying a Tesla is faster than Boeing 777 on a 100 meter track.
- detaro 10y ago
  I'd hope people who have worked at scale still are capable of recognizing when the tools they used there are totally overkill. I'd suspect they would, since they'd also be more aware of their limitations (vs somebody without experience, who has to believe the "you need big data and everything is easy" marketing). That you wouldn't use a Boing 777 IF your problem is just a 100m track is the entire point of the article. It's explicitly not saying that you never should use the big tools.
  - aub3bhat 10y ago
    They are not overkill at all, rather they are tuned towards different set of performance characteristics. E.g. in the Boeing 777 example above, transatlantic journey. In the article above, the data and results stay on the local disk, however in any organization, they need to be stored in a distributed manner, available to multiple users with varying levels of technical expertise. Typically in NFS or HDFS, preferably if they are records stored/indexed via Hive/Presto. At which point the real issue is how do you reduce the delay resulting from transferring data over the network. Which is what the original idea (moving computation closer to data) behind Hadoop/MapReduce.
    - qwertyuiop924 10y ago
      rolls eyes The point is that if you've got such tiny quantities of data, why are you storing it in a distributed manner, and why are you breaking out the 777 for a trip around the racetrack? Grab the 777 when you need it, and take the Tesla when you need the performance characteristics of a Tesla.
nekopa 10y ago
Reminds me of one of my all time favorite comments about Bane's Rule: https://news.ycombinator.com/item?id=8902739 https://news.ycombinator.com/item?id=8902739
- banku_brougham 10y ago
  thanks for sharing, this is exactly what i've been wondering about
- matt_wulfeck 10y ago
  And yet a fourth dimension to the problem: time/difficulty multiplier. Going Multiplier ----- --------- High 1x Wide 4x Deep 8x Make the best decision given your current circumstances and engineering resources.
  - combatentropy 10y ago
    Bane's "deep" approach saved tens of millions of dollars in one example, months of time in another. In all cases it was many times easier for the team to keep up, day after day, year after year. Bane advocates good old-fashioned refactoring. With humble tools like Perl, SQL, and an old desktop, he bested new, fancy, expensive products. Bane deserves the salary of a CEO, or at least a vice president, for the good he has brought to his company. I think ego leads us to make choices that appeal in the short run but are bad in the long run. Which is more impressive sounding: that you bought a distributed network of a thousand of the newest, shiniest machines, running the latest version of DataLaserSharkAttack; or that you cobbled together some Perl, SQL, and shell one-liners on a four-year-old PC? Also good-old fashioned hard work is painful. It is a good kind of pain, like working out your body, rather than a bad kind of pain, like accidentally cutting yourself. But it is just good old-fashioned, humble, hard work to sit down, work through the details, and come up with a better plan. Before that, it is even more humble, hard work, to learn the things that Bane had learned. Not just anybody could have done what he did. First you have to learn the ins and outs of Perl, SQL, and all the little shell commands, and all their little options. He knows about a lot of different programming problems, like what a "semantic graph" is (I can't say I do), what an "adjacency matrix" is (nope), whether something is an O(N^2) problem or an O(k+n^2) problem (I know I've seen that notation before).
banku_brougham 10y ago
i agree that scalable infrastructure is needed to manage a production pipeline, as others have explained well. i found this article was a useful reminder, because sometimes a job doesnt require a fully grown infrastructure. i commonly get these requests that dont overlap with existing infrastructure and wont need any followup. in that particular case a hadoop cluster, heck even loading into a pg db would be wasted effort. but i wouldnt want to manage our clickstream analytics pipeline with shell scripts and cron jobs. is there any lightweight tooling out there that can schedule/run basic pipeline jobs in a shell environment?
- hcrisp 10y ago
  Airflow? It might not be what you consider lightweight, though.
- qwertyuiop924 10y ago
  Manta? Definitely not lightweight.
dpcx 10y ago
The Hadoop article linked is available at https://web.archive.org/web/20140119221101/http://tomhayden3.com/2013/12/27/chess-mr-job/ https://web.archive.org/web/20140119221101/http://tomhayden3...?
matt_wulfeck 10y ago
There's a lot of operational benefits to running on Hadoop/yarn as well. You get operational benefits from node resiliency (host went down? Run the application over there). You also get the Hadoop filesystem which conveniently stores your data in S3 and distributed HDFS. These systems were designed by people who probably managed difficult etl pipelines that were nothing but what the author suggests: simplified shell scripts using UNIX pipes. Besides going up against Hadoop MR is easy. I'd like to see you compete against something like Facebook's presto or spark which are optimized for network and memory.
regularfry 10y ago
In my experience you can classify people into two herds: those who, when faced with a problem, solve it directly; and those who, faced with the same problem, try to fit it to the tools they want to use. I like to think this is a maturity question, but I can't think I've actually seen someone make the transition from the latter to the former type.
- reflexive 10y ago
  > those who, when faced with a problem, solve it directly I think you mean "use the tools they already know". > those who, faced with the same problem, try to fit it to the tools they want to use I think you mean "use the correct tools for the job". > I like to think this is a maturity question It is, but the direction of maturity is from the first case to the second.
bobivl 10y ago
If you like to do data analyses in bash, you might also enjoy bigbash[1]. This tool generates quite performant bash one-liners from SQL Select statements that easily crunch GB of csv data. Disclaimer: I am the main author. [1] http://bigbash.it http://bigbash.it
- qwertyuiop924 10y ago
  That's pretty cool. Do you think you can get it to support Manta? I think a lot of people in that ecosystem could benefit from it if you could. I'd help, but I don't really know Java all that well :-(.
zobzu 10y ago
Not just faster to run but also much faster to write
cottonseed 10y ago
Is there a place to download the database that isn't a self-extracting executable? (Seriously?)
linuxhansl 10y ago
Yes, you do not need a 100 node cluster to crunch 1.75GB of data. I can do that on my phone. What's the author's point?!
- danudey 10y ago
  That hadoop's reputation of being worth the hassle and complexity that managing it entails is undeserved.
RoxetteGirl 10y ago
Both disk and CPU failures are recoverable on expensive hardware
deleted 10y ago
[deleted]
eva1984 10y ago
What is the point? Who would want to use Hadoop for something below 10GB? Hadoop is not good at doing what it is not designed for? How useful.
- virmundi 10y ago
  Kind of depends on what the 10 GB is. For example, on my project, we started on files that were about 10 GB a day. The old system took 9 hours to enhance the data (add columns from other sources based on simple joins). So we did it with Hadoop on two Solaris boxes (18 virtual cores between them). Same data; 45 minutes. But wait there's more. We then created a two fraud models that took that 10+ GB file (enhancement added about 10%) and executed within about 30 minutes a piece. But concurrently. All on Hadoop. All on arguably terrible hardware. Folks at Twitter and Facebook had never though about using Solaris. We've continued this pattern. We've switched tooling from Pig to Cascading because Cascading works in the small (your PC without a cluster) and in the large. It's testable with JUnit in a timely manner (looking at you PigUnit). Now we have some 70 fraud models chewing over anywhere from that 10+ GB daily file set to 3 TB. All this in our little 50 node cluster. All within about 14 hours. Total processed data is about 50 TB a day. As pointed out earlier, Hadoop provides an efficient, scalable, easy distributed application development platform. Cascading makes life very Unix-like (pipes and filters and aggregators). This coupled with a fully async eventing pipe line for workflows built on RabbitMQ makes for an infinitely expandable fraud detection system. Since all processors communicate only through events and HDFS, we add new fraud models without necessarily dropping the entire system. New models may arrive daily, conform to a set structure, and are literally self-installed from a zip file within about 1 minute. We used the same event + Hadoop architecture to add claim line edits. These are different from fraud models in that fraud models calculate multiple complex attributes then apply a heuristic to the claim lines. Edits look at a smaller operation scope. But in cascading this is pipe from HDFS -> filter for interesting claim lines -> buffer for denials -> pipe to HDFS output location. Simple, scalable, logical, improvable, testable. I've seen all of these. As the community comes out with new tools, we get more options. My team is working on machine learning and graphs. Mahout and Giraph. Hard to do all of this easily with a home grown data processing system. As always, research your needs. Don't get caught up in the hype of a new trend. Don't be closed minded either.
necessity 10y ago
>cat | grep why

362 points by 0xmohit 10y ago

Arguable if you can keep everything on one box it will almost always be faster (and cheaper!) than any soft of distributed system. That said, scalability is generally more important than speed because once a task can be distributed you can add performance by adding hardware. As well, depending on your use case you can often get fault tolerance "for free".

TickleSteve 10y ago
No. Show that you need scalability first. Chances are you don't. When you do, scale the smallest possible part of your system that is the bottleneck, not the whole thing.
- lmm 10y ago
  An established, standardised, existing platform is often more maintainable than a custom solution, even if that platform includes a bit more scalability than you actually need.
  - HelloNurse 10y ago
    But straightforward use of cat, grep, xargs and gawk is much less "custom" than depending on specific versions of exotic tools and committing servers for use as part of the current "platform". If I want to run a simple pipeline of UNIX tools on a random system, the worst case prerequisite scenario is reclaiming disk space and installing less common tools (e.g. mawk).
    - lmm 10y ago
      > But straightforward use of cat, grep, xargs and gawk is much less "custom" than depending on specific versions of exotic tools and committing servers for use as part of the current "platform". There are often subtle incompatibilities between the versions of those tools found on different systems (e.g. GNU vs BSD). Worse, there's no test suite or way to declare the dependency, and you may not even notice until the job fails halfway through. Whereas on a Spark job built with maven or similar the dependencies are at least explicit and very easy to reproduce. "Exotic" is relative. At my current job I would expect a greater proportion of my colleagues to know Spark than gawk. Unix is a platform too - and an underspecified one with poor release/version discipline at that. In an organization where Unix is standard and widely understood and Spark is not, use Unix; where the reverse applies, it's often more maintainable to use Spark, even if you don't need distribution. But my original point was: if you need a little more scalability than Unix offers then it may well be worth going all the way to Spark (even though it's far more scalability than you likely need) rather than hacking up some custom job-specific solution that scales just as far as you need, just because Spark is standard, documented and all that.
      - qwertyuiop924 10y ago
        You can look for the GNU utils, you know. And you can do explicit dependancy declaration by grepping the version output to see if it's GNU or bsd, and the version number. It's not convenient, and it could be done better, but it is by no means impossible. Also, how do your colleagues not know AWK? In a primarily UNIX world, that's something that everybody should know. Besides, you can learn the basics in about 15 minutes.
        lmm 10y ago
        > You can look for the GNU utils, you know. And you can do explicit dependancy declaration by grepping the version output to see if it's GNU or bsd, and the version number. > It's not convenient, and it could be done better, but it is by no means impossible. In principle it may be possible, but in practice it's vaporware at best. There's no standard, established way to do this, which means there's no way that other people could maintain - the closest is probably autoconf, but I never saw anyone ship an aclocal.m4 with their shell script. Since unix utilities tend to be installed system-wide, it's not really practical to develop or test (not that the unix platform is at all test-friendly to start with) against old target versions (docker may eventually get there, but it's currently immature) - if your development machine has version 2.6 of some utility, you'll probably end up accidentally using 2.6-only features without noticing. > Also, how do your colleagues not know AWK? In a primarily UNIX world, that's something that everybody should know. Besides, you can learn the basics in about 15 minutes. My organization isn't primarily-unix, and people don't use awk often enough to learn it. I'm sure one could pick up the basics fairly quickly, but that's true of Spark too.
        qwertyuiop924 10y ago
        Okay, I guess. For utilities, it's usually safe to assume that whoever is running it is running a comparable version - the most commonly used options are decades old at this point, and it's relatively unlikely you'll use the new stuff. As for checking for GNU tooling, a grep against <tool> -v does the trick. This can also get you the version number. You can probably even write a command to do it. It's nonstandard and suboptimal, but it is, once again, possible.
zzzcpan 10y ago
There is nothing preventing distributed systems to be faster than one box for this kind of thing. But they don't always bother to pursue efficiency on that level, because things are very different once you have a lot of boxes and something that used to look important for a couple of boxes doesn't anymore.
- kuschku 10y ago
  Yes, there is, you have a lot of overhead in any case for the same tools.
  - zzzcpan 10y ago
    You don't have the same tools. You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system. EDIT: For something that beats a single box easily I envision an interpreter with JIT running on each node in a distributed system and on the same process that stores data, having pretty much no overhead to access and process it.
    - qwertyuiop924 10y ago
      >You are probably thinking about emulating POSIX filesystem API and things like that and using those command-line tools on top of that in a single-box kind of way. That's not how you treat your distributed system. Yeah, but Manta's mapreduce does something close, and it seems to work okay.
- arielb1 10y ago
  Fancy highly-scalable distributed algorithms have that annoying tendency of starting at 10x slower than the most naïve single-machine algorithm.
toast0 10y ago
My hadoop experience is dated (circa 2011), do the work nodes still poll the scheduler to see if they have work to do? If so, that's still a giant impediment to speed for smaller tasks. Especially if poll times are in the range of minutes. If hadoop put effort into making small tasks time efficient, I think your argument has merit, if there's a reasonable chance of actually needing to scale, or to pick up ancillary benefits (fault tolerance, access to other data that needs to be processed with hadoop etc)

amelius 10y ago

But what would happen if those exact same command-line tools were used inside a Hadoop node? What would be the optimum number of processors then?

TickleSteve 10y ago
you would just be adding management-overhead. More software != more efficient software.
- amelius 10y ago
  But faster because parallel.
beagle3 10y ago
That depends on the tradeoff between management/transfer overhead and actually doing work. Always in the "word count" style examples, but quite often in real life, the "get the data into the process" takes more time than actually processing it. When you need to distribute, you need to distribute. However, the point where "you need to distribute" is about 100x more data than the time most hadoop users do, and the overhead costs are far from negligible - in fact, they dominate everything until you get to 100x more data.

sgt101 10y ago

Ok, now do it for >2tb. Our prod hadoop dataset is now > 130tb, try that!

Twirrim 10y ago
Note the magic words were "can be faster", not "are faster". If you'd read the entire article you'd even have picked up that he's explicitly calling out use of hadoop for data that easily fits in memory, not large data sets.
pessimizer 10y ago
> Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools. If you have a huge amount of data or really need distributed processing, then tools like Hadoop may be required, but more often than not these days I see Hadoop used where a traditional relational database or other solutions would be far better in terms of performance, cost of implementation, and ongoing maintenance.
teh 10y ago
What kind of data do you have? Is this mostly text, or more like compressed time series?
- cottonseed 10y ago
  We're analyzing genome sequence data on that scale: https://github.com/hail-is/hail https://github.com/hail-is/hail
- sgt101 10y ago
  It's long rows of lots of numbers that have to be crunched with each other in quite straightforward ways.
daemonk 10y ago
I am a bioinformatician. 130tb of raw reads or processed data? Are you trying to build a general purpose platform for all *-seq or focusing on something specific (genotyping)?

ben_bai 10y ago

This seems to be the Manta [0] way. Letting you run your beloved Unix command pipeline on your Object Store files. [0] https://www.joyent.com/manta https://www.joyent.com/manta But the youtube videos with Bryan Cantrill are even better at explaining.

mej10 10y ago

> 1.75GB > 3.46GB These will fit in memory on modest hardware. No reason to use Hadoop. The title could be: "Using tools suitable for a problem can be 235x faster than tools unsuitable for the problem"

TickleSteve 10y ago
This is exactly the point he was making. People have a desire to use the 'big' tools instead of trying to solve the real problem. People both underestimate the power of their desktop machine and the 'old' tools and overestimate the size of their task.
- cloudjacker 10y ago
  I love it when clients think they need a server workstation Specs be damned! I need to start selling boxes
- sotojuan 10y ago
  Occasionally designers seem to seek credit merely for possessing a new technology, rather than using it to make better designs. Computers and their affiliated apparatus can do powerful things graphically, in part by turning out the hundreds of plots necessary for good data analysis. But at least a few computer graphics only evoke the response "Isn’t it remarkable that the computer can be programmed to draw like that?" instead of "My, what interesting data". - Edward Tufte Applies to more than just design.
- cheez 10y ago
  Yeah, someone was telling me they need big data for a million rows. I laughed and said SQLite handles that...
  - matt_wulfeck 10y ago
    I would not want to be the one on-call for a million row SQLite database!

sfifs 10y ago

This is just plain click bait. Obviously if a dataset is small enough to possibly fit in memory, it will be much faster to run on a single computer.

unethical_ban 10y ago
I guess the author should have called it out more explicitly for some, but I think that's the point. I've seen the testimony dozens of times on HN, and I've heard it from a friend who manages Hadoop at a bank, and I've seen it with people building scaled ELK stacks for log analysis: People are too eager to scale out when things can be done locally, given moderate datasets.
- Anderkent 10y ago
  Though sometimes hadoop makes sense even if local computation is faster. For example you might just be using hadoop for data replication.
  - gopalv 10y ago
    > For example you might just be using hadoop for data replication. Good point. The reason someone who holds data for 7+ years uses hadoop is not because it is faster. The processing aspect of the system is only tangential to the failure tolerance when you consider the age of the data set. HDFS does waste a significant amount of IO merely reading through cold data and validating the checksums, so that it safe against HDD bit-rot (dfs.datanode.scan.period.hours). The general argument about failure tolerance is off-site backups, but the backups tend to have availability problems (i.e machine failed, the Postgres backup restore takes 7 hours). The system is built for constant failure of hardware, connectivity and in some parts, the software itself (java over C++) - because those are unavoidable concerns for a distributed multi-machine system. The requirement that it be fast takes a backseat to the availability, reliability and scalability - an unreliable, but fast system is only useful for a researcher digging through data at random, not a daily data pipeline where failures cascade up.
blakeyrat 10y ago

chajath 10y ago

It's all about picking the right tool for the job. I think shell scripting is a great prototyping tool and often a good place to start. As the problem gets more complex and bigger, eventually it will warrant a full scale development.

astrobe_ 10y ago
I think people overlook the fact that the author made an even more strong point by using shell scripting, which is relatively inefficient compared to using a compiled language. I guess it would hit the I/O cap without even going parallel.

qwertyuiop924 10y ago

...And when you do have 140 TB of chess data, you can move to Manta, and you get to keep your data processing pipeline almost exactly the same. Upwards scalability! I don't know how the performance would stack against Hadoop, but it'd work.

0xmohit 10y ago
Manta storage service teaser: https://www.youtube.com/watch?v=d2KQ2SQLQgg https://www.youtube.com/watch?v=d2KQ2SQLQgg
elcritch 10y ago
Actually, just posted essentially the same thing, before reading your comment. I'm wondering as well how the performance would/will scale. It likely depends on how the data is scattered / replicated, but presumable they've worked out decent schedulers for the system. If not, it is open source! Lovin it.
- qwertyuiop924 10y ago
  Well, all the code running against the data would already have the paralellization advantages of a shell script, as described in this article. It would additionally probably be running accross multiple nodes, meaning that the IO speeds increase the number of records that can be processe)d simultaneously. The disadvantage is that that data has to be streamed over a network to the reducer node, which could add a good chunk of latency, depending on how fast that is (if you can do some reduction during the map, it would help, but it's possible that Manta spawns one process and virtualized nods per object (and indeed, this is likely), meaning this is impossible), and how many virtual nodes are running on the same physical hardware (but then you're running into the same boundaries you hit on a laptop, just on a much beefier system), as the network latency is near zero if the reducer and the mapper nodes are on the same physical system. But if you're processing terrabytes, the network latency is probably barely factoring into your considerations, given how much time you're saving by processing data in parallel in the first place.
  - 10y ago

yelnatz 10y ago

(2014) Here's the previous comments from the submission a couple years ago: https://news.ycombinator.com/item?id=8908462 https://news.ycombinator.com/item?id=8908462

eggy 10y ago

Date on article: Sat 25 January 2014 I am not a Big Data expert, but does that change any of the comments below with reference to large datasets and memory available? I use J and Jd for fun with great speed on my meager datasets, but others have used it on billion row queries [1]. Along with q/kdb+, it was faster than Spark/Shark last I checked, however, I see Spark has made some advances recently I have not checked into. J is interpreted and can be run from the console, from a Qt interface/IDE, or in a browser with JHS. [1] http://www.jsoftware.com/jdhelp/overview.html http://www.jsoftware.com/jdhelp/overview.html

justinsaccount 10y ago
There isn't exactly a direct relationship between the size of the data set and the amount of memory required to process it. It depends on the specific reporting you are doing. In the case of this article, the output is 4 numbers: games, white, black, draw Processing 10 items takes the same amount of memory as processing 10 billion items. If the data set in this case was 50TB instead of a few GB, it would benefit from running the processing pipeline across many machines to increase the IO performance. You could still process everything on a single machine, it would just take longer. Some other examples of large data sets+reports that don't require a large amount of memory to process: * input: petabytes of web log data. output: count by day by country code * input: petabytes of web crawl data. output: html tags and their frequencies * input: petabytes of network flow data. output: inbound connection attempts by port Reports that require no grouping (like this chess example) or group things into buckets with a defined size (ports that are in a range of 1-65535) are easy to process on a single machine with simple data structures. Now, as soon as you start reporting over more dimensions things become harder to process on a single machine, or at least, harder to process using simple data structures. * input: petabytes of web crawl data. output: page rank * input: petabytes of network flow data. output: count of connection attempts by source address and destination port I kinda forget what point I was trying to make.. I guess.. Big data != Big report. I generated a report the other day from a few TB of log data, but the report was basically for day in /data/* ; do #YYYY-MM-DD echo $day $(zcat $day/*.gz | fgrep -c interestingValue) done

vlahmot 10y ago

I feel like every time something like this comes up people completely skip over the benefit of having as much of your data processing jobs in one ecosystem as possible. Many of our jobs operate on low TBs and growing but even if the data for a job is super small I'll write it in Hadoop (Spark these days) so that the build, deployment, and schedluing of the job is handled for free by our curent system. Sure spending more time listing files on S3 at startup than running the job is a waste but far less than the man hours to build and maintain a custom data transformation. The main benefit of these tools is not the scale or processing speed though. The main benefits are the fault tolerance, failure recovery, elasticity, and the massive ecosystem of aggregations, data types and external integrations provided by the community.

ende 10y ago
This is a very good point. I think many people are so caught up in bashing the hype train around big data and data science that they just casually dismiss these incredibly valid points. It's not necessarily about how big your data is right now, but how big your data will be in the very near future. Retooling then is often a lot more difficult than just tooling properly up front, so even if some Spark operations might seem to add unnecessary overhead right now the lower transition cost down the road is often worth it.
- jacques_chester 10y ago
  Day 2 problems don't come to mind during days 0 and 1.
  - shitgoose 10y ago
    Sometimes they don't come at all.
    - shitgoose 10y ago
      That was quote from Austin Powers. And a reference to the fact that Day 2 requirements almost never come.

deadgrey19 10y ago

This idea was the subject of a paper at a major systems conference. The paper is called "Scalability! But at what cost?" - It goes well beyond this simple example above to explore how most major systems papers produce results that can be beaten by a single laptop. Here's the paper and the blog post describing it. http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

aub3bhat 10y ago
The paper is frankly stupid and a great example of difference between practice and academia. it looks good because they are using a snapshot of Twitter network from 2010. In reality the work flow is complex, e.g. the follower graph gets updates every hour. 10 different teams have their different requirements as to how to set up the graph and computations. These computations need to be run at different (hourly, weekly, daily) granularity. 100 downstream jobs are also dependent on them and need to start as soon as previous job finishes. The output of the jobs gets imported/indexed in database which is then pushed to production systems and/or used by analysts who might update and retry/rerun computations. Unlike a bunch of out of touch researchers the key concern isn't how "fast" calculations finish, but several others such as ability to reuse, fault tolerance, multi user support etc. I can outrun a Boeing 777 on my bike in a 3 meter race but no would care. The single laptop example is essentially that.
- pjmlp 10y ago
  How many companies out there playing with big data are at least half of the size of Twitter?
  - aub3bhat 10y ago
    You don't need to be "half the size of Twitter". What does that even mean, in headcount, in TB stored, half of the snapshot they used? The benefits of using a distributed/hadoop style approach to managing your data assets becomes evident as soon as you have more than 5 employees who access such systems. Unless your workload is highly specific, e.g. in Deep Learning, it makes total sense to use a single machine with as many GPUs as possible. Let me clarify that I used the exact snapshot, in 2012 (here is post that was even cited by few papers [0]) , However I knew that reality of using this data was far complex, and even though you can write "faster" programs on your laptop (I used GraphLab) than a cluster (I had access to 50 nodes Cornell cluster), it didn't mean much. [0] https://scholar.google.com/citations?view_op=view_citation&hl=en&user=udmAQC4AAAAJ&citation_for_view=udmAQC4AAAAJ:W7OEmFMy1HYC https://scholar.google.com/citations?view_op=view_citation&h...

cs702 10y ago

Most "big data" problems are really "small data" by the standards of modern hardware: * Desktop PCs with up to 6TB of RAM and many dozens of cores have been available for over a year.[1] * Hard drives with 100TB capacity in a 3.5-inch form factor were recently announced.[2] CORRECTION: THE FIGURE IS 60TB, NOT 100TB. See MagnumOpus's comment below. In a haste, I searched Google and mistakengly linked to an April Fool's story. Now I feel like a fool, of course. Still, the point is valid. * Four Nvidia Titan X GPUs can give you up to 44 Teraflops of 32-bit FP computing power in a single desktop.[3] Despite this, the number of people who have unnecessarily spent money and/or complicated their lives with tools like Hadoop is pretty large, particularly in "enterprise" environments. A lot of "big data" problems can be handled by a single souped-up machine that fits under your desk. [1] http://www.alphr.com/news/enterprise/387196/intel-xeon-e7-v2-servers-support-6tb-of-ram http://www.alphr.com/news/enterprise/387196/intel-xeon-e7-v2... [2] http://www.storagenewsletter.com/rubriques/hard-disk-drives/incredible-record-of-100tb-into-3-5-inch-hdd-with-hamrhelium/ http://www.storagenewsletter.com/rubriques/hard-disk-drives/... [3] https://news.ycombinator.com/item?id=12141334 https://news.ycombinator.com/item?id=12141334

rdtsc 10y ago
Well basically as soon as big data hit the news everyone stopped doing data and started doing big data. Same thing with data science. Opened a Jupyter notebook, loaded 100 rows, and displayed a graph - "I am a data scientist".
MagnumOpus 10y ago
> Hard drives with 100TB capacity in a 3.5-inch form factor were recently announced That is an April Fools story. (Of course you can still get a Synology DS2411+/DX1211 24-bay NAS combo for a few thousand bucks, but it will take up a lot of space under your desk and keep your legs toasty...)
- onion2k 10y ago
  Putting a 100TB of storage on your network isn't hard though. There are off-the-shelf NAS servers that big - eg http://www.ebuyer.com/671448-qnap-ts-ec2480u-rp-144tb-24-x-6tb-hitach-ultrastar-24-bay-4u-nas-ts-ec2480u-rp-144tb-us http://www.ebuyer.com/671448-qnap-ts-ec2480u-rp-144tb-24-x-6...

aub3bhat 10y ago

This article is a great litmus test for checking if someone has experience working at scale (Multi Terabytes, Multiple analysts, Multiple job types) or not. Anyone who has had that experience will instantly describe why this article is wrong. It's akin to saying a Tesla is faster than Boeing 777 on a 100 meter track.

detaro 10y ago
I'd hope people who have worked at scale still are capable of recognizing when the tools they used there are totally overkill. I'd suspect they would, since they'd also be more aware of their limitations (vs somebody without experience, who has to believe the "you need big data and everything is easy" marketing). That you wouldn't use a Boing 777 IF your problem is just a 100m track is the entire point of the article. It's explicitly not saying that you never should use the big tools.
- aub3bhat 10y ago
  They are not overkill at all, rather they are tuned towards different set of performance characteristics. E.g. in the Boeing 777 example above, transatlantic journey. In the article above, the data and results stay on the local disk, however in any organization, they need to be stored in a distributed manner, available to multiple users with varying levels of technical expertise. Typically in NFS or HDFS, preferably if they are records stored/indexed via Hive/Presto. At which point the real issue is how do you reduce the delay resulting from transferring data over the network. Which is what the original idea (moving computation closer to data) behind Hadoop/MapReduce.
  - qwertyuiop924 10y ago
    rolls eyes The point is that if you've got such tiny quantities of data, why are you storing it in a distributed manner, and why are you breaking out the 777 for a trip around the racetrack? Grab the 777 when you need it, and take the Tesla when you need the performance characteristics of a Tesla.

nekopa 10y ago

Reminds me of one of my all time favorite comments about Bane's Rule: https://news.ycombinator.com/item?id=8902739 https://news.ycombinator.com/item?id=8902739

banku_brougham 10y ago
thanks for sharing, this is exactly what i've been wondering about
matt_wulfeck 10y ago
And yet a fourth dimension to the problem: time/difficulty multiplier. Going Multiplier ----- --------- High 1x Wide 4x Deep 8x Make the best decision given your current circumstances and engineering resources.
- combatentropy 10y ago
  Bane's "deep" approach saved tens of millions of dollars in one example, months of time in another. In all cases it was many times easier for the team to keep up, day after day, year after year. Bane advocates good old-fashioned refactoring. With humble tools like Perl, SQL, and an old desktop, he bested new, fancy, expensive products. Bane deserves the salary of a CEO, or at least a vice president, for the good he has brought to his company. I think ego leads us to make choices that appeal in the short run but are bad in the long run. Which is more impressive sounding: that you bought a distributed network of a thousand of the newest, shiniest machines, running the latest version of DataLaserSharkAttack; or that you cobbled together some Perl, SQL, and shell one-liners on a four-year-old PC? Also good-old fashioned hard work is painful. It is a good kind of pain, like working out your body, rather than a bad kind of pain, like accidentally cutting yourself. But it is just good old-fashioned, humble, hard work to sit down, work through the details, and come up with a better plan. Before that, it is even more humble, hard work, to learn the things that Bane had learned. Not just anybody could have done what he did. First you have to learn the ins and outs of Perl, SQL, and all the little shell commands, and all their little options. He knows about a lot of different programming problems, like what a "semantic graph" is (I can't say I do), what an "adjacency matrix" is (nope), whether something is an O(N^2) problem or an O(k+n^2) problem (I know I've seen that notation before).

banku_brougham 10y ago

i agree that scalable infrastructure is needed to manage a production pipeline, as others have explained well. i found this article was a useful reminder, because sometimes a job doesnt require a fully grown infrastructure. i commonly get these requests that dont overlap with existing infrastructure and wont need any followup. in that particular case a hadoop cluster, heck even loading into a pg db would be wasted effort. but i wouldnt want to manage our clickstream analytics pipeline with shell scripts and cron jobs. is there any lightweight tooling out there that can schedule/run basic pipeline jobs in a shell environment?

hcrisp 10y ago
Airflow? It might not be what you consider lightweight, though.
qwertyuiop924 10y ago
Manta? Definitely not lightweight.

dpcx 10y ago

The Hadoop article linked is available at https://web.archive.org/web/20140119221101/http://tomhayden3.com/2013/12/27/chess-mr-job/ https://web.archive.org/web/20140119221101/http://tomhayden3...?

matt_wulfeck 10y ago

There's a lot of operational benefits to running on Hadoop/yarn as well. You get operational benefits from node resiliency (host went down? Run the application over there). You also get the Hadoop filesystem which conveniently stores your data in S3 and distributed HDFS. These systems were designed by people who probably managed difficult etl pipelines that were nothing but what the author suggests: simplified shell scripts using UNIX pipes. Besides going up against Hadoop MR is easy. I'd like to see you compete against something like Facebook's presto or spark which are optimized for network and memory.

regularfry 10y ago

In my experience you can classify people into two herds: those who, when faced with a problem, solve it directly; and those who, faced with the same problem, try to fit it to the tools they want to use. I like to think this is a maturity question, but I can't think I've actually seen someone make the transition from the latter to the former type.

reflexive 10y ago
> those who, when faced with a problem, solve it directly I think you mean "use the tools they already know". > those who, faced with the same problem, try to fit it to the tools they want to use I think you mean "use the correct tools for the job". > I like to think this is a maturity question It is, but the direction of maturity is from the first case to the second.

bobivl 10y ago

If you like to do data analyses in bash, you might also enjoy bigbash[1]. This tool generates quite performant bash one-liners from SQL Select statements that easily crunch GB of csv data. Disclaimer: I am the main author. [1] http://bigbash.it http://bigbash.it

qwertyuiop924 10y ago
That's pretty cool. Do you think you can get it to support Manta? I think a lot of people in that ecosystem could benefit from it if you could. I'd help, but I don't really know Java all that well :-(.

zobzu 10y ago

Not just faster to run but also much faster to write

cottonseed 10y ago

Is there a place to download the database that isn't a self-extracting executable? (Seriously?)

linuxhansl 10y ago

Yes, you do not need a 100 node cluster to crunch 1.75GB of data. I can do that on my phone. What's the author's point?!

danudey 10y ago
That hadoop's reputation of being worth the hassle and complexity that managing it entails is undeserved.

RoxetteGirl 10y ago

Both disk and CPU failures are recoverable on expensive hardware

deleted 10y ago

[deleted]

eva1984 10y ago

What is the point? Who would want to use Hadoop for something below 10GB? Hadoop is not good at doing what it is not designed for? How useful.

virmundi 10y ago
Kind of depends on what the 10 GB is. For example, on my project, we started on files that were about 10 GB a day. The old system took 9 hours to enhance the data (add columns from other sources based on simple joins). So we did it with Hadoop on two Solaris boxes (18 virtual cores between them). Same data; 45 minutes. But wait there's more. We then created a two fraud models that took that 10+ GB file (enhancement added about 10%) and executed within about 30 minutes a piece. But concurrently. All on Hadoop. All on arguably terrible hardware. Folks at Twitter and Facebook had never though about using Solaris. We've continued this pattern. We've switched tooling from Pig to Cascading because Cascading works in the small (your PC without a cluster) and in the large. It's testable with JUnit in a timely manner (looking at you PigUnit). Now we have some 70 fraud models chewing over anywhere from that 10+ GB daily file set to 3 TB. All this in our little 50 node cluster. All within about 14 hours. Total processed data is about 50 TB a day. As pointed out earlier, Hadoop provides an efficient, scalable, easy distributed application development platform. Cascading makes life very Unix-like (pipes and filters and aggregators). This coupled with a fully async eventing pipe line for workflows built on RabbitMQ makes for an infinitely expandable fraud detection system. Since all processors communicate only through events and HDFS, we add new fraud models without necessarily dropping the entire system. New models may arrive daily, conform to a set structure, and are literally self-installed from a zip file within about 1 minute. We used the same event + Hadoop architecture to add claim line edits. These are different from fraud models in that fraud models calculate multiple complex attributes then apply a heuristic to the claim lines. Edits look at a smaller operation scope. But in cascading this is pipe from HDFS -> filter for interesting claim lines -> buffer for denials -> pipe to HDFS output location. Simple, scalable, logical, improvable, testable. I've seen all of these. As the community comes out with new tools, we get more options. My team is working on machine learning and graphs. Mahout and Giraph. Hard to do all of this easily with a home grown data processing system. As always, research your needs. Don't get caught up in the hype of a new trend. Don't be closed minded either.

necessity 10y ago

>cat | grep why