11 ms·
Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
- killingtime74 10mo agoHadoop, blast from the past
- deleted 10mo ago[deleted]
- ejoebstl 10mo agoGreat article. Hadoop (and other similar tools) are for datasets so huge they don't fit on one machine.
- vjerancrnjak 10mo agohttps://www.scylladb.com/2019/12/12/how-scylla-scaled-to-one-billion-rows-a-second/ https://www.scylladb.com/2019/12/12/how-scylla-scaled-to-one... I like this one where they put a dataset on 80 machines only then for someone to put the same dataset on 1 Intel NUC and outperform in query time. https://altinity.com/blog/2020-1-1-clickhouse-cost-efficiency-in-action-analyzing-500-billion-rows-on-an-intel-nuc https://altinity.com/blog/2020-1-1-clickhouse-cost-efficienc... Datasets never become big enough…
- literalAardvark 10mo agoWell yeah, but that's a _very_ different engineering decision with different constraints, it's not fully apples to apples. Having materialised views increases insert load for every view, so if you want to slice your data in a way that wasn't predicted, or that would have increased ingress load beyond what you've got to spare, say, find all devices with a specific model and year+month because there's a dodgy lot, you'll really wish you were on a DB that can actually run that query instead of only being able to return your _precalculated_ results.
- DetroitThrow 10mo ago>Datasets never become big enough… Not only is this a contrived non-comparison, but the statement itself is readily disproven by the limitations basically _everyone_ using single instance ClickHouse often run into if they actually have a large dataset. Spark and Hadoop have their place, maybe not in rinky dink startup land, but definitely in the world of petabyte and exabyte data processing.
- zX41ZdbW 10mo agoWhen a single server is not enough, you deploy ClickHouse on a cluster, up to thousands of machines, e.g., https://clickhouse.com/blog/how-clickhouse-powers-ahrefs-the-worlds-most-active-web-crawler https://clickhouse.com/blog/how-clickhouse-powers-ahrefs-the...
- saberience 10mo agoWell, at my old company we had some datasets in the 6-8 PB range, so tell me how we would run analytics on that dataset on an Intel NUC. Just because you don't have experience of these situations, it doesn't mean they don't exist. There's a reason Hadoop and Spark became synonymous with "big data."
- deleted 10mo ago[deleted]
- dapperdrake 10mo agoThese situations are rare not difficult. The solutions are well known even to many non-programmers who actually have that problem: There are also sensor arrays that write 100,000 data points per millisecond. But again, that is a hardware problem not a software problem.
- PunchyHamster 10mo agoAnd we can have pretty fucking big single machines right now
- dapperdrake 10mo agohttps://yourdatafitsinram.net/ https://yourdatafitsinram.net/
- rcarmo 10mo agoThis has been a recurring theme for ages, with a few companies taking it to extremes—there are people transpiring COBOL to bash too…
- paranoidrobot 10mo agoA selection of times it's been previously posted: (2018, 222 comments) https://news.ycombinator.com/item?id=17135841 https://news.ycombinator.com/item?id=17135841 (2022, 166 comments) https://news.ycombinator.com/item?id=30595026 https://news.ycombinator.com/item?id=30595026 (2024, 139 comments) https://news.ycombinator.com/item?id=39136472 https://news.ycombinator.com/item?id=39136472 - by the same submitter as this post.
- nasretdinov 10mo agoAnd now with things like DuckDB and clickhouse-local you won't have to worry about data processing performance ever again. Just kidding, but especially with ClickHouse it's so much better to handle the large data volume compared to the past, and even a single beefy server is often enough to satisfy all data analytics needs for a moderate-to-large company.
- torginus 10mo agoWhen I worked as a data engineer, I rewrote some Bash and Python scripts into C# that were previously processing gigabytes of JSON at 10s of MB/s - creating a huge bottleneck. By applying some trivial optimizations, like streaming the parsing, I essentially managed to get it to run at almost disk speed (1GB/s on an SSD back then). Just how much data do you need when these sort of clustered approaches really start to make sense?
- embedding-shape 10mo ago> I rewrote some Bash and Python scripts into C# that were previously processing gigabytes of JSON Hah, incredibly funny, I remember doing the complete opposite about 15 years ago, some beginner developer had setup a whole interconnected system with multiple processes and what not in order to process a bunch of JSON and it took forever. Got replaced with a bash script + Python! > Just how much data do you need when these sort of clustered approaches really start to make sense? I dunno exactly what thresholds others use, but I usually say if it'd take longer than a day to process (efficiently), then you probably want to figure out a better way than just running a program on a single machine to do it.
- g8oz 10mo agoA C# executable is more similar to a bash/Python script than it is to a interconnected system using multiple processes.
- embedding-shape 10mo agoYeah, I realize now that my comment actually misses the most important point, the "interconnected system with multiple processes" I was talking about was made in C#, that's why the whole "I did the reverse as you" was funny to me in the first place.
- commandersaki 10mo agoHow do you stream parse json? I thought you need to ingest it whole to ensure it is syntactically valid, and most parsers don't work with inchoate or invalid json? Or at least it doesn't seem trivial.
- MarginalGainz 10mo agoThe saddest part about this article being from 2014 is that the situation has arguably gotten worse. We now have even more layers of abstraction (Airflow, dbt, Snowflake) applied to datasets that often fit entirely in RAM. I've seen startups burning $5k/mo on distributed compute clusters to process <10GB of daily logs, purely because setting up a 'Modern Data Stack' is what gets you promoted, while writing a robust bash script is seen as 'unscalable' or 'hacky'. The incentives are misaligned with efficiency.
- petcat 10mo ago> a robust bash script These hardly exist in practice. But I get what you mean.
- sam_lowry_ 10mo agoYoy don't. It's bash only because the parent process is bash, but otherwise it's all grep, sort, tr, cut and othe textutils piped together.
- mjevans 10mo agoawk can do some heavy lifting too if the environment is too locked down to import a kitchen sink of python modules.
- RobinL 10mo agoWorse in some ways, better in others. DuckDB is often an excellent tool for this kind of task. Since it can run parallelized reads I imagine it's often faster than command line tool, and with easier to understand syntax
- mrgoldenbrown 10mo agoIMHO the main point of the article is that typical unix command pipeline pipeline IS parallelized already. The bottleneck in the example was maxing out disk IO, which I don't think duckdb can help with.
- rented_mule 10mo agoA little bit of history related to the article for any who might be interested... mrjob, the tool mentioned in the article, has a local mode that does not use Hadoop, but just runs on the local computer. That mode is primarily for developing jobs you'll later run on a Hadoop cluster over more data. But, for smaller datasets, that local mode can be significantly faster than running on a cluster with Hadoop. That's especially true for transient AWS EMR clusters — for smaller jobs, local mode often finishes before the cluster is up and ready to start working. Even so, I bet the author's approach is still significantly faster than mrjob's local mode for that dataset. What MapReduce brought was a constrained computation model that made it easy to scale way up. That has trade-offs that typically aren't worth it if you don't need that scale. Scaling up here refers to data that wouldn't easily fit on disks of the day — the ability to seamlessly stream input/output data from/to S3 was powerful. I used mrjob a lot in the early 2010s — jobs that I worked on cumulatively processed many petabytes of data. What it enabled you to do, and how easy it was to do it, was pretty amazing when it was first released in 2010. But it hasn't been very relevant for a while now.
- KolmogorovComp 10mo ago> The first thing to do is get a lot of game data. This proved more difficult than I thought it would be, but after some looking around online I found a git repository on GitHub from rozim that had plenty of games. I used this to compile a set of 3.46GB of data, which is about twice what Tom used in his test. The next step is to get all that data into our pipeline. It would be interesting to redo the benchmark but with a (much) larger database. Nowadays the biggest open-data for chess must comes from Lichess https://database.lichess.org https://database.lichess.org, with ~7B games and 2.34 TB compressed, ~14TB uncompressed. Would Hadoop win here?
- woooooo 10mo agoIf you get all the data on fast SSDs in a single chassis, you probably still beat EMR over S3. But then you have a whole dedicated server to manage your 14TB of chess games. The "EMR over S3" paradigm is based on the assumption that the data isn't read all that frequently, 1-10x a day typically, so you want your cheap S3 storage but once in a while you'll want to crank up the parallelism to run a big report over longer time periods.
- dapperdrake 10mo agoProbably not. The compressed data can fit onto a local SSD. Decompression can definitely be streamed.
- djsjajah 10mo agoNot only can it be streamed, but lz4 will probably make things quicker.
- _zoltan_ 10mo agowhat would you calculate in the data? I could be tempted to do some fun on an NVL72 ;-)
- hunterpayne 10mo ago> Would Hadoop win here? Hadoop never wins. Its the worst of all possible worlds.
- fmajid 10mo agoI've contributed to PrestoDB, but the availability of DuckDB and fast multi core machines with even faster SSDs makes the need for distribution all the more niche, or even cargo-culting Google or Meta.
- fifilura 10mo agoThe benefit of prestodb is that it can be used without even starting one of these expensive instances in AWS Athena.
- cryptoboy2283 10mo agoEarlier in 2010 - http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...
- EdwardCoffin 10mo agoThis makes me think of Bane's rule, described in this comment here [1]: Bane's rule, you don't understand a distributed computing problem until you can get it to fit on a single machine first. [1] https://news.ycombinator.com/item?id=8902739 https://news.ycombinator.com/item?id=8902739
- forinti 10mo agoI think many devs learn the trade with Windows and don't get exposure to these tools. Plus, they require a bit of reading because they operate on a higher level of abstraction than loops and ifs. You get implicit loops, your fields get cut up automatically, and you can apply regexes simultaneously on all fields. So it's not obvious to the untrained eye. But you get a lot of power and flexibility on the cli, which enable you to rapidly put together an ad hoc solution which can get the job done or at least serve as a baseline before you reach for the big guns.
- benrutter 10mo agoThis times a zillion! I think there's been a huge industry push to convince managers and more junior engineers that spark and distributed tools are the correct way to do data engineering. I think its a similar pattern to web dev influencers have convinced everyone to build huge hydrated-spa-framework-craziness where a static site would do. My advice to get out of this mess: - Managers, don't ask for specific solutions (spark, react). Ask for clever engineers to solve problems and optimise / track what you vare about (cost, performance etc). You hired them to know best, and they probably do. - Technical leads, if your manager is saying "what about hyperscale?" You don't have to say "our existing solution will scale forever". It's fine to say, "our pipelines handle dataset up to 20GB, we don't expect to see anything larger soon, and if we do we'll do x/y/z to meet that scale". Your manager probably just wants to know scaling isn't going to crash everything, not that you've optimised the hell out of everything for your excel spreadsheet processing pipeline.
- zug_zug 10mo agoAbsolutely, when I worked at (semi-well-known unicorn) a half-dozen years ago on the data-engineering team the manager told me "Hey we want to use spark next quarter, that's a huge initiative." And I immediately asked, "in what capacity?" And the answer was don't-know/doesn't-matter, it's just important that we can say we're using it. I really wish I understood where that was coming from (his manager resume-building? somebody getting a kickback?)
- spauldo 10mo agoThat's when you rewrite your codebase in the SPARK dialect of Ada and play innocent when your management questions you about it.
- coliveira 10mo agoThey'll never say it's resume building or kickbacks, they'll invent some technically sounding and/or business reason to achieve the same result.
- thwarted 10mo agoThe most interesting part is that you can say you're doing/using something entirely independent of if you actually are. Sure, that's a lie, but so is only using something so you can say you're using it (sure, they admitted to you that was the reason, but that won't be the reason they put on LinkedIn).
- phyzix5761 10mo agoIt’s easy to overlook how often straightforward approaches are the best fit when the data and problem are well understood. Large expensive tools can become problems in their own right creating complexity that then requires even more tooling to manage. (Maybe that's the intent?) The issue is that teams and companies often adopt optimization frameworks earlier than necessary. Starting with simpler tools can get you most of the way there and in many cases they turn out to be all that’s needed.
- jonathanhefner 10mo agoAnd since AI agents are extremely good at using them, command-line tools are also probably 235x more effective for your data science needs.
- fifilura 10mo agoNo joins in that article? The comments here smell of "real engineers use command line". But I am not sure they ever actually worked with analysing data more than using it as a log parser. Yes Hadoop is 2014. These days you obviously don't set up a Hadoop cluster. You use the cloud provider service provided (BigQuery or AWS Athena for example). Or map your data into DuckDB or use polars if it is small.
- ziml77 10mo ago> But I am not sure they ever actually worked with analysing data more than using it as a log parser. It really feels that way. Real data analysis involves a lot more than just grepping logs. And the reason to be wary of starting out unprepared for that kind of analysis is that migrating to a better solution later is a nightmare.
- noo_u 10mo agoIn many ways HN is Reddit in denial at this point :) Comments and upvotes that are based mostly on vibes, with depth and discussion usually happening somewhere towards the middle of the comment tree.
- dapperdrake 10mo agoWhere else would you JOIN in?
- noo_u 10mo agoI'm about to start looking - this JOINt's way past its prime
- christophilus 10mo agoIt depends. I’ve done plenty of data processing, including at large fortune 10s. Most of the big data could be shrunk to small data if you understood the use case— pre-aggregating, filtering to smaller datasets based on known analysis patterns, etc. Now, you could argue that that’s cheating a bit and introduces preprocessing that is as complex as running Hadoop in the first place, but I think it depends. In my experience, though, most companies really don’t have big data, and many that do don’t really need to. Most companies aren’t fortune 500s. I used to work at Elastic, and I noticed that most (not all!) of the customers who walked up to me at the conferences were there to ask about datasets that easily fit into memory on a cheap VPS.
- jeffbee 10mo agoSomething to note here is that the result of xargs -P is unlikely to be satisfactory, since all of the subprocesses are simply connected to the terminal and stomp over each other's outputs. A better choice would be something like rush or, for the Perl fans, parallel.
- hmokiguess 10mo agoTangential, but this reminds of the older K website when it used to be shakti.com that had an intro like this in their about section: 1K rows: use excel 1M rows: use pandas/polars 1B rows: use shakti 1T rows: only shakti Source: https://web.archive.org/web/20230331180931/https://shakti.com/ https://web.archive.org/web/20230331180931/https://shakti.co...
- meken 10mo agoI’m curious about the memory usage of the cat | grep part of the pipeline. I think the author is processing many small files? In which case it makes the analysis a bit less practical, since the main use case I have for fancy data processing tools is when I can’t load a whole big file into memory.
- dapperdrake 10mo agoMemory footprint is tiny: Unix shell pipelines are task-parallel. Every tool gets spun up as its own unix process — think "program" (fork-exec). Standard input and standard output (stdin, stdout) get hooked up to pipes. Pipes are like temporary files managed by the kernel (hand-wave). Pipe buffer size is a few KB. Grep does a blocking read on stdin. Cat writes to stdout. Both on a kernel I/O boundary. Here the kernel can context-switch the process when waiting for I/O. In the past there was time-slicing. Now with multiple cores and hardware threads they actually run concurrently. This is very similar to old-school approach to something like multiple threads, but processes don’t share virtual address spaces in the CPU's memory management unit (MMU). Further details: look up McIlroy's pipeline design.
- srcreigh 10mo agoMapReduce is from a world with slow HDDs, expensive ram, expensive enterprise class servers, fast network. In that case to get best performance, you’d have to shard your data across a cluster and use mapreduce. Even in the authors 2014 SSDs multi-core consumer PC world, their aggregate pipeline would be around 2x faster if the work was split across two equivalent machines. The limit of how much faster distributed computing is comes down to latency more than throughput. I’d not be surprised if this aggregate query could run in 10ms on pre sharded data in a distributed cluster.
- dapperdrake 10mo agoConfusing the concept and the implementation.
- srcreigh 10mo agoSomebody has to go back to first principles. I wrote pig scripts in 2014 in Palo Alto. Yes, it was shit. IYKYK. But the author, and near everybody in this thread, are wrong to generalize. PCIe would have to be millions of times faster than Ethernet before command line tools are actually faster than distributed computing and I don't see that happening any time soon.
- jeswin 10mo agoThe same thing is true with Sqlite vs Postgres. Most startups need Sqlite, not Postgres. Many queries run an order of magnitude faster. Not only is it better for your users, it's life changing to see the test suites (which would take minutes to run) complete in mere seconds
- shimman 10mo agoFeels like quibbling over the differences between two databases that are going to act the same for 90% of projects out there doesn't really matter. If you want speed, just have your database stored in the same place as your application, locally, rather than hopping across the world to retrieve data that can be located next to the code. That would probably be the easiest thing to do to get a real measured performance gains. As other commentators pointed out, computers are extremely powerful. This isn't 1995, you can easily host everything in the same local area and get a very responsive application with very minimal needs to worry about resource constraints.
- Sesse__ 10mo ago> Many queries run an order of magnitude faster. Given how primitive SQLite's optimizer is and how similar the storage and execution engines between the two are in terms of architecture, this seems unlikely to be the norm unless you did something wrong on the Postgres side. (Of course, no RDBMS optimizer will always give the best answer, so there's bound to be such cases.)
- tucnak 10mo agoI don't know where this SQLite obsession is coming from these days, but it doesn't help that Cloudflare/Fly/other EngBrands are doing it.
- mbb70 10mo agoThe bigness of your data has always depended on the what you are doing with it. Consider the following table of medical surgeries: date,physician_name, surgery_name,success. "What are the top 10 most common surgeries?" - easy in bash "Who are the top physicians (% success) in the last year for those surgeries?" - still easy in bash "Which surgeries are most affected by physician experience?" - very hard in bash, requires calculating for every surgery how many times that physician had performed that surgery on that day, then compare low and high experience outcomes. A researcher might see a smooth continuum of increasingly complex questions, but there are huge jumps in computational complexity. At 50gb dataset might be 'bigger' than a 2tb one if you are asking tough questions. It's easier for a business to say "we use Spark for data processing", than "we build bespoke processing engines on a case by case basis".
- christophilus 10mo ago50GB and 2TB are both sizes that SQLite supports and could handle. You could probably solve all of the problems you mentioned with simple tools on a single server, in the language of your choice.
- esafak 10mo agoSounds like a good fit for DuckDB.
- adamdrake 10mo agoAuthor here! It's great to see this post I wrote years ago still being useful for people. I agree with many here that the situation is arguably worse in many ways. However, along similar lines, I've been pleased to see a move away from cargo culting microservices (another topic I addressed in a separate post on that site). To all those helping companies and teams improve performance, keep it up! There is hope!
- dapperdrake 10mo agoAdam, Thank you very much! Been re-reading your post multiple times. You inspired me to port Waters-Series (kind-of streams) to JavaScript to get pipelining for stream processing.
- olq_plo 10mo agoAnd now you can do this with polars in parallel on all your cores and the GPU, using almost the same syntax as in pyspark.
- jgord 10mo agohighly recommend xsv by BurntSushi [ csv parser / wrangler written in rust ]
- esafak 10mo agoIt's retired in favor of qsv and xan: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv
- danpalmer 10mo agoIt was a fun moment to finally work on a data problem that did not fit on any (practical) machine. I needed about 50TiB of memory to process a multi-PiB set of logs. It's worth remembering however that even though it's less efficient per-CPU or whatever to split a large task into many smaller tasks, it may be more efficient overall alongside other workloads as you can bin-pack tasks more efficiently on a cluster, not to mention if tasks fail you are retrying less of the overall work. All this is to say, the article makes a very good point, but doing it all on one machine also has problems. Just don't cargo cult engineering decisions.
- dostick 10mo agoRemember back in the day it being called “Hadpoop”
- vismit2000 10mo agoLecture 2: Command-line Environment of Missing Semester 2026 released today! - https://www.youtube.com/watch?v=ccBGsPedE9Q https://www.youtube.com/watch?v=ccBGsPedE9Q
- irskep 10mo agoWorking on mrjob was a big part of my first job out of college. Fun to see it get mentioned more than ten years later. What some commenters don't realize about these bureaucratic IO-heavy expensive tools is that sometimes they are used in order to apply a familiar way of thinking, which has Business Benefits. Sometimes you don't know if your task will take seconds, minutes, hours, days, or weeks on one fast machine with a well-thought-out program, but you really need it to take at most hours, and writing well-thought-out-programs takes time you could spend on other stuff. If you know you can scale the program in advance, it's lower risk to just write it as a Hadoop job and be done with it. Also, it helps to have an "easy" pattern for processing Data That Feels Big Even If It Isn't That Big, Although Yelp's Data Actually Was Big. Such was the case with mrjob stuff at Yelp in 2012. They got a lot of mileage out of it! The other funny thing about mrjob is that it's a layer on Hadoop Streaming, which is a term for when the Java process actually running the Hadoop worker opens a subprocess to your Python script which accepts input on stdin and writes output on stdout, rather than working on values in memory. A high I/O price to pay for the convenience of writing Python!
- willtemperley 10mo agoThat's a good point. Hadoop may not be the most efficient way, but when a deliverable is required, Hadoop is a known quantity and really works. I did some interesting work ten years ago, building pipelines to create global raster images of the entire Open Street Map road network [1]. I was able to process the planet in 25 minutes on a $50k cluster. I think I had the opposite problem: Hadoop wasn't shiny enough and Java had a terrible reputation in academic tech circles. I wish I'd known about mrjob because that would have kept the Python maximalists happy. I had lengthy arguments with people who wanted to use Spark which simply did not have the chops for this. With Spark, attempting to process OSM for a small country failed. Another interesting side-effect of using the map-reduce paradigm was with processing vector datasets. PostGIS took multiple days to process the million-vertex Norwegian national parks, however splitting the planet into data density sensitive tiles (~ 2000 vertices) I could process the planet in less than an hour. Then Google Earth Engine came along and I had to either use that, or change career. Somewhat ironically GEE was built in Java. [1] https://github.com/willtemperley/osm-hadoop https://github.com/willtemperley/osm-hadoop
- Juliate 10mo agoYes, but you both don’t raise investor money with efficient understanding of the chaining of CLI tools that already do the job, neither convince your clients that they can get value for their money.
- eisbaw 10mo agomawk is fast
- deafpolygon 10mo agoI had a similar very tiny-scale problem that was solved much quicker by awk+perl. I had a relatively large dataset across many YAML files that I needed to compute a result. Turns out that using yq / jq to performn a query was much (order of magnitudes) slower (>10m) to compute any kind of result. Outputting the data into CSV, then iterating over that was much, much faster (seconds). Of course, dumping it into SQlite and querying that was nearly instantaneous. I know it’s not a direct 1:1 comparison, but it brings to mind that solutions that were made common decades ago are still relevant today.