8 ms·
Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)
- hobos_delight 3y ago…yes - processing 3.2G of data will be quicker on a single machine. This is not the scale of Hadoop or any other distributed compute platform. The reason we use these is for when we have a data set _larger_ than what can be done on a single machine.
- ralph84 3y agoMost people who wasted $millions setting up Hadoop didn’t have data sets larger than could fit on a single machine.
- hiAndrewQuinn 3y agoMoore's law and its analogues makes this harder to back-predict than one might think, though. A decade ago computers had only had about an eighth (rough upper bound) of the resources modern machines tend to have at similar price points.
- hobos_delight 3y agoI completely agree. I love the tech and have spent a lot of time in it - but come on people, let’s use the right tool for the right job!
- saberience 3y agoDo you have any examples of companies building Hadoop clusters for amounts of data that fit on a single machine? I’ve heard this anecdote on HN before but without ever seeing actual evidence it happened, it reads like an old wives tale and I’m not sure I believe it. I’ve worked on a Hadoop cluster and setting it up and running it takes quite serious technical skills and experience and those same technical skills and experience would mean the team wouldn’t be doing it unless they needed it. Can you really imagine some senior data and infrastructure engineers setting up 100 nodes knowing it was for 60GB of data? Does that make any sense at all?
- oblio 3y agoI used to work for a pretty famous 2nd tier US company (smaller and less cool than FAANG). They had a team working on a Hadoop based solution and their biggest internal implementations was about what you're describing, in practice. It makes sense because internal politics.
- jerven 3y agoIn 2014 I was at Oracle Open World. A 3rd party hardware vendor was saying (and having customers) for Hadoop "clusters" that had 8 cpu cores. Basically their pitch was that Oracle Hardware (ex sun) started at a dense full rack of about a 1 million USD or so, but with the 3d party you could have a hadoop "cluster" in 2U and for 20K. The oracle thing was actually quite price competitive at the time, if you needed hadoop. The 3rd party thing was overpriced for what it was. Yet, I am sure that 3rd party hardware vendor made out like bandits.
- dijit 3y agoI did some data processing at Ubisoft. each node in our hadoop cluster had 64GiB of ram (which is the max amount you should have for a single node java application, where 32G is allocated for heap FWIW), we had I think 6 of these nodes for a total of 384GiB memory. Our storage was something like 18TiB across all nodes. It would be a big machine, but our entire cluster could easily fit. Largest machine on the market right now is something like 128CPU's and 20TiB of Memory. 384GiB was available in a single 1U rackmount server at least as early as 2014. Storage is basically unlimited with direct-attached-storage controllers and rackmount units.
- genewitch 3y agoI had an HP from 2010 that supported 1.5TB of ram with 40 cores, but it was 4U. I'm not sure what the height has to do with memory other than a 1U doesn't have the luxury of the backplane(s) being vertical or otherwise above the motherboard, so maybe it's limited space?
- 3y ago
- faet 3y agoI've worked places where it would be 1000x harder getting a spare laptop from the IT closet to run some processing than it would be to spend $50k-100k at Azure.
- MrBuddyCasino 3y agoThis will not stop BigCorp to spend weeks to setup a big ass data analytics pipeline to process a few hundred MB from their „Data Lake“ via Spark. And this isn’t even wrong, bc what they need is a long-term maintainable method that scales up IF needed (rarely), is documented and survives loss of institutional knowledge three layoffs down the line.
- hobos_delight 3y agoScaling _if_ needed has been the death knell of many companies. Every engineer wants to assume that they will need to scale to millions of QPS, most of the time this is incorrect, and when it is not then the requirement have changed and it needs to be rebuilt anyway.
- MrBuddyCasino 3y agoThis is true for startups an small companies, Big Corps IT is so far away from operating efficiently that this doesn't really matter.
- hobos_delight 3y agoI think it completely matters - yes these orgs are a lot more wasteful, but there is still an opportunity to save money here, especially is this economy, if not for the internal politics wins. I’ve spent time in some of the largest distributed computing deployments and cost was always a constant factor we had to account for. The easiest promos were always “I saved X hundred million” because it was hard to argue against saving money. And these happened way more than you would guess.
- MrBuddyCasino 3y ago> I’ve spent time in some of the largest distributed computing deployments Yeah obviously if you run hundreds or thousands of severs then efficiency matters a lot, but then there isn't really the option to use a single machine with a lot of RAM instead, is there? I'm talking about the typical BigCorp whose core business is something else than IT, like insurance, construction, mining, retail, whatever. Saving a single AKS cluster just doesn't move the needle.
- OskarS 3y agoThis is exactly the point of the article. From the conclusion: > Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.
- the8472 3y agoWhat can be done on a single machine grows with time though. You can have terabytes of ram and petabytes of flash in a single machine now.
- visarga 3y agocurrent day version - if you got less than 10K - 100K vectors use numpy.dot instead of the vector store databases
- eigenvalue 3y agoEven if you have a lot more than that you could easily use SQLite with FAISS. It works great.
- bravura 3y agoBingo. This combination is underappreciated.
- eigenvalue 3y agoYes, I’ve used this approach in my Swiss Army Llama project with huge numbers of vectors and it can scale massively. Also it’s free! These vector db as a service companies charge insane prices for this! It really does feel like snake oil to me. https://github.com/Dicklesworthstone/swiss_army_llama https://github.com/Dicklesworthstone/swiss_army_llama
- jbverschoor 3y agoYou can easily add two zeros to that.
- tokai 3y agoI for one does not miss having "big data" being mentioned in every meeting, talk, memo, etc. Sure it's AI now, but even that doesn't become as annoying and misunderstood as the big data fad was.
- BiteCode_dev 3y agoIt's all cycle. Remember when XML was the future ? Money quote from https://www.bitecode.dev/p/hype-cycles https://www.bitecode.dev/p/hype-cycles: > geeks think they are rational beings, while they are completely influenced by buzz, marketing, and their emotions. Even more so than the average person, because they believe they are less susceptible to it than normies, so they have a blind spot.
- upupupandaway 3y agoThis made me remember the amazing “parable of the languages” that had XML as the main antagonist of the story. We need an AI update for this one. https://burningbird.net/the-parable-of-the-languages/ https://burningbird.net/the-parable-of-the-languages/
- pjmlp 3y agoHere I am seated in meetings discussing MACH architectures and serverless, while thinking about "The Network is the Computer" in Sun's manuals.
- marcosdumay 3y agoFirst time I hear about MACH. Is the name because of the Mach OS? (The top reference I get for it is a spam-site that is trying to hype the name and sell a domain.)
- pjmlp 3y agoMicroservices, API-first, Cloud-native, and Headless. https://machalliance.org/ https://machalliance.org/ Yet another trend in our fashion driven industry.
- zzbn00 3y agoThis was nicely foreseen in the original Map - Reduce paper, where the authors write: > The issues of how to parallelize the computation, distribute the data, and > handle failures conspire to obscure the original simple computation with large > amounts of complex code to deal with these issues. As a reaction to this > complexity,we designed anew abstraction that allows us to express the simple > computations we were trying to perform but hides the messy details of > parallelization, fault-tolerance, data distribution and load balancing in > a library . If you are not meeting this complexity (and today with 16 TB of RAM and 192 cores, many jobs don't) then Map-Reduce / Hadoop is not for you...
- dijit 3y agoThere is an incentive for people to go horizontal rather than permitting themselves to go vertical. Makes sense, we are told that vertical has limits in university and we should prioritise horizontal; but I feel a little like the "mid-wit" meme, once we realise how vertical we can go then we can end up using significantly fewer resources in aggregate (as there is overhead in distributed systems of course). I also think we are disincentivised from going vertical as most cloud providers prioritise splitting workloads, most people don't have 16TiB of RAM available to them, but they might have a credit card on file for a cloud provider/hyperscaler. *EDIT*: Largest AWS Instance is, I think, the x2iedn.metal ith 128vCPU and 4TiB RAM *EDIT2*: u-24tb1.metal seems larger; 448vCPU and 24TiB Memory, but I'm not sure if you can actually use it for anything that's not SAP HANA.
- saberience 3y agoSo I can use a command line tool to process queries that are processing 100 TB of data? The last time I used Hadoop it was on a cluster with roughly 8PB of data. Let me know when I can do it locally.
- pinkgolem 3y agofor the given question, sure.. you can there are 60tb ssd's out there.. you might even fit all of the 8tb on a given server
- anthk 3y agoUnix has a split(1) tool.
- vidarh 3y ago"Can be 235x faster" != "will always be 235x faster", nor indeed "will always be faster" or "will always be possible". The point is not that there are no valid uses for Hadoop, but that most people who think they have big data do not have big data. Whereas your use case sounds like it (for the time being) genuinely is big data, or at least at a size where it is a reasonable tradeoff and judgement call. To people's beliefs on this, here's a Forbes article on Big Data [1] (yes, I know Forbes is now a glorified blog for people to pay for exposure). It uses as example a company with 2.2 million pages of text and diagrams. Unless those are far above average, they fit in RAM on a single server, or on a small RAID array of NVMe drives. That's not Big Data. I've indexed more than that as a side-project on a desktop-class machine with spinning rust. The people who think that is big data are the audience of this, not people with actual big data. [1] https://www.forbes.com/sites/forbestechcouncil/2023/05/24/the-new-era-of-big-data/ https://www.forbes.com/sites/forbestechcouncil/2023/05/24/th...
- rakoo 3y agoHave you read the article ?
- chx 3y agoMore succinct version of the same from Gary Bernhardt of WAT fame (from 2015, same era) https://twitter.com/garybernhardt/status/600783770925420546 https://twitter.com/garybernhardt/status/600783770925420546 > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000.
- CodesInChaos 3y agoReminds me of one of my favourite twitter posts: > Small Data is when is fit in RAM. Big Data is when is crash because is not fit in RAM. https://twitter.com/DEVOPS_BORAT/status/299176203691098112 https://twitter.com/DEVOPS_BORAT/status/299176203691098112
- tetha 3y agoDEVOPS_BORAT contains a lot of truth if you think about it, hah. Our sarcastic team-motto is very much this: https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312
- h4kor 3y agohttps://yourdatafitsinram.net/ https://yourdatafitsinram.net/
- thelastgallon 3y agoI see servers with more RAM, 32 TB[1], 48TB RAM[2], probably more at: https://buy.hpe.com/us/en/compute/mission-critical-x86-servers/c/1010550750 https://buy.hpe.com/us/en/compute/mission-critical-x86-serve... [1]https://buy.hpe.com/us/en/compute/mission-critical-x86-servers/compute-scale-up-servers/compute-scale-up-servers/hpe-compute-scale-up-server-3200/p/1014774076 https://buy.hpe.com/us/en/compute/mission-critical-x86-serve... [2]https://buy.hpe.com/us/en/compute/mission-critical-x86-servers/integrity-mc990-x-servers/integrity-mc900-x-servers/hpe-integrity-mc990-x-server/p/1008798952 https://buy.hpe.com/us/en/compute/mission-critical-x86-serve...
- ryukoposting 3y agoI remember reading this article several years ago. Good to see it again. I remember when everyone thought their data was big data. How provincial we were.
- _xivi 3y agoRelated: https://news.ycombinator.com/item?id=30595026 https://news.ycombinator.com/item?id=30595026 - 1 year ago (166 comments) https://news.ycombinator.com/item?id=22188877 https://news.ycombinator.com/item?id=22188877 - 3 years ago (253 comments) https://news.ycombinator.com/item?id=17135841 https://news.ycombinator.com/item?id=17135841 - 5 years ago (222 comments) https://news.ycombinator.com/item?id=12472905 https://news.ycombinator.com/item?id=12472905 - 7 years ago (171 comments)
- pvg 3y agoI think you have an off by one year error in these.
- deleted 3y ago[deleted]
- hliyan 3y agoI've written this comment before: in 2007, there was a period where I used to run an entire day's worth of trade reconciliations of one of the US's primary stock exchanges on my laptop (I was on-site engineer). It was a Perl script, and it completed in minutes. A decade later, I watched incredulously as a team tried to spin up a Hadoop cluster (or Spark -- I forget which) over several days, to run a work load an order of magnitude smaller.
- liveoneggs 3y agoyeah but who was getting better stuff on their resume? didn't you get the memo about perl? Just because your throw-away 40 line script worked from cron for five years without issue doesn't mean that a seven node hadoop cluster didn't come with benefits. You got to write in a language called "pig"! so fun.
- jan_Sate 3y agoI still think that it'd be easier to maintain the script that runs on a single computer than to maintain a hadoop cluster.
- rbanffy 3y agoThe resume would look better if you used Python and Polars ;-)
- wilkystyle 3y agoUsing Rust and Polars over here; am I doing this right?
- marcosdumay 3y agoYou are probably too early, but looks like a good investment.
- 2-718-281-828 3y ago
- nickdothutton 3y agoLike a lot of things, people tend to make a decision for horizontal vs vertical and then stick with it even as the platforms or "physics" change underneath them over time. Same for memory bandwidth (which people, like Sun, thought would remain more of a bottleneck than it actually turned out to be).
- anthk 3y agoThere was some environment (Ice surface movements in the South Pole I think) related researcher who rewrote his calculations from Nvidia and GPU computing to a plain C file. The NV task lasted for months; later, seconds.
- datadeft 3y agoThe raise of single node computing is powered by two things: - desktop computers are really powerful (Apple Mx, AMD Epyc etc.) - software like Polars
- 0xbadcafebee 3y agoIt's so funny how there is almost no 'science' - or 'engineering' - in modern 'computer science' or 'software engineering'. The finding on OP's website, of what is or isn't fast for what purpose, should not be surprising us, 79 years after the first programmable computer. Yet we go about our work, blissfully ignorant of what the actual capabilities of what we're doing are. We don't have hypotheses, experiments, and results published, of what a given computing system X, made up of Y, does or doesn't achieve. There are certainly research papers, algorithms and proof-of-concepts, but (afaict) no scientific evidence for most of the practices we follow and results we get. We don't have engineering specifications or tolerances for what a given thing can do. We don't have calculations for how to estimate, given X computing power, and Y system or algorithm, how much Z work it can do. We don't even have institutional knowledge of all the problems a given engineering effort faces, and how to avoid those problems. When we do have institutional knowledge, it's in books from 4 decades ago, that nobody reads, and everyone makes the same mistakes again and again, because there is no institutional way to hold people to account to avoid these problems. What we do have, is some tool someone made, that then millions of dollars is poured into using, without any realistic idea whatsoever what the result is going to be. We hope that we get what we want out of it once we're done building something with it. Like building a bridge over a river and hoping it can handle the traffic.
- deleted 3y ago[deleted]
- swader999 3y agoIsn't a program really a scientific experiment? About as pure as you can get. It's not formalized as such but that's trivial.
- ycombobreaker 3y agoIt can be, but usually isn't. Similarly, dropping a feather and a bowling ball simultaneously might be science, or might not be. Did I make observations? Or am I just delivering some things to my friend at the bottom?
- 3y ago
- hiAndrewQuinn 3y agoI bought a Raspberry Pi 4 for Christmas. It's connected to my dev laptop directly via wired connection. My self imposed challenge for this year is to try to offload as much work to this little Pi as I can. So I'm a fan of this approach.
- rbanffy 3y agoOn a similar reasoning, in 2008 or such, I observed that, while our Java app would be able to run more user requests per second than our Python version, it’d take months for the Java app to overtake the Python one in total requests served because it’d have to account for a 6 month head start. Far too often we waste time optimising for problems we don’t have, and, most likely, will never have.
- donatj 3y agoMy work sent me to a Hadoop workshop in 2016 where in the introduction the instructor said Hadoop would replace the traditional RDBMS within five years. We went on to build a system to search the full text of Shakespeare for word instances that took a solid minute to scan maybe 100k of text. An RDBMS with decent indexes could have done that work instantly; hell, awk | grep | sort | uniq -c could have done that work instantly. It’s been 8 years and I think RDBMS is stronger than ever? Colored the entire course with a “yeah right”. Frankly is Hadoop still popular? Sure, it’s still around but I don’t hear much about it anymore. Never ended up using it professionally, I do most of my heavy data processing in Go and it works great. https://twitter.com/donatj/status/740210538320273408 https://twitter.com/donatj/status/740210538320273408
- michaelmior 3y agoHadoop has largely been replaced by Spark which eliminates a lot of the inefficiencies from Hadoop. HDFS is still reasonably popular, but in your use case, running locally would still be much better.
- wenc 3y agoSpark is still pretty non performant. If the workload fits in memory and a single machine, DuckDb is so much more lightweight and faster.
- cmiles74 3y agoMy current task at my day job is analyzing a large amount of data stored in a Spark cluster. I'd say, so far, 80% of the job has been extracting data from the cluster so that I can work with it interactively with DuckDB. This data is all read-only, I suspect a set of PostgreSQL servers would perform much better.
- wenc 3y agoYes. My job involves pulling a ton of data off Redshift into Parquet files, and then working with them using DuckDB (sooo much faster — DuckDB is parallelized, vectorized and just plain fast on Parquet datasets)
- habibur 3y agoThat also means using Hadoop makes sense only when cluster size > 235 machines.
- Wonnk13 3y agoOne of my favorite posts. I'll always upvote this. Of course there are use cases one or two standard deviations outside the mean that require truly massive distributed architectures, but not your shitty csv / json files. Reflecting on a decade in the industry I can say cut, sort, uniq, xargs, sed, etc etc have taken me farther than any programming language or ec2 instance.
- gpapilion 3y agoIt’s worth noting nvme and ssds make this possible. If this were off an hdd this approach would likely be slower.
- genewitch 3y ago272MB/s is "consumer spinning rust RAID 5 via USB3" speeds. I know this because that's roughly what my NAS backup machine does on writes/reads. From a NAS on gbit it's overpowered by 150%, for sure.
- belval 3y agoEven in with large scale data Hadoop/Spark tend to be used in ways that makes no sense, as if something being self described as big data means that as soon as you cross some threshold you SHOULD be using it. Recently had an argument with a senior engineer on our team because a pipeline that processed several PB of data, scaled to +1000 machines and was all account a success was just a Python script using multiprocessing distributed with ECS and didn't use Spark.
- mukti 3y agoCommon command line tools are often the best for analyzing and understanding HPC clusters and issues. People have often asked me for tools and web pages to figure out how to understand and figure out issues in our cluster, or asked if we could use some tool like Hadoop, Spark, or some Azure/GCP/AWS tool to do it faster. I've said that if they want to spend the effort to use those tools, it could be valuable; but if it takes me 10min to use those tools and <1min using command line tools, I'll always fall back to the command line. That's not to say that fancy tools don't have their use; but people often forget how much you can do with a few simple commands if you understand a pipeline and how the commands work.
- HackerThemAll 3y agoIf we write dedicated tools, speeds boost can be enormous. We can process 1 billion rows from a simple CSV in just 2 seconds. In slow Java. It just requires some skills, which is hard to find nowadays. https://github.com/gunnarmorling/1brc https://github.com/gunnarmorling/1brc
- esafak 3y agoWhat is the largest data set people here are processing daily for ETL on one machine? What tools are you using, and what does the job do? I want to know how capable new libraries like polars are, and how far you can delay transitioning to Spark. Are terabyte datasets feasible yet?
- bblaylock 3y agoThese posts always remind me of the [Manta Object Storage](https://www.tritondatacenter.com/triton/object-storage https://www.tritondatacenter.com/triton/object-storage) project by Joyent. This project was basically a combination of object storage with the added ability to run arbitrary programs against your data in situ. The primary, and key, difference being that you kept the data in place and distributed the program to the data storage nodes (the opposite of most data processing as I understand it), I think of this as a superpowered version of using [pssh](https://linux.die.net/man/1/pssh https://linux.die.net/man/1/pssh) to grep logs across a datacenter. Yet another idea before its time. Luckily, Joyent [open sourced](https://github.com/TritonDataCenter/manta https://github.com/TritonDataCenter/manta) the work, but the fact that it still hasn't caught on as "The Way" is telling. Some of the projects I remember from the Joyent team were: dumping recordings of local mariokart games to manta and running analytics on the raw video to generate office kart racer stats, the bog standard dump all the logs and map/reduce/grep/count them, and I think there was one about running mdb postmortems on terabytes of core dumps.
- sbalamurugan 3y agoThis is one of my favourite posts. Part of my PhD was based on this post. https://discovery.ucl.ac.uk/id/eprint/10085826/ https://discovery.ucl.ac.uk/id/eprint/10085826/ (Section 4.1). I presented this section of my research in a conference and won best paper award as well. Another post I love is https://stackoverflow.com/questions/2908822/speed-up-the-loop-operation-in-r https://stackoverflow.com/questions/2908822/speed-up-the-loo... where the guy manages to speed up a function which will run for days to milliseconds.