16 ms·

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

362 points by graderjs 5y ago

gnuvince 5y ago
This article is such a good demonstration (IMO) that computers are really fast, but that we have gotten really bad at actually making reasonable use of them.
wenc 5y ago
This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=command%20line%20tools%20hadoop&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.
- commandlinefan 5y ago
  Well, a lot of AWS and GCP infrastructure use it, but they hide it from you.
  - fulafel 5y ago
    Any specifics about which services?
    - ryapric 5y ago
      AWS: Glue, and by proxy Athena GCP: Dataproc Those are just the obvious ones though.
      - mr_toad 5y ago
        Glue often uses EMR under the hood, which is often Spark. And Athena is PrestoDB, as far as I know it has nothing to do with Hadoop other than you can use it to query Hadoop data stores.
        trashtester 5y ago
        The way I see it, Hadoop is still in common use as the storage layer for Spark and related implementations, whether that is in the form of HDFS or something like EMRFS: Quote from AWS: "EMRFS is an implementation of the Hadoop file system ..." https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-file-systems.html https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...
        commandlinefan 5y ago
        Ah - that's a good point. Usually when people say "Hadoop", I assume they're referring to HDFS, but there is (was...) the Hadoop MR system that ran on top of it that has been almost entirely replaced by Spark.
- jamal-kumar 5y ago
  Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
  - bpodgursky 5y ago
    I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
    - mohanmcgeek 5y ago
      But I can write SQL in Spark just fine. Can't I?
      - bpodgursky 5y ago
        You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is doing on the backend to support their SQL performance, but it's really freaking fast. There's just no other way to cut it. I've spent 5-10 years eking every ounce of performance I can get out of a Hadoop/Spark cluster. I'm not trying to be unreasonable about this. I would love for OSS to be competitive; it's great for the world, and it would be great for my skill set and earning potential. But it's not a contest, and if you think standalone Spark is going to be a viable competitor in a couple years, you are deluding yourself. Make informed choices about your career and investment.
        rxin 5y ago
- Puts 5y ago
  Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.
  - spullara 5y ago
    I mean, it isn't like Hadoop wasn't used to parse text files. Also, it is all fun and games until you need a join.
    - kragen 5y ago
      comm(1) and join(1) can do joins on text files. Make sure they're sorted in the same locale you're joining them in; LANG=C tends to be the fastest.
  - IanCal 5y ago
    The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.
    - stingraycharles 5y ago
      I’d argue that for as long as Excel doesn’t support Parquet files, we haven’t seen the last of CSVs for a long time. Parquet is great, but it’s simply nowhere near as ubiquitous as CSV.
      - gehen88 5y ago
        I hadn't even heard of Parquet until now, and I'm sure this goes for lots of developers who don't do much data engineering.
        jsjohnst 5y ago
- StreamBright 5y ago
  > Outside of legacy systems, Hadoop isn't widely used anymore. This is not true at all. Almost all the cloud providers have their own Hadoop distributions that is used a lot in many companies.
  - rizkeyz 5y ago
    Exactly. Basically most big data lives in hdfs and hdfs is part of hadoop. Even if you use Spark and Flink I would classify that as using Hadoop (under the hood). > However, a very common setup is to use Flink to analyze data stored in the Hadoop Distributed File System (HDFS). -- https://wints.github.io/flink-web//faq.html https://wints.github.io/flink-web//faq.html
- tus666 5y ago
  What about if you had 500 petabytes of data to analyze? Big data does exist in the wild.
  - usrbinbash 5y ago
    > What about if you had 500 petabytes of data to analyze? Then we probably need the big tools. > Big data does exist in the wild. So does little data. The problem is that a "one-size-fits-all" approach has become common, not just in data analysis; think of all the low-medium traffic webpages that use giant, complex frameworks and huge distributed systems just to display essentially a small CRUD app that would have been ALOT easier to cobble together in plain JS on a simple LAMP server. What the article shows is the importance on deciding for the right tool for the job: When I want to plant a little tree in my backyard, bringing one of these https://upload.wikimedia.org/wikipedia/commons/0/01/Bucket_wheel_excavator_in_Ferropolis.jpg https://upload.wikimedia.org/wikipedia/commons/0/01/Bucket_w... to dig the hole is proooobably overengineering it a tiny little bit, and will likely take longer than getting a shovel.
  - HelloNurse 5y ago
    Those 500 PB imply a competent staff, a world-class infrastructure, and worthwhile applications. These things don't materialize suddenly: they start maturing when the data set is 500 GB or even just planned and they usually mature enough to switch technology multiple times as the scale rises.
- auggierose 5y ago
  So what is used instead of Hadoop currently?
  - serial_dev 5y ago
    Whether a technology can replace Hadoop in an organization depends on many factors, but some technologies that solve at least in part similar problem are Apache Storm, Spark, Flink, Kafka Streams, and maybe BigQuery? Or, as the original article says, some companies just use some command line tools, shell scripts. It's been a couple of years since I was interested in Data Engineering, so my knowledge on this topic is some years behind.
    - riknox 5y ago
      I've not seen Storm being used anywhere sane for a few years at least now, and from a glance at job postings it looks unlikely. Spark, Kafka Streams etc. are definitely used in a modern data platform from my experience. I think we're seeing a big shift with Hadoop-like workloads being moved onto cloud providers, so BigQuery, Amazon EMR etc.
      - bsenftner 5y ago
        I'm curious what constitutes "big data" anymore. In an intermediate machine learning course, we train on nearly a petabyte of data using Google Colab and Jupyter Notebooks. Nobody discusses the size of the data requiring any special treatment due to its size... would not 95% of a petabyte be "big data"?
        happymellon 5y ago
        Big data is a shifting concept as computers gain more storage and faster commodity processors. My general rule of thumb is whether it is too big to put on my laptop. So greater than a couple of Tb's.
        pilotneko 5y ago
        What course are you taking? Imagenet is only 150 GB, and Common Crawl is only 320 TB. Big data is a moving target, but I’m comfortable defining it as data too large to fit in memory. Obviously, you can always get a bigger node, my rule is thumb is that if you need generators, you are working with big data.
- morelisp 5y ago
  The article still applies to the things which have replaced Hadoop. The operational, development, and computational overhead of distributing/re-aggregating work remains huge.
grenran 5y ago
Walking to the store is faster than catching a plane
- rovr138 5y ago
  But if you're going to keep adding destinations and going to them, an airport is a better idea.
  - Cthulhu_ 5y ago
    Depends if it's just another store within walking distance. I think the gist of the GP comment was that they use a big tool for a small job. Many small jobs don't make the big tool any more useful.
    - rovr138 5y ago
      Of course, there are uses. But if you have many jobs that you have to maintain and run, standardizing the tool and workflow is really helpful.
  - mellavora 5y ago
    might depend on the distance between destinations
    - rovr138 5y ago
      Fine. Maybe a bus or taxi service. But standardizing helps and hadoop and other tools provides that.
- yakshaving_jgt 5y ago
  I believe the point of the article is that people seem to have a poor intuition for when it’s appropriate to choose the plane over walking.
- throwaway984393 5y ago
  But it's hard to get investors to give you hundreds of millions of dollars to buy tennis shoes, easier to get it for a Gulfstream
deleted 5y ago
[deleted]
truth_seeker 5y ago
Crontab for scheduling + BASH (or NodeJS/Python/Lua) scripts for map-reduce or whatever kind of computation you want to do + NFS v4.1 (something like AWS EFS) will do the job. You get immense flexibility, however the team of developer who is maintaining this must be more skilful.
- klysm 5y ago
  I hate the failure semantics of cron. Unreasonably hard to get right or to not run duplicate jobs. I’ve replaced it with systemd timers and never looked back.
- NavinF 5y ago
  NFS is such a pain in the ass for this use case. If the client crashes, everything’s corrupted. If the server crashes, everything’s corrupted. You can’t even atomically write 4k with O_APPEND. Why anyone would choose to build distributed systems your way is beyond me
donatj 5y ago
I went to a Hadoop workshop in 2016 where the speaker was insistent Hadoop would replace traditional relational databases in the next 5 years. It’s been six and I think the death of relational databases has still been greatly exaggerated. https://twitter.com/donatj/status/740210538320273408 https://twitter.com/donatj/status/740210538320273408
- pjmlp 5y ago
  Having done OLAP and used other DB models like ISAM and xBase, I am a firm beliver that relational databases will keep being the horse worth betting on. Whatever else keeps poping up against them are only usefull in special use cases, not needed for 90% of the common use cases, and even then, it isn't like relational database vendors are frozen in time without improving them.
- Cthulhu_ 5y ago
  Things look to be quite cyclic; ten years ago, NoSQL was pushed as The Future, but these days companies still build their core data on top of a traditional relational database. I don't even know which NoSQL is even popular. It feels like it's been pushed to the fringes, e.g. key/value stores, or the Q in a CQRS setup where it acts more as a readonly cache.
  - chakkepolja 5y ago
    MongoDB is pretty popular among newbie nodejs developers. (That's not a complement, btw).
    - Thaxll 5y ago
      "newbie", maybe get a look outside, MongoDB is used by very large and successful companies. Fortnite which makes multiple $B a year is heavily using MongoDB.
theden 5y ago
Would be interested if the final one-liner would be faster if updated to use the rust equivalent utils like https://github.com/ezrosent/frawk https://github.com/ezrosent/frawk and https://github.com/sharkdp/fd https://github.com/sharkdp/fd
JimmyRuska 5y ago
If you don't have to do any complex sorting or grouping, yes a simple script works way better. It doesn't have the overhead of using the scheduler, or distributing the data into chunks on many servers. Also consider using sqlite, postgres, or your EDW if you have one. Tools like CSVkit and XSV are useful for preprocessing, exploration. I've seen many ETL scripts written where a simple SQL statement would have been better. SQL queries tend to work after a few queries have been verified to be correct, ETL jobs in languages like java can dump mysterious stack traces referencing many frameworks breaking due to data issues, memory issues, or unhandled cases.
- sitkack 5y ago
  As the predictable discussion unfolds below (above probably?), using the best subset you can means your analytics workloads can potentially scale from sqlite to bigquery, going through spark or over to clickhouse. SQL is everywhere and it is fast and kinda portable. I have done sqlite analytics jobs over 4-6GB databases on a desktop machine 10 years ago to generate really complex reports. It worked wonderfully and was the shortest time from raw data (xml, html, csvs) to useful results.
- toto444 5y ago
  > I've seen many ETL scripts written where a simple SQL statement would have been better. SQL queries tend to work after a few queries have been verified to be correct, ETL jobs in languages like java can dump mysterious stack traces referencing many frameworks breaking due to data issues, memory issues, or unhandled cases. This is so true. I write data pipelines at work. I only use SQL to move data around for this very reason. You never get any bug to fix with SQL. You sometimes need to adjust some queries to some new bad input but never ever will you see : 'index out of bound' or 'wrong argument' or whatever.
  - tremon 5y ago
    You sometimes need to adjust some queries to some new bad input but never ever will you see : 'index out of bound' or 'wrong argument' or whatever. Just wait until you encounter the "String or binary data would be truncated" error. I guarantee you it will make you long even for Java stack traces.
charcircuit 5y ago
Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?
- pjmlp 5y ago
  Easy lesson from distributed systems, most of the time is spent in communication instead of actual work.
  - charcircuit 5y ago
    26 minutes worth? That can't be right. Sending 250 MB (uncompressed) to 7 machines shouldn't take that long.
    - usrbinbash 5y ago
      How many sends are we talking about? Into how many messages is the data turned, how often does it get sent around? If I send 1MiB of data by packing it up into messages of 10 byte each, it will ikely be slower than sending 10MiB in a single message. Messages == Overhead. Envelopes, packing, unpacking, parsing, assembling, etc. all eat up cycles.
      - charcircuit 5y ago
        This is a distributed sum. You don't need extra messages being sent around. You send 7 messages giving each node 1/7th of all games. Then 1 message per node is sent with all the tallies to be merged together. And then maybe an extra message to give you the results if you weren't the one that merged it. 15 messages shouldn't take 26 minutes to send.
  - anonymoushn 5y ago
    This is what we've found. For a certain kind of simulation, we can process data on a single core at ~.5x line rate (I think this could be a lot faster if we took this problem seriously) so if we move to some arrangement where hosts have to download the data in order to process it, they can only use 2 cores each, and every 2 cores we want use requires another storage box to download the data from! Paying for hardware to increase "line rate" and sending zstd-compressed files over the wire might get us a factor of ~60 here, but even then we pretty rapidly run into issues where the data has to be placed on a bunch of distinct hosts in order to be useful, since 120 cores is not a lot of machines. In the future we expect to have workloads that do less computation per piece of data, which makes the need to move our compute to our data much more acute.
kohlerm 5y ago
There is no point running something like Hadoop or Spark or Flink if your problem is not bottlenecked by the resources of of a single machine. E.g. how would this solution work if the data is so large or the problem requires more context (looking at more than one line at the time) and therefore more memory? Answer: it wouldn't .
- Cthulhu_ 5y ago
  That's it, for a long time, what a lot of companies that jumped on the big data bandwagon didn't realize is that they didn't actually have big data. I'm sure there's some arbitrary lines that can be drawn, but anything under 1 TB is not big data anymore and can be processed on a single machine. Other things to consider is data volume though. A popular use case of hadoop was to take e.g. server access logs - high volume data that traditionally you wouldn't hold onto for long - and get something meaningful out of that.
  - rizkeyz 5y ago
    At this point in time (2022) I consider everything below say 40TB not big (textual) data at all. It can be compressed 40TB -> 10TB (or less) and that fits fine on a single 16T drive. For many questions, you won't need all the raw data, so you end up with some form of projection of the data that is maybe 1/10 in size, so 10TB -> 1TB. Heck, if you tune GNU sort a bit, it will blast through that TB quite quickly.
    - trashtester 5y ago
      If you just want cold storage, you can put 10TB of compressed textual data on a spinning hard drive. If want to run some processing of that data within a workday, you need multiple drives in parallel (still possible on a single server). However, if you want to process it in less than 30 mins you need a cluster.
    - anonymousDan 5y ago
      The problem isn't necessarily (just) capacity. It's also the he I/O bandwidth needed to read the data.
oxplot 5y ago
Hmm, no mention of GNU Parallel — that's my go-to replacement for xargs. It has numerous options to treat input as blocks of text, lines, etc and can even run in a distributed fashion over SSH.
ed_elliott_asc 5y ago
It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Logging and diagnostics are available and you can report on metrics easily. That isn’t true with custom data pipelines written in shell scripts. The value isn’t in the pure execution time, it is in everything around it.
- jbverschoor 5y ago
  Processes that take ages are never "easy to write" nor "easy to troubleshoot". This turns into shitty code, because you won't get any sane person to spend weeks instead of days on that. It's bad for morale. It's the same narrative people use for many complexity or slow things
- wodenokoto 5y ago
  Can't you just ask one of the machines in the databricks/spark cluster to run those shell commands? Or is that more of a kubernetes thing?
  - mr_toad 5y ago
    You can use Linux shell commands in Map Reduce jobs using Hadoop streaming.
  - westurner 5y ago
    That's pretty much how Dask works; though Dask doesn't throw away data types for newline-delimited strings between each failed partitioned IPC pipeline process, and then the new hire didn't appropriately trap errors in their shell script, so the log messages are chronologically non-sequential and text-only. https://github.com/dask/dask-labextension https://github.com/dask/dask-labextension : > This package provides a JupyterLab extension to manage Dask clusters, as well as embed Dask's dashboard plots directly into JupyterLab panes. Something like ml-hub allows MLops teams to create resource-quota'd containers with k8s and IAM, though even signed code can DoS an unauditable system with no logs of which processes ran which signed archive of which code at what time, with bash and ssh. https://github.com/ml-tooling/ml-hub https://github.com/ml-tooling/ml-hub
renonce 5y ago
I wonder what is the best way to process huge amounts of time-series data (several thousand records per second). When I search for "time series database" I get something like MySQL, InfluxDB, TimescaleDB etc, but all of them are way too powerful and have their own query language and storage engine etc which are hard to learn and manipulate. In case you run out of storage/memory/CPU etc or when you want to query something that the database doesn't support, there is no way out of it without waiting for the database vendor to provide some new feature. Now I'm just writing all data to plain text files, one JSON object per line, and query and process them with cli tools like jq. Regular compression tools like zstd, pattern matchers like grep works. All the Unix philosophy applies and it's easy to do anything I want without being restricted to the features of a certain database.
- LoriP 5y ago
  Just wanted to say that TimescaleDB is an extension of PostgreSQL and so it uses SQL. MySQL would use SQL too. InfluxDB does, indeed, have its own query language though.
- Beltalowda 5y ago
  TimescaleDB "upgrades" fairly gracefully from regular PostgreSQL; there's a bunch of Timescale-specific functions, but it's not that much: mostly it's just regular SQL. I don't know how well it performs with thousand records per second though.
- spaniard89277 5y ago
  Mmm maybe DuckDB?
- ajoseps 5y ago
  I have a similar issue. I've been using feather files/parquet files for storage, and just using pandas to do analysis. There is an issue where the initial load/convert essentially doubles the memory usage of the file itself as it converts to a pandas dataframe. This can be avoided if you use a feather file and follow its recommendations for a zero-copy conversion (no NaNs/nulls). I think it's a bit more flexible than using cli tools since you can set some sort of time index and query specific timeslices fairly easily
kenoph 5y ago
You can do a lot with just bash + pipes + unix tools. It can get messy as your pipeline grows though, and there are a lot of edge cases. Relevant: "bashML: Why Spark when you can Bash?" (https://rev.ng/blog/bashml/post.html https://rev.ng/blog/bashml/post.html), aka how to deduplicate git repositories using `comm` + `awk`.
- yung_steezy 5y ago
  Bash is a godsend for quick debugging and I can see the temptation to start writing production code using bash. It basically boils down to a few things IMO: - large bash scripts are hard to read/maintain - complex modelling chains need intermediary points in the processing On the latter point I can't count the amount of times where being able to query an athena database has saved a lot of headaches. The overhead from parquet and AWS bills pales in comparison. I'm sure almost everyone already agrees with me here but it's a classic case of the whole being more than the sum of its parts.
  - randomswede 5y ago
    And bash is really painful once you're trying to do clever things with structured data. Excellent glue, but there's also a skill in knowing when you should port your increasingly complicated shell script to another language.
treeman79 5y ago
25 years ago my company Needed to match lists against a collection numbers not to call. Probably ran lists of 20k daily or more. So I wrote up a simple python program to match against a special format txt file. A few seconds for big lists at most. Higher up IT people were horrified, and commissioned a proper oracle soliton. It ran around a 1-2 records per second. So all day for a typical list. They were quite proud of this. I think they spent 25-50 grand on IBM support to get it setup. They were not happy when we said we would only consider using it if they could speed it up 5000 times faster. We never heard back.
- AitchEmArsey 5y ago
  > proper oracle soliton I know this was a typo, but the idea of Oracle upselling someone on a quantum field theory based solution to a string matching problem amuses me greatly.
gigatexal 5y ago
I remember when this first was posted and laughed. My investment in learning unix tools has helped me a ton just as a single dev working on small datasets but I can totally see how it could work here.
ImaCake 5y ago
Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of things from a single bash script and they will be automatically allocated to different cores on whatever node/machine you are on. Just append `&` to each line and stick a `wait` at the end and voila, you have a very hack-y but robust way of running a bunch of processes in parallel.
- efxhoy 5y ago
  If you care about errors you can use function wait_all { for job in $(jobs -p); do wait "${job}"; done } A lone wait wont tell you if any of the jobs fail.
- TacticalCoder 5y ago
  Forking in the background is one way, you can also use GNU "parallel" ("apt-get install parallel" for those on Debian and the likes). For example I've got all my CD ripped to FLAC files, but my car only takes mp3 or wav... So I did a batch convert of FLAC to mp3, making sure to put all cores at work by piping the output of "find" into "parallel". Some commands also allows to directly parallelize (like, say, "make -j 16 ..." to build using 16 cores/hyperthreads).
  - chockchocschoir 5y ago
    Another tip is using wget2 instead of wget if you're mirroring a site (but this is more I/O tip than computationally heavy) https://gitlab.com/gnuwget/wget2/-/wikis/home https://gitlab.com/gnuwget/wget2/-/wikis/home Sadly, wget2 doesn't support WARC last time I checked, but wget2 comes with a `--max-threads` parameter that together with `--mirror` and `--tries` makes it trivial to mirror even the slowest websites out there. Edit: your parallel to `parallel` made me think of wget2 as I often see scripts that use `parallel` together with `wget` when `wget2` can be much better to use alone instead of pairing the two. Just wanted to add some context.
- burntsushi 5y ago
  Do note that unlike GNU (or BSD) grep, ripgrep will automatically use multiple threads to execute a search on a directory.
athenot 5y ago
Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutions first, before entertaining something bigger. Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice.
- chockchocschoir 5y ago
  > Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can handle fitting all your data in RAM: https://yourdatafitsinram.net/ https://yourdatafitsinram.net/ Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data. Just like parent said here, do try to fit things on machines like this before even trying out large-scale storage clusters, because they are a hassle to deal with and generally not worth the cost. Edit: For the Cloud-hosting crowd out there, I took a quick look at https://instances.vantage.sh/ https://instances.vantage.sh/ and sorted it by memory. Largest instances you can get at AWS is "U-24TB1 Metal", which comes with 24TiB of RAM (and 448 vCPU so the computation itself gets as fast as possible too [granted you can parallelize it]), which should fit most cases of "big data" I've seen in the wild today. Unsure about running costs though, but I'm 90% sure it's cheaper to have that instance for a couple of hours than the cost for having to deal with storage clusters, which are also slower.
  - phkahler 5y ago
    >> And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! That's why the "big data" industry also encourages collection of absolutely every bit of data you can find. They want you to need their tools. You may not think there's a use for it, but vague promises of AI finding needles in the haystack are used to get you to keep it and bloat your system.
sriku 5y ago
This reminded me of Frank McSherry's article "Scalability! But at what COST?" (2015) - https://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html https://www.frankmcsherry.org/graph/scalability/cost/2015/01... Where he shows a single laptop beating a spark cluster.
api 5y ago
The mis-use of "big data" tools like Hadoop to do with entire giant clusters of systems what can be done more quickly on a MacBook Air encapsulates quite a bit of what is wrong with all modern software development: overengineering, dick size based infrastructure decisions ("look at how big my cluster is!"), stacking up tools to pad one's resume, mindlessly copying what bigger companies do, needless layers of abstraction, etc. Look up the largest hard drive of any type that you can find for sale. Now spec out a consumer or small business grade NAS with 2-4 of those drives. If your data will fit there, you do not have "big data." If the cost bothers you consider that the cloud footprint (or on-prem mini data center) required to use your big sexy "big data" approach will cost far more than one of those NAS systems, possibly every month. The only real exception is if you need performance and the computations you are doing are CPU bound or highly parallelizable. If you need rapid turnaround you may want some kind of distributed replicated cluster approach that can do things in parallel. For the majority of jobs though these are periodic or internal facing analytics jobs and getting the results faster is not worth 10X-100X the hardware cost or cloud bill and 10X the developer time.
lildata 5y ago
I'm surprised nobody mentioned nushell already https://www.nushell.sh/book/dataframes.html https://www.nushell.sh/book/dataframes.html I guess supporting high perf dataframes in a shell will make those big data platforms irrelevant for most use cases.
mathogre 5y ago
At my old company, Hadoop, JSON, Parquet were the solutions looking for a problem to solve. Yes we handled lots of data, but none of it was time critical. No matter, we had BIG DATA™. Bow when I say that. We weren't Google or Twitter. We were not surgically retrieving data, and while we had lots of data, we didn't have huge data centers; Dremel wasn't relevant to us. It's funny! It would take our Hadoop team three weeks to get data together for our use. I often didn't have three weeks. In those times when I needed to use the data from the previous day, I'd just grab the raw data, organize it, process it, and be done with it in a few hours, using Unix/Linux tools and a bit of mathemagical wizardry. "You're supposed to use Hadoop." "You wanted to know what happened yesterday." "I did!" "If you want it from Hadoop, it will be ready in three weeks. Probably. That's if they have everything done." <Crickets>
- AtlasBarfed 5y ago
  A formal enterprise data group should probably - be a clone/repo for disparate databases so you don't need to figure out access/security/location or impact production systems - an interface to management types that aren't technical or don't have tech people to do these things - should provide a "librarian" knowledge of the enterprise's data and data sources - should have knowledge on how to analyze data using different tools - be able to schedule movements/reports and manage that If you don't need any of that, then ... yeah, don't use it. But those sets of requirements should be useful to anything that deems itself an "enterprise".
elmolino89 5y ago
I had to ingest and explore enough dirty CSV/TSV data to use CLI tools only to get a first glimpse what's there. Whenever facing anything non trivial I go for CSV to parquet, and preferably write intermediate parquet data sets. Then DuckDB / SQL queries for slicing and dicing. I am yet to encounter some readable to a newcomers awk/uniq/sed combos for intersecting bunch of CSVs with 30+ columns. On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it.
- p33p 5y ago
  > On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it. I thought parquet was columnar stored? Is this a fault of parquet or just the shear number of columns trying to get accessed? I agree with your general premise though. I'd rather take a dirty dataset, throw it into S3, spin up a Redshift cluster, do what I need, spin down the cluster. You can work with billions of records fairly easily with plain old SQL and c-store databases.
slowhand09 5y ago
I've seen this article before and still consider it valid, especially for one-offs. Another similar and valuable resource is "https://datascienceatthecommandline.com/ https://datascienceatthecommandline.com/" Data Science at the Command Line, by Jeronen Janssens. Free to read at the link.
ufmace 5y ago
I did a vaguely similar thing once. We once upon a time had a EMR / Hadoop based process that simply joined a some thousands of files of a few kb-ish from S3 each into one big file totaling 20GB or so. It worked, but took a while. After a bit of mucking with it, I noticed that the built-in tools for it used the wrong S3 API for the purpose. I wrote a simple dumb single-threaded Ruby script using the right API, and it ran like 4x faster IIRC. Same lesson - before you reach for Big Data tools, make sure you've fully explored simple conventional solutions.
vs4vijay 5y ago
I read similar on HN sometime back, this is call Taco Bell Programming - http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...

362 points by graderjs 5y ago

This article is such a good demonstration (IMO) that computers are really fast, but that we have gotten really bad at actually making reasonable use of them.

wenc 5y ago

This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=command%20line%20tools%20hadoop&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.

commandlinefan 5y ago
Well, a lot of AWS and GCP infrastructure use it, but they hide it from you.
- fulafel 5y ago
  Any specifics about which services?
  - ryapric 5y ago
    AWS: Glue, and by proxy Athena GCP: Dataproc Those are just the obvious ones though.
    - mr_toad 5y ago
      Glue often uses EMR under the hood, which is often Spark. And Athena is PrestoDB, as far as I know it has nothing to do with Hadoop other than you can use it to query Hadoop data stores.
      - trashtester 5y ago
        The way I see it, Hadoop is still in common use as the storage layer for Spark and related implementations, whether that is in the form of HDFS or something like EMRFS: Quote from AWS: "EMRFS is an implementation of the Hadoop file system ..." https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-file-systems.html https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...
        commandlinefan 5y ago
        Ah - that's a good point. Usually when people say "Hadoop", I assume they're referring to HDFS, but there is (was...) the Hadoop MR system that ran on top of it that has been almost entirely replaced by Spark.
jamal-kumar 5y ago
Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?
- bpodgursky 5y ago
  I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.
  - mohanmcgeek 5y ago
    But I can write SQL in Spark just fine. Can't I?
    - bpodgursky 5y ago
      You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is doing on the backend to support their SQL performance, but it's really freaking fast. There's just no other way to cut it. I've spent 5-10 years eking every ounce of performance I can get out of a Hadoop/Spark cluster. I'm not trying to be unreasonable about this. I would love for OSS to be competitive; it's great for the world, and it would be great for my skill set and earning potential. But it's not a contest, and if you think standalone Spark is going to be a viable competitor in a couple years, you are deluding yourself. Make informed choices about your career and investment.
      - rxin 5y ago
Puts 5y ago
Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.
- spullara 5y ago
  I mean, it isn't like Hadoop wasn't used to parse text files. Also, it is all fun and games until you need a join.
  - kragen 5y ago
    comm(1) and join(1) can do joins on text files. Make sure they're sorted in the same locale you're joining them in; LANG=C tends to be the fastest.
- IanCal 5y ago
  The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.
  - stingraycharles 5y ago
    I’d argue that for as long as Excel doesn’t support Parquet files, we haven’t seen the last of CSVs for a long time. Parquet is great, but it’s simply nowhere near as ubiquitous as CSV.
    - gehen88 5y ago
      I hadn't even heard of Parquet until now, and I'm sure this goes for lots of developers who don't do much data engineering.
      - jsjohnst 5y ago
StreamBright 5y ago
> Outside of legacy systems, Hadoop isn't widely used anymore. This is not true at all. Almost all the cloud providers have their own Hadoop distributions that is used a lot in many companies.
- rizkeyz 5y ago
  Exactly. Basically most big data lives in hdfs and hdfs is part of hadoop. Even if you use Spark and Flink I would classify that as using Hadoop (under the hood). > However, a very common setup is to use Flink to analyze data stored in the Hadoop Distributed File System (HDFS). -- https://wints.github.io/flink-web//faq.html https://wints.github.io/flink-web//faq.html
tus666 5y ago
What about if you had 500 petabytes of data to analyze? Big data does exist in the wild.
- usrbinbash 5y ago
  > What about if you had 500 petabytes of data to analyze? Then we probably need the big tools. > Big data does exist in the wild. So does little data. The problem is that a "one-size-fits-all" approach has become common, not just in data analysis; think of all the low-medium traffic webpages that use giant, complex frameworks and huge distributed systems just to display essentially a small CRUD app that would have been ALOT easier to cobble together in plain JS on a simple LAMP server. What the article shows is the importance on deciding for the right tool for the job: When I want to plant a little tree in my backyard, bringing one of these https://upload.wikimedia.org/wikipedia/commons/0/01/Bucket_wheel_excavator_in_Ferropolis.jpg https://upload.wikimedia.org/wikipedia/commons/0/01/Bucket_w... to dig the hole is proooobably overengineering it a tiny little bit, and will likely take longer than getting a shovel.
- HelloNurse 5y ago
  Those 500 PB imply a competent staff, a world-class infrastructure, and worthwhile applications. These things don't materialize suddenly: they start maturing when the data set is 500 GB or even just planned and they usually mature enough to switch technology multiple times as the scale rises.
auggierose 5y ago
So what is used instead of Hadoop currently?
- serial_dev 5y ago
  Whether a technology can replace Hadoop in an organization depends on many factors, but some technologies that solve at least in part similar problem are Apache Storm, Spark, Flink, Kafka Streams, and maybe BigQuery? Or, as the original article says, some companies just use some command line tools, shell scripts. It's been a couple of years since I was interested in Data Engineering, so my knowledge on this topic is some years behind.
  - riknox 5y ago
    I've not seen Storm being used anywhere sane for a few years at least now, and from a glance at job postings it looks unlikely. Spark, Kafka Streams etc. are definitely used in a modern data platform from my experience. I think we're seeing a big shift with Hadoop-like workloads being moved onto cloud providers, so BigQuery, Amazon EMR etc.
    - bsenftner 5y ago
      I'm curious what constitutes "big data" anymore. In an intermediate machine learning course, we train on nearly a petabyte of data using Google Colab and Jupyter Notebooks. Nobody discusses the size of the data requiring any special treatment due to its size... would not 95% of a petabyte be "big data"?
      - happymellon 5y ago
        Big data is a shifting concept as computers gain more storage and faster commodity processors. My general rule of thumb is whether it is too big to put on my laptop. So greater than a couple of Tb's.
      - pilotneko 5y ago
        What course are you taking? Imagenet is only 150 GB, and Common Crawl is only 320 TB. Big data is a moving target, but I’m comfortable defining it as data too large to fit in memory. Obviously, you can always get a bigger node, my rule is thumb is that if you need generators, you are working with big data.
morelisp 5y ago
The article still applies to the things which have replaced Hadoop. The operational, development, and computational overhead of distributing/re-aggregating work remains huge.

grenran 5y ago

Walking to the store is faster than catching a plane

rovr138 5y ago
But if you're going to keep adding destinations and going to them, an airport is a better idea.
- Cthulhu_ 5y ago
  Depends if it's just another store within walking distance. I think the gist of the GP comment was that they use a big tool for a small job. Many small jobs don't make the big tool any more useful.
  - rovr138 5y ago
    Of course, there are uses. But if you have many jobs that you have to maintain and run, standardizing the tool and workflow is really helpful.
- mellavora 5y ago
  might depend on the distance between destinations
  - rovr138 5y ago
    Fine. Maybe a bus or taxi service. But standardizing helps and hadoop and other tools provides that.
yakshaving_jgt 5y ago
I believe the point of the article is that people seem to have a poor intuition for when it’s appropriate to choose the plane over walking.
throwaway984393 5y ago
But it's hard to get investors to give you hundreds of millions of dollars to buy tennis shoes, easier to get it for a Gulfstream

deleted 5y ago

[deleted]

truth_seeker 5y ago

Crontab for scheduling + BASH (or NodeJS/Python/Lua) scripts for map-reduce or whatever kind of computation you want to do + NFS v4.1 (something like AWS EFS) will do the job. You get immense flexibility, however the team of developer who is maintaining this must be more skilful.

klysm 5y ago
I hate the failure semantics of cron. Unreasonably hard to get right or to not run duplicate jobs. I’ve replaced it with systemd timers and never looked back.
NavinF 5y ago
NFS is such a pain in the ass for this use case. If the client crashes, everything’s corrupted. If the server crashes, everything’s corrupted. You can’t even atomically write 4k with O_APPEND. Why anyone would choose to build distributed systems your way is beyond me

donatj 5y ago

I went to a Hadoop workshop in 2016 where the speaker was insistent Hadoop would replace traditional relational databases in the next 5 years. It’s been six and I think the death of relational databases has still been greatly exaggerated. https://twitter.com/donatj/status/740210538320273408 https://twitter.com/donatj/status/740210538320273408

pjmlp 5y ago
Having done OLAP and used other DB models like ISAM and xBase, I am a firm beliver that relational databases will keep being the horse worth betting on. Whatever else keeps poping up against them are only usefull in special use cases, not needed for 90% of the common use cases, and even then, it isn't like relational database vendors are frozen in time without improving them.
Cthulhu_ 5y ago
Things look to be quite cyclic; ten years ago, NoSQL was pushed as The Future, but these days companies still build their core data on top of a traditional relational database. I don't even know which NoSQL is even popular. It feels like it's been pushed to the fringes, e.g. key/value stores, or the Q in a CQRS setup where it acts more as a readonly cache.
- chakkepolja 5y ago
  MongoDB is pretty popular among newbie nodejs developers. (That's not a complement, btw).
  - Thaxll 5y ago
    "newbie", maybe get a look outside, MongoDB is used by very large and successful companies. Fortnite which makes multiple $B a year is heavily using MongoDB.

theden 5y ago

Would be interested if the final one-liner would be faster if updated to use the rust equivalent utils like https://github.com/ezrosent/frawk https://github.com/ezrosent/frawk and https://github.com/sharkdp/fd https://github.com/sharkdp/fd

JimmyRuska 5y ago

If you don't have to do any complex sorting or grouping, yes a simple script works way better. It doesn't have the overhead of using the scheduler, or distributing the data into chunks on many servers. Also consider using sqlite, postgres, or your EDW if you have one. Tools like CSVkit and XSV are useful for preprocessing, exploration. I've seen many ETL scripts written where a simple SQL statement would have been better. SQL queries tend to work after a few queries have been verified to be correct, ETL jobs in languages like java can dump mysterious stack traces referencing many frameworks breaking due to data issues, memory issues, or unhandled cases.

sitkack 5y ago
As the predictable discussion unfolds below (above probably?), using the best subset you can means your analytics workloads can potentially scale from sqlite to bigquery, going through spark or over to clickhouse. SQL is everywhere and it is fast and kinda portable. I have done sqlite analytics jobs over 4-6GB databases on a desktop machine 10 years ago to generate really complex reports. It worked wonderfully and was the shortest time from raw data (xml, html, csvs) to useful results.
toto444 5y ago
> I've seen many ETL scripts written where a simple SQL statement would have been better. SQL queries tend to work after a few queries have been verified to be correct, ETL jobs in languages like java can dump mysterious stack traces referencing many frameworks breaking due to data issues, memory issues, or unhandled cases. This is so true. I write data pipelines at work. I only use SQL to move data around for this very reason. You never get any bug to fix with SQL. You sometimes need to adjust some queries to some new bad input but never ever will you see : 'index out of bound' or 'wrong argument' or whatever.
- tremon 5y ago
  You sometimes need to adjust some queries to some new bad input but never ever will you see : 'index out of bound' or 'wrong argument' or whatever. Just wait until you encounter the "String or binary data would be truncated" error. I guarantee you it will make you long even for Java stack traces.

charcircuit 5y ago

Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?

pjmlp 5y ago
Easy lesson from distributed systems, most of the time is spent in communication instead of actual work.
- charcircuit 5y ago
  26 minutes worth? That can't be right. Sending 250 MB (uncompressed) to 7 machines shouldn't take that long.
  - usrbinbash 5y ago
    How many sends are we talking about? Into how many messages is the data turned, how often does it get sent around? If I send 1MiB of data by packing it up into messages of 10 byte each, it will ikely be slower than sending 10MiB in a single message. Messages == Overhead. Envelopes, packing, unpacking, parsing, assembling, etc. all eat up cycles.
    - charcircuit 5y ago
      This is a distributed sum. You don't need extra messages being sent around. You send 7 messages giving each node 1/7th of all games. Then 1 message per node is sent with all the tallies to be merged together. And then maybe an extra message to give you the results if you weren't the one that merged it. 15 messages shouldn't take 26 minutes to send.
- anonymoushn 5y ago
  This is what we've found. For a certain kind of simulation, we can process data on a single core at ~.5x line rate (I think this could be a lot faster if we took this problem seriously) so if we move to some arrangement where hosts have to download the data in order to process it, they can only use 2 cores each, and every 2 cores we want use requires another storage box to download the data from! Paying for hardware to increase "line rate" and sending zstd-compressed files over the wire might get us a factor of ~60 here, but even then we pretty rapidly run into issues where the data has to be placed on a bunch of distinct hosts in order to be useful, since 120 cores is not a lot of machines. In the future we expect to have workloads that do less computation per piece of data, which makes the need to move our compute to our data much more acute.

kohlerm 5y ago

There is no point running something like Hadoop or Spark or Flink if your problem is not bottlenecked by the resources of of a single machine. E.g. how would this solution work if the data is so large or the problem requires more context (looking at more than one line at the time) and therefore more memory? Answer: it wouldn't .

Cthulhu_ 5y ago
That's it, for a long time, what a lot of companies that jumped on the big data bandwagon didn't realize is that they didn't actually have big data. I'm sure there's some arbitrary lines that can be drawn, but anything under 1 TB is not big data anymore and can be processed on a single machine. Other things to consider is data volume though. A popular use case of hadoop was to take e.g. server access logs - high volume data that traditionally you wouldn't hold onto for long - and get something meaningful out of that.
- rizkeyz 5y ago
  At this point in time (2022) I consider everything below say 40TB not big (textual) data at all. It can be compressed 40TB -> 10TB (or less) and that fits fine on a single 16T drive. For many questions, you won't need all the raw data, so you end up with some form of projection of the data that is maybe 1/10 in size, so 10TB -> 1TB. Heck, if you tune GNU sort a bit, it will blast through that TB quite quickly.
  - trashtester 5y ago
    If you just want cold storage, you can put 10TB of compressed textual data on a spinning hard drive. If want to run some processing of that data within a workday, you need multiple drives in parallel (still possible on a single server). However, if you want to process it in less than 30 mins you need a cluster.
  - anonymousDan 5y ago
    The problem isn't necessarily (just) capacity. It's also the he I/O bandwidth needed to read the data.

oxplot 5y ago

Hmm, no mention of GNU Parallel — that's my go-to replacement for xargs. It has numerous options to treat input as blocks of text, lines, etc and can even run in a distributed fashion over SSH.

ed_elliott_asc 5y ago

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Logging and diagnostics are available and you can report on metrics easily. That isn’t true with custom data pipelines written in shell scripts. The value isn’t in the pure execution time, it is in everything around it.

jbverschoor 5y ago
Processes that take ages are never "easy to write" nor "easy to troubleshoot". This turns into shitty code, because you won't get any sane person to spend weeks instead of days on that. It's bad for morale. It's the same narrative people use for many complexity or slow things
wodenokoto 5y ago
Can't you just ask one of the machines in the databricks/spark cluster to run those shell commands? Or is that more of a kubernetes thing?
- mr_toad 5y ago
  You can use Linux shell commands in Map Reduce jobs using Hadoop streaming.
- westurner 5y ago
  That's pretty much how Dask works; though Dask doesn't throw away data types for newline-delimited strings between each failed partitioned IPC pipeline process, and then the new hire didn't appropriately trap errors in their shell script, so the log messages are chronologically non-sequential and text-only. https://github.com/dask/dask-labextension https://github.com/dask/dask-labextension : > This package provides a JupyterLab extension to manage Dask clusters, as well as embed Dask's dashboard plots directly into JupyterLab panes. Something like ml-hub allows MLops teams to create resource-quota'd containers with k8s and IAM, though even signed code can DoS an unauditable system with no logs of which processes ran which signed archive of which code at what time, with bash and ssh. https://github.com/ml-tooling/ml-hub https://github.com/ml-tooling/ml-hub

renonce 5y ago

I wonder what is the best way to process huge amounts of time-series data (several thousand records per second). When I search for "time series database" I get something like MySQL, InfluxDB, TimescaleDB etc, but all of them are way too powerful and have their own query language and storage engine etc which are hard to learn and manipulate. In case you run out of storage/memory/CPU etc or when you want to query something that the database doesn't support, there is no way out of it without waiting for the database vendor to provide some new feature. Now I'm just writing all data to plain text files, one JSON object per line, and query and process them with cli tools like jq. Regular compression tools like zstd, pattern matchers like grep works. All the Unix philosophy applies and it's easy to do anything I want without being restricted to the features of a certain database.

LoriP 5y ago
Just wanted to say that TimescaleDB is an extension of PostgreSQL and so it uses SQL. MySQL would use SQL too. InfluxDB does, indeed, have its own query language though.
Beltalowda 5y ago
TimescaleDB "upgrades" fairly gracefully from regular PostgreSQL; there's a bunch of Timescale-specific functions, but it's not that much: mostly it's just regular SQL. I don't know how well it performs with thousand records per second though.
spaniard89277 5y ago
Mmm maybe DuckDB?
ajoseps 5y ago
I have a similar issue. I've been using feather files/parquet files for storage, and just using pandas to do analysis. There is an issue where the initial load/convert essentially doubles the memory usage of the file itself as it converts to a pandas dataframe. This can be avoided if you use a feather file and follow its recommendations for a zero-copy conversion (no NaNs/nulls). I think it's a bit more flexible than using cli tools since you can set some sort of time index and query specific timeslices fairly easily

kenoph 5y ago

You can do a lot with just bash + pipes + unix tools. It can get messy as your pipeline grows though, and there are a lot of edge cases. Relevant: "bashML: Why Spark when you can Bash?" (https://rev.ng/blog/bashml/post.html https://rev.ng/blog/bashml/post.html), aka how to deduplicate git repositories using `comm` + `awk`.

yung_steezy 5y ago
Bash is a godsend for quick debugging and I can see the temptation to start writing production code using bash. It basically boils down to a few things IMO: - large bash scripts are hard to read/maintain - complex modelling chains need intermediary points in the processing On the latter point I can't count the amount of times where being able to query an athena database has saved a lot of headaches. The overhead from parquet and AWS bills pales in comparison. I'm sure almost everyone already agrees with me here but it's a classic case of the whole being more than the sum of its parts.
- randomswede 5y ago
  And bash is really painful once you're trying to do clever things with structured data. Excellent glue, but there's also a skill in knowing when you should port your increasingly complicated shell script to another language.

treeman79 5y ago

25 years ago my company Needed to match lists against a collection numbers not to call. Probably ran lists of 20k daily or more. So I wrote up a simple python program to match against a special format txt file. A few seconds for big lists at most. Higher up IT people were horrified, and commissioned a proper oracle soliton. It ran around a 1-2 records per second. So all day for a typical list. They were quite proud of this. I think they spent 25-50 grand on IBM support to get it setup. They were not happy when we said we would only consider using it if they could speed it up 5000 times faster. We never heard back.

AitchEmArsey 5y ago
> proper oracle soliton I know this was a typo, but the idea of Oracle upselling someone on a quantum field theory based solution to a string matching problem amuses me greatly.

gigatexal 5y ago

I remember when this first was posted and laughed. My investment in learning unix tools has helped me a ton just as a single dev working on small datasets but I can totally see how it could work here.

ImaCake 5y ago

Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of things from a single bash script and they will be automatically allocated to different cores on whatever node/machine you are on. Just append `&` to each line and stick a `wait` at the end and voila, you have a very hack-y but robust way of running a bunch of processes in parallel.

efxhoy 5y ago
If you care about errors you can use function wait_all { for job in $(jobs -p); do wait "${job}"; done } A lone wait wont tell you if any of the jobs fail.
TacticalCoder 5y ago
Forking in the background is one way, you can also use GNU "parallel" ("apt-get install parallel" for those on Debian and the likes). For example I've got all my CD ripped to FLAC files, but my car only takes mp3 or wav... So I did a batch convert of FLAC to mp3, making sure to put all cores at work by piping the output of "find" into "parallel". Some commands also allows to directly parallelize (like, say, "make -j 16 ..." to build using 16 cores/hyperthreads).
- chockchocschoir 5y ago
  Another tip is using wget2 instead of wget if you're mirroring a site (but this is more I/O tip than computationally heavy) https://gitlab.com/gnuwget/wget2/-/wikis/home https://gitlab.com/gnuwget/wget2/-/wikis/home Sadly, wget2 doesn't support WARC last time I checked, but wget2 comes with a `--max-threads` parameter that together with `--mirror` and `--tries` makes it trivial to mirror even the slowest websites out there. Edit: your parallel to `parallel` made me think of wget2 as I often see scripts that use `parallel` together with `wget` when `wget2` can be much better to use alone instead of pairing the two. Just wanted to add some context.
burntsushi 5y ago
Do note that unlike GNU (or BSD) grep, ripgrep will automatically use multiple threads to execute a search on a directory.

athenot 5y ago

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutions first, before entertaining something bigger. Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice.

chockchocschoir 5y ago
> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can handle fitting all your data in RAM: https://yourdatafitsinram.net/ https://yourdatafitsinram.net/ Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data. Just like parent said here, do try to fit things on machines like this before even trying out large-scale storage clusters, because they are a hassle to deal with and generally not worth the cost. Edit: For the Cloud-hosting crowd out there, I took a quick look at https://instances.vantage.sh/ https://instances.vantage.sh/ and sorted it by memory. Largest instances you can get at AWS is "U-24TB1 Metal", which comes with 24TiB of RAM (and 448 vCPU so the computation itself gets as fast as possible too [granted you can parallelize it]), which should fit most cases of "big data" I've seen in the wild today. Unsure about running costs though, but I'm 90% sure it's cheaper to have that instance for a couple of hours than the cost for having to deal with storage clusters, which are also slower.
- phkahler 5y ago
  >> And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! That's why the "big data" industry also encourages collection of absolutely every bit of data you can find. They want you to need their tools. You may not think there's a use for it, but vague promises of AI finding needles in the haystack are used to get you to keep it and bloat your system.

sriku 5y ago

This reminded me of Frank McSherry's article "Scalability! But at what COST?" (2015) - https://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html https://www.frankmcsherry.org/graph/scalability/cost/2015/01... Where he shows a single laptop beating a spark cluster.

api 5y ago

The mis-use of "big data" tools like Hadoop to do with entire giant clusters of systems what can be done more quickly on a MacBook Air encapsulates quite a bit of what is wrong with all modern software development: overengineering, dick size based infrastructure decisions ("look at how big my cluster is!"), stacking up tools to pad one's resume, mindlessly copying what bigger companies do, needless layers of abstraction, etc. Look up the largest hard drive of any type that you can find for sale. Now spec out a consumer or small business grade NAS with 2-4 of those drives. If your data will fit there, you do not have "big data." If the cost bothers you consider that the cloud footprint (or on-prem mini data center) required to use your big sexy "big data" approach will cost far more than one of those NAS systems, possibly every month. The only real exception is if you need performance and the computations you are doing are CPU bound or highly parallelizable. If you need rapid turnaround you may want some kind of distributed replicated cluster approach that can do things in parallel. For the majority of jobs though these are periodic or internal facing analytics jobs and getting the results faster is not worth 10X-100X the hardware cost or cloud bill and 10X the developer time.

lildata 5y ago

I'm surprised nobody mentioned nushell already https://www.nushell.sh/book/dataframes.html https://www.nushell.sh/book/dataframes.html I guess supporting high perf dataframes in a shell will make those big data platforms irrelevant for most use cases.

mathogre 5y ago

At my old company, Hadoop, JSON, Parquet were the solutions looking for a problem to solve. Yes we handled lots of data, but none of it was time critical. No matter, we had BIG DATA™. Bow when I say that. We weren't Google or Twitter. We were not surgically retrieving data, and while we had lots of data, we didn't have huge data centers; Dremel wasn't relevant to us. It's funny! It would take our Hadoop team three weeks to get data together for our use. I often didn't have three weeks. In those times when I needed to use the data from the previous day, I'd just grab the raw data, organize it, process it, and be done with it in a few hours, using Unix/Linux tools and a bit of mathemagical wizardry. "You're supposed to use Hadoop." "You wanted to know what happened yesterday." "I did!" "If you want it from Hadoop, it will be ready in three weeks. Probably. That's if they have everything done." <Crickets>

AtlasBarfed 5y ago
A formal enterprise data group should probably - be a clone/repo for disparate databases so you don't need to figure out access/security/location or impact production systems - an interface to management types that aren't technical or don't have tech people to do these things - should provide a "librarian" knowledge of the enterprise's data and data sources - should have knowledge on how to analyze data using different tools - be able to schedule movements/reports and manage that If you don't need any of that, then ... yeah, don't use it. But those sets of requirements should be useful to anything that deems itself an "enterprise".

elmolino89 5y ago

I had to ingest and explore enough dirty CSV/TSV data to use CLI tools only to get a first glimpse what's there. Whenever facing anything non trivial I go for CSV to parquet, and preferably write intermediate parquet data sets. Then DuckDB / SQL queries for slicing and dicing. I am yet to encounter some readable to a newcomers awk/uniq/sed combos for intersecting bunch of CSVs with 30+ columns. On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it.

p33p 5y ago
> On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it. I thought parquet was columnar stored? Is this a fault of parquet or just the shear number of columns trying to get accessed? I agree with your general premise though. I'd rather take a dirty dataset, throw it into S3, spin up a Redshift cluster, do what I need, spin down the cluster. You can work with billions of records fairly easily with plain old SQL and c-store databases.

slowhand09 5y ago

I've seen this article before and still consider it valid, especially for one-offs. Another similar and valuable resource is "https://datascienceatthecommandline.com/ https://datascienceatthecommandline.com/" Data Science at the Command Line, by Jeronen Janssens. Free to read at the link.

ufmace 5y ago

I did a vaguely similar thing once. We once upon a time had a EMR / Hadoop based process that simply joined a some thousands of files of a few kb-ish from S3 each into one big file totaling 20GB or so. It worked, but took a while. After a bit of mucking with it, I noticed that the built-in tools for it used the wrong S3 API for the purpose. I wrote a simple dumb single-threaded Ruby script using the right API, and it ran like 4x faster IIRC. Same lesson - before you reach for Big Data tools, make sure you've fully explored simple conventional solutions.

vs4vijay 5y ago

I read similar on HN sometime back, this is call Taco Bell Programming - http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...