18 ms·

Command-line Tools can be 235x Faster than a Hadoop Cluster (2014)

438 points by hd4 8y ago

megaman22 8y ago
If I do one thing this year, I need to learn more about these tried and tested command-line unix shell utilities. It's becoming increasingly obvious that so many complicated and overwrought things are being unleashed on the world because (statistically) nobody knows this stuff anymore.
- trumped 8y ago
  They try to make the cloud relevant for a lot of things that could easily be accomplished locally... Most of the time, they do that just to get your data.
  - SteveNuts 8y ago
    Vendor lock-in is what they're after in the cloud case. Every time we talk to our cloud provider reps they ask us which of their SaaS offerings we're using. Their heads spin when we tell them we're just using VMs and ask us why.
- beokop 8y ago
  So much internal tooling at a company can be avoided if you can just get people comfortable with basic *nix programs. No, we don’t need a web interface to show that data. No, we don’t need to code a dropdown to sort it differently.
  - fleitz 8y ago
    Yes, you do need a web interface for that, thankfully we have cgi-bin from the list of unix tools. Turns out browsers can render text :)
  - digi_owl 8y ago
    And yet the UX people, particularly around Linux, wants to bury the _nix cli so damned deep...
    - AnIdiotOnTheNet 8y ago
      Can't say I blame them. The base concept is sound but the implementation is archaic and obtuse as hell.
    - tormeh 8y ago
      CLIs have a discoverability problem. Knowing how to use any given one is a skill that takes years to hone. Learning a single command can take up to an hour. Most webapps are more constrained, but a common operation can usually be accomplished by someone who hasn't seen it before in less than 3 minutes.
      - jonhendry18 8y ago
        The 'apropos' command is very handy on machine or OS you aren't familiar with. If you want to find out what tools are available on the CLI that operate on jpegs, you'd try 'apropos jpeg' and get a list of things that mention jpeg in their man page.
- deckard1 8y ago
  This applies to much of the software world. Many of the places I've worked have a bad habit of ditching perfectly fine tech in order to chase the latest thing. Part of it is driven by RDD (resume driven development) and positioning the company so they can hire the latest crop of fad chasers. You end up with a bunch of specialists that are siloed into their particular tech. You got the Docker guy. You got the Redis guy. You may even have one of those gray-haired DBAs that still talk about this "Oracle" thing all day long, and no one knows what the hell he's going on about. You got the Ruby guys, but now you guy a bunch of React or Angular guys. What to do with the Ruby guys? Can't just fire them yet. You have a dozen of 'em, and it took you years to hire them. Fine, just banish them to the backend somewhere. Give them some fidget spinners and a few pointless projects to keep them busy.
  - sixdimensional 8y ago
    This is exactly right. This is real life!! And it hurts.
  - tomc1985 8y ago
    Hey now, Ruby ain't dead. I am one of these "Ruby" guys and, in the 48-or-so hours since my profile was activated at a new recruiting shop this week, I've gotten several potential candidate employers (some in seed stage) who want to get in touch. The real issue is why do we silo ourselves so? It's fucking stupid. I do so much more than Ruby but conveying that to the new breed of tech people is just impossible. All they see is your biggest resume badge, and all they care about is how wide you will spread your legs for them.
  - monksy 8y ago
    Let's not forget when you're interviewing: The new places expect the latest technologies. Don't have time to learn spark, flink, kafka streams, and repeat scala spec whilst having intimate knowledge of all of the REST libraries? BAD DEVELOPER.
- eb0la 8y ago
  You just need to master a few commands: Find (take a look at -exec option) Cut (or awk) Sed (for simple text/string substitutions) Xargs Dd (that beast can do charset transtation from ASCII to EBCDIC to use in mainframes and can also wipe disks) And bash of course :)
  - sidlls 8y ago
    awk is so beautiful. A good awk one-liner can beat fancy pandas all day long in some cases.
    - flukus 8y ago
      I hate how awk gets so much praise for one liners when it's so good at building things like state machines too: https://two-wrongs.com/awk-state-machine-parser-pattern.html https://two-wrongs.com/awk-state-machine-parser-pattern.html I've got some similar ones for work that take data csv like data and output sql. A little bit of vim-foo on the csv (yy10000p) and I've got all the test data I need.
      - cosmie 8y ago
        If anything, awk should get some praise for it's flexibility to go from one liners to Doom-style 3d games[1]! [1] https://github.com/TheMozg/awk-raycaster https://github.com/TheMozg/awk-raycaster
    - cwilkes 8y ago
      An example here would be nice.
  - hoffbrau99 8y ago
    Don't forget grep!
  - eanzenberg 8y ago
    "master a few commands" "awk" brb 15 years
- thediehl 8y ago
  I started with Unix Shell Programming http://a.co/au9OUrB http://a.co/au9OUrB
- nemild 8y ago
  I made a list of a good set of commands for a full stack engineer: https://github.com/nemild/cli_for_full_stack https://github.com/nemild/cli_for_full_stack
  - sh87 8y ago
    This is awesome! thanks.
- sitkack 8y ago
  You can write many of these yourself. Just reading records, and sending them across pipes in ways you can compose is a good start. I find PyPy is excellent for this kind of work, fast to write, if the pipeline runs for more than 10 seconds, it has plenty of time for the JIT to warm up and amortize the cost. If you don't need fancy libs, LuaJIT is amazing and great to doing transforms in a pipeline. I rewrote a Hadoop job to use Streaming (pipes, basic unix model) from a Java job to native. Just that switch was 10x faster. Mostly because of Hadoop overheads.
- cosmie 8y ago
  Companies aren't making these complicated and overwrought things because nobody knows this stuff anymore. The ones actually doing the making generally know all the tried and tested ways to do what they want, because they used them to scale. Then they eventually outgrow the existing solutions and created these complicated systems to account for whatever specific technical or business issues that necessitated hopping from existing solutions to creating a solution. The problem tends to come along with the visibility that comes when one of these get released publicly. People use these things without doing the same level of due diligence to see if something less complicated fits their needs. Either deliberately (resume driven development) or simply because it doesn't occur to them to do so and "if it's good enough for this unicorn it's good enough for me". Or in some cases they do due diligence, but incorrectly evaluate their needs and rule out simpler solutions. But the most common reason I've ran into is simply that it's more fun to play with these projects than it is to use crufty old stuff, no matter how tried and tested and potentially appropriate it is. At a previous company, I had a python script running on a cron job every 5 minutes to do some data processing needs. Once or twice a month, the batch would be so large that it took 6-7 minutes to complete and the cron job would trigger the script again before it finished, causing the second instance of the script to see the lock file, log an error, and exit. It didn't cause any problem for these periodically skipped runs, because the business need only required data to be processed within 24 hours of coming in. The 5 minute cron job was just to even out resource usage throughout the day instead of doing a larger nightly batch job. A piece of data not getting processed for 8 minutes instead of 5 did not make any material impact. Another team had noticed the errors popping up in the log and were in the process of testing out a whole bunch of real time data pipelines like Kafka, leveraging my error messages to justify the need for a "real time" system without ever even asking me about the errors. After I found out other people were noticing those superfluous errors, I moved the cron job to a 30 minute window to stop them from happening anymore. Turns out they didn't have any other justification for their data pipeline greenfield project and weren't happy to go back to their normal day to day work. I offered to let them maintain and expand my python scripts if they were interested in data processing work, but for some reason they never took me up on that offer. :(
  - gaius 8y ago
    Companies aren't making these complicated and overwrought things because nobody knows this stuff anymore. The ones actually doing the making generally know all the tried and tested ways to do what they want, because they used them to scale Right, Yahoo made Hadoop because (probably) they needed it. 99% of companies... just don’t.
- shawndumas 8y ago
  Read The AWK Programming Language by Alfred Aho, Peter Weinberger, and Brian Kernighan "This is an absolute classic which although ancient in Computing years is an absolute gem full of relevance yet."
- asicsp 8y ago
  I have a repo dedicated to cli text-processing.. short-descriptions and tons of examples.. I had fun and learned a lot doing this link: https://github.com/learnbyexample/Command-line-text-processing https://github.com/learnbyexample/Command-line-text-processi...
  - svde 8y ago
    This looks exactly like what I was looking for. Thank you.
    - asicsp 8y ago
      you're welcome, happy learning :)
ash_gti 8y ago
Shouldn't this be marked 2014? The articles date is January 18, 2014.
- gaius 8y ago
  It’s probably even truer and more relevant now
noobermin 8y ago
It would benefit people to actually understand algorithmic complexity of what they are doing before they go on these voyages to parallelize everything. It also helps to know what helps to parallelize and what doesn't.
3pt14159 8y ago
I once converted a simulation into cython from plain old python. Because it fit in the CPU cache the speedup was around 10000x on a single machine (numerical simulations, amirite?). Because it was so much faster all the code required to split it up between a bunch of servers in a map reduce job could be deleted, since it only needed a couple cores on a single machine for a ms or three. Because it wasn't a map-reduce job, I could take it out of the worker queue and just handle it on the fly during the web request. Sometimes it's worth it to just step back and experiment a bit.
- vvanders 8y ago
  Yeah, back when I was in gamedev land and multi-cores started coming on the scene it was "Multithread ALL THE THINGS". Shortly there after people realized how nasty cache invalidation is when two cores are contending over one line. So you can have the same issue show up even in a single machine scenario. Good understanding of data access patterns and the right algorithm go a long way in both spaces as well.
  - notacoward 8y ago
    Even earlier, when SMP was hitting the server room but still far from the desktop, there was a similar phenomenon of breaking everything down to use ever finer-grain locks ... until the locking overhead (and errors) outweighed any benefit from parallelism. Over time, people learned to think about expected levels of parallelism, contention, etc. and "right size" their locks accordingly. Computing history's not a circle, but it's damn sure a spiral.
    - Something1234 8y ago
      > Computing history's not a circle, but it's damn sure a spiral. I think that is my new favorite phrase about computing history. Everything old is new again. There's way too much stuff we can extract from past for current problems. It's kind of amazing.
      - 8y ago
mpweiher 8y ago
"You can have a second computer once you've demonstrated you know how to use one".
makapuf 8y ago
in other words, "Too big for excel is not big data" https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
- digi_owl 8y ago
  Cloud, big data, "AI", i wonder what will be the next "me too, look at me" kind of buzzword for the corporate world...
  - cratermoon 8y ago
    "blockchain"
    - digi_owl 8y ago
      Ah yes, how did i forget...
- erikig 8y ago
  I keep coming back to that article everytime someone talks about big data :)
- mseebach 8y ago
  I heard a variation on this: it's not big data until it can't fit in RAM in a single rack.
  - imtringued 8y ago
    If you have two dozen nodes each with 6 TB of RAM + a few petabytes of HDD storage you're definitively going to need a big data solution.
    - anfilt 8y ago
      A C/C++ program can go through that amount data. You just have to use a simple divide and conquer strategy. It also depends on how the data is stored and the system architecture, but if you have any method that lets you break that data up into chunks or even just ranges. Then spin your program up for each chunk across each node or processing module that has quick access to the data your processing. Then take those results and have one or more threads merge results depending on the resulting data. I guess it also depends if this a one off job or continually if you would want to do this. Assuming the locality of data is not a big issue. It can be extremely fast. However, depending on the system architecture reading from the drives can be a bottle neck. If your system has enough drives for enough parallel reads you will turn through data pretty quickly. Moreover from my experience most systems or clusters with a few petabytes have enough drives that one can read quite a lot data in parallel. However, the worst is when the data is referencing non-local pieces of data. So your processing thread will have to fetch data from either a different node or data not in main memory to finish processing. This can be pain since that means either the task is just nor really parallelize-able or the person who originally generated the data did not take into account certain groups of data may be referenced with each other. Sadly, what happens here is that commutation or read costs for each thread start to dominate the cost of the computation. If it's a common task and your data is fairly static it makes sense to start duplicating data to speed up things. Also restructuring data can also be quite helpful and pay off in the long run.
deleted 8y ago
[deleted]
bufferoverflow 8y ago
Previous posts: 2016: https://news.ycombinator.com/item?id=12472905 https://news.ycombinator.com/item?id=12472905 2015: https://news.ycombinator.com/item?id=8908462 https://news.ycombinator.com/item?id=8908462 2018: https://news.ycombinator.com/item?id=16810756 https://news.ycombinator.com/item?id=16810756
- jwilk 8y ago
  The last one has zero comments, so there's no point linking to it.
  - albemuth 8y ago
    Unless you're trying to make a point about reposting.
    - slededit 8y ago
      3 times over 4 years isn't unreasonable for interesting content. With the exception of the unlucky post this has generated good discussion each time and I think is safely in the interesting category.
mpweiher 8y ago
For my performance book, I looked at some sample code for converting public transport data in CSV format to an embedded SQLite DB for use on mobile. A little bit of data optimization took the time from 22 minutes to under a second, or ~1000x, for well over 100MB of Source data. The target data went fro almost 200MB of SQLite to 7MB of binary that could just be mapped into memory. Oh, and lookup on the device also became 1000x faster. There is a LOT of that sort of stuff out there, our “standard” approaches are often highly inappropriate for a wide variety of problems.
- tzahola 8y ago
  Ha! I did the same for my BSc thesis: On-device route planning from GTFS data for Budapest’s public transport network iOS used to kill processes above 50 megs of RAM usage. Good times!
  - mpweiher 8y ago
    Cool! Link?
    - tzahola 8y ago
      Unfortunately it's no longer in the AppStore (ubiquitous cellular data made it obsolete) and the thesis itself is written in Hungarian :D
      - mpweiher 8y ago
        :-( Not so sure it's really obsolete, because connections are often spotty and local data can provide some initial information while waiting for the servers. Github?
        tzahola 8y ago
        It wasn't put on GitHub back then, and I don't want to publish it now. But I've sent it to you in email - check it out if you're interested.
walshemj 8y ago
Well yes in the edge case where you don't really have big data of course it will. Where MapReduce Hadoop etcera shine is when a single dataset one of many you need to process is bigger than the biggest single disk availible - this changes with time Back when I did M/R for BT the data set sizes where smaller - still having all of the Uk's larges single PR!ME superminis running your code was dam cool - even though I used a 110 baud dial up print terminal to control it.
adamdrake 8y ago
Hi all, author here! Many thanks for the feedback and comments. If you have any questions, I'm also happy to try to answer them. I'm also working on a project, https://applybyapi.com https://applybyapi.com which may be of interest to anyone here hiring developers and drowning in resumes.
- badlogic 8y ago
  I gave your service a quick whirl. Love the design language as a developer. What I don't like is that I have to provide my credit card information to get any detailed info (assuming there is such a thing behind the Stripe checkout). The info available to non-signed-up people is simply not enough. It would be great if you had a demo account so one can get a feel for how this works. E.g. I couldn't find info on why I have to add a job description. Wouldn't I post job board, with the posting liniing to the ApplyByAPI test? I could also not find any info on the tests candidates have to do. Yes, APIs, I get it. But an example wouldn't hurt. Interesting idea!
  - adamdrake 8y ago
    Hi, and thank you for the great feedback! We will certainly consider those points, and try to make things both more clear from a communication perspective and also consider adding some demo account (great idea!). To answer the question about the job description, you're right that most customers so far have a post on a jobs board which they use to get traffic over to their ApplyByAPI posting. Once the candidate is there, they get a page with the job description and some API information they can use to generate their application. We'll work on improving the language and information presentation to potential customers, and thank you again for taking the time to give your perspective!
erazor42 8y ago
I closed at "Since the data volume was only about 1.75GB". (And probably would have until 500GB+)
- imtringued 8y ago
  500GB is still laptop SSD territory... unless you meant 500TB but that's still doable with a single 4U Backblaze Storage Pod with 60 x 10TB HDDs.
  - dajonker 8y ago
    Just reading 500TB from 60 Hdds will take at least 12 hours, assuming a very optimistic constant 200MB/s throughput per HDD. If you need things to go faster you won't be able to do that without resorting to distributed computing.
Something1234 8y ago
cat *.pgn | grep "Result" | sort | uniq -c This pipeline has a useless use of cat. Over time I've found cat to be kind of slow as compared to actually passing a filename to a command when I can. If you rewrite it to be: grep -h "Result" *.pgn | ... It would be much faster. I found this when I was fiddling with my current log processor to analyze stats on my blog.
- adamdrake 8y ago
  Yes, it does. Spoiler alert: I remove it at the end. ;)
  - Something1234 8y ago
    Still I like the article. I just learned about useless use of cat the hard way. The biggest way I learned was cat /dev/zero | pv > /dev/null pv > /dev/null < /dev/zero The second one is much faster.
- tzahola 8y ago
  Argument list too long!
  - Something1234 8y ago
    Increase your stack size then. ulimit -s 65536 https://unix.stackexchange.com/a/45584/115135 https://unix.stackexchange.com/a/45584/115135
  - pwg 8y ago
    Then do: find -name \*.pgn -print0 | xargs -0 grep -h "Result" Each grep invocation will consume a maximum number of arguments, and xargs will invoke the minimum number of greps to process everything, with no "args too long" errors.
tzahola 8y ago
What is this “grep” people keep mentioning? I’ve tried installing it from npm to no avail :/
eb0la 8y ago
Un My case it was 2015. I was struggling with a 28GB CSV file that i needed to cut grabbing only 5 columns. Tried spark on my laptop: waste if time. After 4h I killed al processes because it didn't read 25% of the file yet. Same for hadoop, python and pandas, and a shiny new tool from google whose name I forgot long time ago. Finally I installed cygwin con My laptop and 20 minutes later 'cut' gave me the results file I needded.
- rsynnott 8y ago
  28GB is really too small for Hadoop to get out of bed for, in general. Though I would wonder why it was _that_ slow with Spark (or Hadoop, for that matter).
  - sfifs 8y ago
    Because it was running on a laptop instead of on a cluster :-)
  - mr_toad 8y ago
    Spark is going to try and ingest all the data, and it won’t fit in RAM. Wrong tool for the job basically.
    - rsynnott 8y ago
      Depends on exactly how you do it, I suppose, but it shouldn't necessarily. Most Hadoop-y work can also be accomplished in Spark without much fuss.
- jzwinck 8y ago
  cut solved your problem. Let's talk about why. cut is line oriented, like most Unix style filters. It needs to keep only one line in memory at most. If you say: pd.read_csv(f)[[x,y,z]] It has to read and parse the entire 28GB into memory (because it is not lazily evaluated; cf Julia). If you actually need to operate on three columns in memory and discard the rest, you should: pd.read_csv(f, usecols=[x,y,z]) Then you get exactly what you need, and avoid swapping. The lack of lazy evaluation does inhibit composition--just look at the myriad options in read_csv(), some of which are only there to enable eager evaluation to remain efficient.
jakosz 8y ago
You can get very good improvements over Spark too. I've been using GNU Parallel + redis + Cython workers to calculate distance pairs for a disambiguation problem. But then again, if it fits into a few X1 instances, it's not big data!
exelius 8y ago
I feel that we have spent 30 years replicating Bash across multiple computer systems. The further my clients move to the cloud, the more shell scripts they write at the exclusion of other languages. and just like this, I have clients who have ripped out expensive enterprise data streaming tools and replaced them with bash. The future of enterprise software is going to be a bloodbath.
- hoffbrau99 8y ago
  I want to believe, but instead it will be javascript-driven blockchain at the command line
  - konradb 8y ago
    how else would you control your ML IoT cluster?
ram_rar 8y ago
I think much of this issue can be attributed to 2 most underrated things 1. Cache line misses. 2. So called definition of BigData. (if data can be easily fit into memory, then its not Big period! ) Many times, I have seen simple awk / grep commands will outperform Hadoop jobs. I personally feel, its lot better to spin up larger instances, compute your jobs and shut it down than bearing the operational overhead of managing hadoop cluster.
nisa 8y ago
See also the wonderful COST paper: https://www.usenix.org/conference/hotos15/workshop-program/presentation/mcsherry https://www.usenix.org/conference/hotos15/workshop-program/p... But the article is kind of wrong. It depends on your data size and problem - you can even use commandline-tools with Hadoop Map/Reduce and the Streaming API and Hadoop is still useful if you have a few terabytes of data that you can tackle with map and reduce algorithms and in that case multiple machines do help quite a lot. anything that fits on your local ssd/hdd probably does not need hadoop... however you can run the same unix commands from the article just fine on a 20tb dataset with Hadoop. Hadoop MapReduce/HDFS is a tool for a specific purpose not magic fairy dust. Google did build it for indexing and storing the web and probably not to calculate some big excel sheets...
insaneirish 8y ago
Something, something, Joyent Manta: https://apidocs.joyent.com/manta/job-patterns.html https://apidocs.joyent.com/manta/job-patterns.html
jgord 8y ago
anecdota - We used fast csv tools and custom node.js scripts to wrangle import of ~500GB of geo polygon data into a large single vol postgresql+postGIS host. We generate svg maps in psuedo-realtime from this data-set : 2MB maps render sub-second over the web, which feels 'responsive'. I only mention this as many marketing people will call 50Million rows or 1TB "Big Data" and therefore suggest big / expensive / complex solutions. Recent SSD hosts have pushed up the "Big Data" watermark, and offer superb performance for many data applications. [ yes, I know you can't beat magnetic disks for storing large videos, but thats a less common use-case ]
- sitkack 8y ago
  How do you partition your data and/or build your indexes?
  - jgord 8y ago
    two kinds of indexes, for different purposes ... a) basically a preprocessed static inverted index on keywords [ using GIN / tsv_ etc ] b) geo location index on GIS geometry field - relying on postGIS to be performant over geo queries The data changes infrequently, so these are computed at time of import / regular update. We do have a fair amount of ram set aside for postgres [ circa 20GB ]
tluyben2 8y ago
I have rewritten incredibly overarchitected stuff, Cassandra, Hadoop, Kafka, Node, Mongo etc with a plethora of the ‘the latest cool programming languages’ running on big clusters of Amazon and Google to simple, but not sexy, c# and mysql or pgsql. Despite people commenting on the inefficiency of ORMs and unscalable nature of the solution I picked, it easily outperformes in every way for the real worldcases these systems were used for. Meaning; far easier achitecture, easier to read and far better peformance in both latency and throughput for workloads that will probably never happen. Also; one language, less engineers needed, less maintenance and easily swappable databases. I understand that all that other tech is in fact ‘learning new stuff’ for RDD, but it was costing these companies a lot of money with very little benefit. If I need something for very high traffic and huge data, I still do not know if I would opt for Cassandra or Hadoop; even with proper setup, sure they scale but at what cost? I had far better results with kdb+, which requires very little setup and very minimal overhead if you do it correctly. Then again, we will never have to mine petabytes, so maybe the use case works there: would love to hear from people who tried different solutions objectively.
- dajonker 8y ago
  kdb+ is very cool. Just a single binary, less than 1 MB and yet an extremely fast and powerful tool for analytics. There's a very nice talk on CMU's series of database videos if you want to know more. https://www.youtube.com/watch?v=AiGdfmxEP68 https://www.youtube.com/watch?v=AiGdfmxEP68 And it scores extremely well on the 1 billion taxi drives benchmarks (no cluster but some seriously big machine): http://tech.marksblogg.com/billion-nyc-taxi-kdb.html http://tech.marksblogg.com/billion-nyc-taxi-kdb.html
- moray 8y ago
  I have a very similar experience, in particular with my current project one of the users (a sysadmin actually) asked if I was using elasticsearch or similar because he noticed that searching a record and filtering was very fast. My response: nope, MySQL! (plus an ORM, a very hated programming language and very few config optimisations on the server side). This project DB has a couple of tables with thousands of records, not billions, and, for now, a few users (40-50).. a good schema and a few well done queries can do the trick. I guess some people are so used to see sluggish application that as soon as they see something that goes faster than average they think it must use some cool latest big tech.
xvilka 8y ago
Partially because of the bloated architecture of those Hadoop, Kafka, etc. And of course Java. Implementing modern and lighter alternative to those in C++, Go or Rust would a step forward.
- jfoutz 8y ago
  Wierd. I’d always thought of go as a slower Java with an shallower pool of good libraries and weak dev tools. Is the compiler and gc that good? Maybe I’ll have to give it another try.
  - virmundi 8y ago
    I’m not saying don’t look at other languages, but don’t do so given what was said. Java still runs better than Go. Hadoop, et al won’t necessarily run better as C++ or Rust. Distributed system performance is often dominated by setup and other negotiating between the components in the system. Java still has better, well tested in prod libs. Shiny is not better.
  - rufius 8y ago
    Anecdotally from my own use cases, I’ve found Go to be good for problems easily made parallel. This is especially true for things that commonly fall under “microservices.” On the other hand, for more intensive work, I’ve found it doesn’t perform as well as JVM-based applications. A lot of this is either JRuby or data models in Java doing interesting things with data structures. Finally, for sheer processing speed, we’ve started converting some of our components to Rust. This has a 3-4x speed advantage to Java and Go for similar implementations. Again - all anecdotes and I’m not really in a position to deeply explain how it all works (proprietary, etc). Just my two cents.
  - Groxx 8y ago
    Depends on what you're targeting. For raw computation, Go's similar and sometimes noticeably faster, even after hotspot has its turn. For garbage collection pause time, Go's rather amazing, typically measuring less than a millisecond even for gigabytes of heap mem[1]. For bin startup time (e.g. for a CLI tool) try Go instead just because it'll be done way before the JVM even hands control to your code. For dev tools, oh hell yes. Stick to Java. Go comes with some nice ones out-of-the-box which is always appreciated, but the ecosystem of stuff for Java is among the very best, and even the standard "basic" stuff vastly out-does Go's builtins. [1]: https://twitter.com/brianhatfield/status/634166123605331968?lang=en https://twitter.com/brianhatfield/status/634166123605331968?... and there's also a blog/video? about these same charts and how it got there. pretty neat transformations.
ikeboy 8y ago
Recently was sorting a 10 million line CSV by the second field which was numerical. After an hour went by and it wasn't done, I poked around online and saw a suggestion to put the field sorted on first. One awk command later my file was flipped. Run same exact sort command on this but without specifying field. Completed in 12 seconds. Morals: 1. Small changes can have a 3+ orders of magnitude effect on performance 2. Use the Google, easier than understanding every tool on a deep enough level to figure this out yourself ;)
- dorfsmay 8y ago
  csv files are extremely easy to import in postgres, and 10 M rows (assuming not very large) isn't much to compute even in a 6 or 7 year old laptop. Keep it in mind if you've got something slightly more complicated to analyse.
  - hobs 8y ago
    If SQL is your game but you dont want to get PG setup - try SQLite - wget "https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv" rows.csv sqlite3 .mode csv .import ./rows.csv newyorkdata SELECT * FROM newyorkdata ORDER BY `COUNT PARTICIPANTS`;
    - mxuribe 8y ago
      Nice! Talk about being a productive engineer! Kudos!
    - jillesvangurp 8y ago
      Or use csvkit ...
      - hobs 8y ago
        Oh nice, I hadn't seen this before, so a similar query would be a bit shorter! wget "https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv" csvsql --query "SELECT * FROM newyorkdata ORDER BY `COUNT PARTICIPANTS`" rows.csv > new.csv
0xcde4c3db 8y ago
See also: Scalability! But at what COST? [1] [2] > The COST of a given platform for a given problem is the hardware configuration required before the platform outperforms a competent single-threaded implementation. COST weighs a system’s scalability against the overheads introduced by the system, and indicates the actual performance gains of the system, without rewarding systems that bring substantial but parallelizable overheads. [1] http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf [2] https://news.ycombinator.com/item?id=11855594 https://news.ycombinator.com/item?id=11855594
- nisa 8y ago
  Keep in mind it's for graph processing - Hadoop/HDFS still shines for data-intensive streaming workloads like indexing a few hundred terabytes of data - where you can exploit the parallel disk io of all disks in the cluster - if you have 20 machines with 8 disks in a cluster that's 20 * 8 * 100mbyte/s = 16gbyte/s throughput - for 200 machines it's 160gbyte/s. However for iterative calculations like pagerank the overhead for distributing the problems is often not worth it.
jandrese 8y ago
Most people who think they have big data don't.
- paulmd 8y ago
  But we have millions of rows! -- my actual employer (the actual problem is our SAN is getting us a grand total of 10 iops. No, that's not wrong, 10. The "what the christ, just buy us a system that actually works" momentum has been building for 2 years now but hasn't managed to take over. Hardware comes out of a different budget than wasted time, yo.)
  - mirceal 8y ago
    So you have big data. Now you just need the big hardware /s
  - candiodari 8y ago
    A cynical person might point out that for empire-building managers the goals are : 1) minimize hardware spend at all costs 2) maximize lost time (if spent under the manager's control)
- cortesoft 8y ago
  I don’t know why these ‘you don’t have big data’ type articles bother me so much, but they really do. I know it isn’t saying NO ONE has big data, but I feel defensive anyway. Some of us DO work for companies that need to process millions of log lines a second. I know the article is not for me, but it still feels like a dismissal of our real, actual, big data problem.
  - realusername 8y ago
    These articles are not targeted at you indeed but to the thousands of companies trying to setup a massive architecture to process 3GB of data per year. All these big data solutions are still necessary of course, it's just not for everyone.
dorfsmay 8y ago
If you need to do something more complicated where SQL would be really handy, and like here you're not going to update your data, give monetDB a try. Quite often when I'm about to break down and implore the hadoop gods, I remember about monetDB, and most of the time it's sufficient to solve my problem. MonrtDB: columnar DB for a single machine with a psql like interface: https://www.monetdb.org/ https://www.monetdb.org/
mirceal 8y ago
People are suckers for complex, over-engineered things. They associate mastering complexity with intelligence. Simple things that just work are boring / not sexy. I’ll be in my corner doing asking “do we really need this?”/“have you tested this?”
blablabla123 8y ago
I've heard this so often, especially from my boss - whoever this is. I come to the conclusion this is just a lame excuse because the people feel out of control and overwhelmed when using this kind of software. It is heavily Java based and to those who never touched javac, the Stack traces must look both intimidating and ridiculous. On the other hand, processing data over several steps with a homegrown solution needs a lot of programming discipline and reasoning, otherwise your software turns into an unmaintainable and unreliable mess. In fact this is the case where I work right now...
hans_castorp 8y ago
So there are 2197188 games in that file. Extracting only the lines with "[Result]" in them into new file (using grep) takes about 3 seconds. Importing that into a local Postgres database on my laptop takes about 1.5 seconds. Then running a simple: select result, count(*) from results group by result; Takes about 0.5 seconds. So the total process took only 5 seconds (and now I can run much more aggregation queries on that)
angel_j 8y ago
the hadoop solution may have taken more computer time, but that is far less valuable than a person's the command-line solution probably took that person a couple or more hours to perfect
raghava 8y ago
For many, the incentive from the gig is a) resume enrichment by way of buzzword addition b) huge budget grants and allocation, purportedly for lofty goals while management is really unaware of real technology needs/options Much has been talked about this already; sharing them again! [1] https://news.ycombinator.com/item?id=14401399 https://news.ycombinator.com/item?id=14401399 [2] https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html [3] http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra... [4] https://www.reddit.com/r/programming/comments/3z9rab/taco_bell_programming/ https://www.reddit.com/r/programming/comments/3z9rab/taco_be... [5] https://www.mikecr.it/ramblings/taco-bell-programming/ https://www.mikecr.it/ramblings/taco-bell-programming/
forinti 8y ago
This reminds me of the early EJBs which were really complicated and most people who used them didn't really need them.
verytrivial 8y ago
Anchoring the search would allow the regex to terminate faster for non-matching lines: ... | mawk '/^\[Result' ...
notacoward 8y ago
A lot of people are saying how they've worked on single-machine systems that performed far better than distributed alternatives. Yawn. So have I. So have thousands of others. It should almost be a prerequisite for working on those distributed systems, so that they can understand the real point of those systems. Sometimes it's about performance, and even then there's no "one size fits all" answer. Just as often it's about capacity. Seen any exabyte single machines on the market lately? Even more often that that, it's about redundancy and reliability. What happens when your single-machine wonder has a single hardware failure? Sure, a lot of tyros are working on distributed systems because it's cool or because it enhances their resumes, but there are also a lot of professionals working on distributed systems because they're the only way to meet requirements. Cherry-picking examples to favor your own limited skill set doesn't seem like engineering to me.
koomi 8y ago
It can be colder at night than outside.
icc97 8y ago
Relevant article from 2013: 'Don't use Hadoop - your data isn't that big' [0] and the most recent HN discussion [1]. [0]: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html [1]: https://news.ycombinator.com/item?id=14401399 https://news.ycombinator.com/item?id=14401399
libx 8y ago
Once, I saw a presentation of Unicage, a big data solution that had (perhaps still has) a free version https://youtu.be/h_C5GBblkH8?t=2566 https://youtu.be/h_C5GBblkH8?t=2566 It seems that it has evolved to a company now: http://www.unicage.com/products.html http://www.unicage.com/products.html Did anyone try the unicage solution?
dunk010 8y ago
'Nuff said: https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce
hajderr 8y ago
Ah yes this is so true!
rurban 8y ago
Esp. if you use multipipe-enhanced coreutils, like dgsh. https://www2.dmst.aueb.gr/dds/sw/dgsh/ https://www2.dmst.aueb.gr/dds/sw/dgsh/

438 points by hd4 8y ago

If I do one thing this year, I need to learn more about these tried and tested command-line unix shell utilities. It's becoming increasingly obvious that so many complicated and overwrought things are being unleashed on the world because (statistically) nobody knows this stuff anymore.

trumped 8y ago
They try to make the cloud relevant for a lot of things that could easily be accomplished locally... Most of the time, they do that just to get your data.
- SteveNuts 8y ago
  Vendor lock-in is what they're after in the cloud case. Every time we talk to our cloud provider reps they ask us which of their SaaS offerings we're using. Their heads spin when we tell them we're just using VMs and ask us why.
beokop 8y ago
So much internal tooling at a company can be avoided if you can just get people comfortable with basic *nix programs. No, we don’t need a web interface to show that data. No, we don’t need to code a dropdown to sort it differently.
- fleitz 8y ago
  Yes, you do need a web interface for that, thankfully we have cgi-bin from the list of unix tools. Turns out browsers can render text :)
- digi_owl 8y ago
  And yet the UX people, particularly around Linux, wants to bury the _nix cli so damned deep...
  - AnIdiotOnTheNet 8y ago
    Can't say I blame them. The base concept is sound but the implementation is archaic and obtuse as hell.
  - tormeh 8y ago
    CLIs have a discoverability problem. Knowing how to use any given one is a skill that takes years to hone. Learning a single command can take up to an hour. Most webapps are more constrained, but a common operation can usually be accomplished by someone who hasn't seen it before in less than 3 minutes.
    - jonhendry18 8y ago
      The 'apropos' command is very handy on machine or OS you aren't familiar with. If you want to find out what tools are available on the CLI that operate on jpegs, you'd try 'apropos jpeg' and get a list of things that mention jpeg in their man page.
deckard1 8y ago
This applies to much of the software world. Many of the places I've worked have a bad habit of ditching perfectly fine tech in order to chase the latest thing. Part of it is driven by RDD (resume driven development) and positioning the company so they can hire the latest crop of fad chasers. You end up with a bunch of specialists that are siloed into their particular tech. You got the Docker guy. You got the Redis guy. You may even have one of those gray-haired DBAs that still talk about this "Oracle" thing all day long, and no one knows what the hell he's going on about. You got the Ruby guys, but now you guy a bunch of React or Angular guys. What to do with the Ruby guys? Can't just fire them yet. You have a dozen of 'em, and it took you years to hire them. Fine, just banish them to the backend somewhere. Give them some fidget spinners and a few pointless projects to keep them busy.
- sixdimensional 8y ago
  This is exactly right. This is real life!! And it hurts.
- tomc1985 8y ago
  Hey now, Ruby ain't dead. I am one of these "Ruby" guys and, in the 48-or-so hours since my profile was activated at a new recruiting shop this week, I've gotten several potential candidate employers (some in seed stage) who want to get in touch. The real issue is why do we silo ourselves so? It's fucking stupid. I do so much more than Ruby but conveying that to the new breed of tech people is just impossible. All they see is your biggest resume badge, and all they care about is how wide you will spread your legs for them.
- monksy 8y ago
  Let's not forget when you're interviewing: The new places expect the latest technologies. Don't have time to learn spark, flink, kafka streams, and repeat scala spec whilst having intimate knowledge of all of the REST libraries? BAD DEVELOPER.
eb0la 8y ago
You just need to master a few commands: Find (take a look at -exec option) Cut (or awk) Sed (for simple text/string substitutions) Xargs Dd (that beast can do charset transtation from ASCII to EBCDIC to use in mainframes and can also wipe disks) And bash of course :)
- sidlls 8y ago
  awk is so beautiful. A good awk one-liner can beat fancy pandas all day long in some cases.
  - flukus 8y ago
    I hate how awk gets so much praise for one liners when it's so good at building things like state machines too: https://two-wrongs.com/awk-state-machine-parser-pattern.html https://two-wrongs.com/awk-state-machine-parser-pattern.html I've got some similar ones for work that take data csv like data and output sql. A little bit of vim-foo on the csv (yy10000p) and I've got all the test data I need.
    - cosmie 8y ago
      If anything, awk should get some praise for it's flexibility to go from one liners to Doom-style 3d games[1]! [1] https://github.com/TheMozg/awk-raycaster https://github.com/TheMozg/awk-raycaster
  - cwilkes 8y ago
    An example here would be nice.
- hoffbrau99 8y ago
  Don't forget grep!
- eanzenberg 8y ago
  "master a few commands" "awk" brb 15 years
thediehl 8y ago
I started with Unix Shell Programming http://a.co/au9OUrB http://a.co/au9OUrB
nemild 8y ago
I made a list of a good set of commands for a full stack engineer: https://github.com/nemild/cli_for_full_stack https://github.com/nemild/cli_for_full_stack
- sh87 8y ago
  This is awesome! thanks.
sitkack 8y ago
You can write many of these yourself. Just reading records, and sending them across pipes in ways you can compose is a good start. I find PyPy is excellent for this kind of work, fast to write, if the pipeline runs for more than 10 seconds, it has plenty of time for the JIT to warm up and amortize the cost. If you don't need fancy libs, LuaJIT is amazing and great to doing transforms in a pipeline. I rewrote a Hadoop job to use Streaming (pipes, basic unix model) from a Java job to native. Just that switch was 10x faster. Mostly because of Hadoop overheads.
cosmie 8y ago
Companies aren't making these complicated and overwrought things because nobody knows this stuff anymore. The ones actually doing the making generally know all the tried and tested ways to do what they want, because they used them to scale. Then they eventually outgrow the existing solutions and created these complicated systems to account for whatever specific technical or business issues that necessitated hopping from existing solutions to creating a solution. The problem tends to come along with the visibility that comes when one of these get released publicly. People use these things without doing the same level of due diligence to see if something less complicated fits their needs. Either deliberately (resume driven development) or simply because it doesn't occur to them to do so and "if it's good enough for this unicorn it's good enough for me". Or in some cases they do due diligence, but incorrectly evaluate their needs and rule out simpler solutions. But the most common reason I've ran into is simply that it's more fun to play with these projects than it is to use crufty old stuff, no matter how tried and tested and potentially appropriate it is. At a previous company, I had a python script running on a cron job every 5 minutes to do some data processing needs. Once or twice a month, the batch would be so large that it took 6-7 minutes to complete and the cron job would trigger the script again before it finished, causing the second instance of the script to see the lock file, log an error, and exit. It didn't cause any problem for these periodically skipped runs, because the business need only required data to be processed within 24 hours of coming in. The 5 minute cron job was just to even out resource usage throughout the day instead of doing a larger nightly batch job. A piece of data not getting processed for 8 minutes instead of 5 did not make any material impact. Another team had noticed the errors popping up in the log and were in the process of testing out a whole bunch of real time data pipelines like Kafka, leveraging my error messages to justify the need for a "real time" system without ever even asking me about the errors. After I found out other people were noticing those superfluous errors, I moved the cron job to a 30 minute window to stop them from happening anymore. Turns out they didn't have any other justification for their data pipeline greenfield project and weren't happy to go back to their normal day to day work. I offered to let them maintain and expand my python scripts if they were interested in data processing work, but for some reason they never took me up on that offer. :(
- gaius 8y ago
  Companies aren't making these complicated and overwrought things because nobody knows this stuff anymore. The ones actually doing the making generally know all the tried and tested ways to do what they want, because they used them to scale Right, Yahoo made Hadoop because (probably) they needed it. 99% of companies... just don’t.
shawndumas 8y ago
Read The AWK Programming Language by Alfred Aho, Peter Weinberger, and Brian Kernighan "This is an absolute classic which although ancient in Computing years is an absolute gem full of relevance yet."
asicsp 8y ago
I have a repo dedicated to cli text-processing.. short-descriptions and tons of examples.. I had fun and learned a lot doing this link: https://github.com/learnbyexample/Command-line-text-processing https://github.com/learnbyexample/Command-line-text-processi...
- svde 8y ago
  This looks exactly like what I was looking for. Thank you.
  - asicsp 8y ago
    you're welcome, happy learning :)

ash_gti 8y ago

Shouldn't this be marked 2014? The articles date is January 18, 2014.

gaius 8y ago
It’s probably even truer and more relevant now

noobermin 8y ago

It would benefit people to actually understand algorithmic complexity of what they are doing before they go on these voyages to parallelize everything. It also helps to know what helps to parallelize and what doesn't.

3pt14159 8y ago

I once converted a simulation into cython from plain old python. Because it fit in the CPU cache the speedup was around 10000x on a single machine (numerical simulations, amirite?). Because it was so much faster all the code required to split it up between a bunch of servers in a map reduce job could be deleted, since it only needed a couple cores on a single machine for a ms or three. Because it wasn't a map-reduce job, I could take it out of the worker queue and just handle it on the fly during the web request. Sometimes it's worth it to just step back and experiment a bit.

vvanders 8y ago
Yeah, back when I was in gamedev land and multi-cores started coming on the scene it was "Multithread ALL THE THINGS". Shortly there after people realized how nasty cache invalidation is when two cores are contending over one line. So you can have the same issue show up even in a single machine scenario. Good understanding of data access patterns and the right algorithm go a long way in both spaces as well.
- notacoward 8y ago
  Even earlier, when SMP was hitting the server room but still far from the desktop, there was a similar phenomenon of breaking everything down to use ever finer-grain locks ... until the locking overhead (and errors) outweighed any benefit from parallelism. Over time, people learned to think about expected levels of parallelism, contention, etc. and "right size" their locks accordingly. Computing history's not a circle, but it's damn sure a spiral.
  - Something1234 8y ago
    > Computing history's not a circle, but it's damn sure a spiral. I think that is my new favorite phrase about computing history. Everything old is new again. There's way too much stuff we can extract from past for current problems. It's kind of amazing.
    - 8y ago

mpweiher 8y ago

"You can have a second computer once you've demonstrated you know how to use one".

makapuf 8y ago

in other words, "Too big for excel is not big data" https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

digi_owl 8y ago
Cloud, big data, "AI", i wonder what will be the next "me too, look at me" kind of buzzword for the corporate world...
- cratermoon 8y ago
  "blockchain"
  - digi_owl 8y ago
    Ah yes, how did i forget...
erikig 8y ago
I keep coming back to that article everytime someone talks about big data :)
mseebach 8y ago
I heard a variation on this: it's not big data until it can't fit in RAM in a single rack.
- imtringued 8y ago
  If you have two dozen nodes each with 6 TB of RAM + a few petabytes of HDD storage you're definitively going to need a big data solution.
  - anfilt 8y ago
    A C/C++ program can go through that amount data. You just have to use a simple divide and conquer strategy. It also depends on how the data is stored and the system architecture, but if you have any method that lets you break that data up into chunks or even just ranges. Then spin your program up for each chunk across each node or processing module that has quick access to the data your processing. Then take those results and have one or more threads merge results depending on the resulting data. I guess it also depends if this a one off job or continually if you would want to do this. Assuming the locality of data is not a big issue. It can be extremely fast. However, depending on the system architecture reading from the drives can be a bottle neck. If your system has enough drives for enough parallel reads you will turn through data pretty quickly. Moreover from my experience most systems or clusters with a few petabytes have enough drives that one can read quite a lot data in parallel. However, the worst is when the data is referencing non-local pieces of data. So your processing thread will have to fetch data from either a different node or data not in main memory to finish processing. This can be pain since that means either the task is just nor really parallelize-able or the person who originally generated the data did not take into account certain groups of data may be referenced with each other. Sadly, what happens here is that commutation or read costs for each thread start to dominate the cost of the computation. If it's a common task and your data is fairly static it makes sense to start duplicating data to speed up things. Also restructuring data can also be quite helpful and pay off in the long run.

deleted 8y ago

[deleted]

bufferoverflow 8y ago

Previous posts: 2016: https://news.ycombinator.com/item?id=12472905 https://news.ycombinator.com/item?id=12472905 2015: https://news.ycombinator.com/item?id=8908462 https://news.ycombinator.com/item?id=8908462 2018: https://news.ycombinator.com/item?id=16810756 https://news.ycombinator.com/item?id=16810756

jwilk 8y ago
The last one has zero comments, so there's no point linking to it.
- albemuth 8y ago
  Unless you're trying to make a point about reposting.
  - slededit 8y ago
    3 times over 4 years isn't unreasonable for interesting content. With the exception of the unlucky post this has generated good discussion each time and I think is safely in the interesting category.

mpweiher 8y ago

For my performance book, I looked at some sample code for converting public transport data in CSV format to an embedded SQLite DB for use on mobile. A little bit of data optimization took the time from 22 minutes to under a second, or ~1000x, for well over 100MB of Source data. The target data went fro almost 200MB of SQLite to 7MB of binary that could just be mapped into memory. Oh, and lookup on the device also became 1000x faster. There is a LOT of that sort of stuff out there, our “standard” approaches are often highly inappropriate for a wide variety of problems.

tzahola 8y ago
Ha! I did the same for my BSc thesis: On-device route planning from GTFS data for Budapest’s public transport network iOS used to kill processes above 50 megs of RAM usage. Good times!
- mpweiher 8y ago
  Cool! Link?
  - tzahola 8y ago
    Unfortunately it's no longer in the AppStore (ubiquitous cellular data made it obsolete) and the thesis itself is written in Hungarian :D
    - mpweiher 8y ago
      :-( Not so sure it's really obsolete, because connections are often spotty and local data can provide some initial information while waiting for the servers. Github?
      - tzahola 8y ago
        It wasn't put on GitHub back then, and I don't want to publish it now. But I've sent it to you in email - check it out if you're interested.

walshemj 8y ago

Well yes in the edge case where you don't really have big data of course it will. Where MapReduce Hadoop etcera shine is when a single dataset one of many you need to process is bigger than the biggest single disk availible - this changes with time Back when I did M/R for BT the data set sizes where smaller - still having all of the Uk's larges single PR!ME superminis running your code was dam cool - even though I used a 110 baud dial up print terminal to control it.

adamdrake 8y ago

Hi all, author here! Many thanks for the feedback and comments. If you have any questions, I'm also happy to try to answer them. I'm also working on a project, https://applybyapi.com https://applybyapi.com which may be of interest to anyone here hiring developers and drowning in resumes.

badlogic 8y ago
I gave your service a quick whirl. Love the design language as a developer. What I don't like is that I have to provide my credit card information to get any detailed info (assuming there is such a thing behind the Stripe checkout). The info available to non-signed-up people is simply not enough. It would be great if you had a demo account so one can get a feel for how this works. E.g. I couldn't find info on why I have to add a job description. Wouldn't I post job board, with the posting liniing to the ApplyByAPI test? I could also not find any info on the tests candidates have to do. Yes, APIs, I get it. But an example wouldn't hurt. Interesting idea!
- adamdrake 8y ago
  Hi, and thank you for the great feedback! We will certainly consider those points, and try to make things both more clear from a communication perspective and also consider adding some demo account (great idea!). To answer the question about the job description, you're right that most customers so far have a post on a jobs board which they use to get traffic over to their ApplyByAPI posting. Once the candidate is there, they get a page with the job description and some API information they can use to generate their application. We'll work on improving the language and information presentation to potential customers, and thank you again for taking the time to give your perspective!

erazor42 8y ago

I closed at "Since the data volume was only about 1.75GB". (And probably would have until 500GB+)

imtringued 8y ago
500GB is still laptop SSD territory... unless you meant 500TB but that's still doable with a single 4U Backblaze Storage Pod with 60 x 10TB HDDs.
- dajonker 8y ago
  Just reading 500TB from 60 Hdds will take at least 12 hours, assuming a very optimistic constant 200MB/s throughput per HDD. If you need things to go faster you won't be able to do that without resorting to distributed computing.

Something1234 8y ago

cat *.pgn | grep "Result" | sort | uniq -c This pipeline has a useless use of cat. Over time I've found cat to be kind of slow as compared to actually passing a filename to a command when I can. If you rewrite it to be: grep -h "Result" *.pgn | ... It would be much faster. I found this when I was fiddling with my current log processor to analyze stats on my blog.

adamdrake 8y ago
Yes, it does. Spoiler alert: I remove it at the end. ;)
- Something1234 8y ago
  Still I like the article. I just learned about useless use of cat the hard way. The biggest way I learned was cat /dev/zero | pv > /dev/null pv > /dev/null < /dev/zero The second one is much faster.
tzahola 8y ago
Argument list too long!
- Something1234 8y ago
  Increase your stack size then. ulimit -s 65536 https://unix.stackexchange.com/a/45584/115135 https://unix.stackexchange.com/a/45584/115135
- pwg 8y ago
  Then do: find -name \*.pgn -print0 | xargs -0 grep -h "Result" Each grep invocation will consume a maximum number of arguments, and xargs will invoke the minimum number of greps to process everything, with no "args too long" errors.

tzahola 8y ago

What is this “grep” people keep mentioning? I’ve tried installing it from npm to no avail :/

eb0la 8y ago

Un My case it was 2015. I was struggling with a 28GB CSV file that i needed to cut grabbing only 5 columns. Tried spark on my laptop: waste if time. After 4h I killed al processes because it didn't read 25% of the file yet. Same for hadoop, python and pandas, and a shiny new tool from google whose name I forgot long time ago. Finally I installed cygwin con My laptop and 20 minutes later 'cut' gave me the results file I needded.

rsynnott 8y ago
28GB is really too small for Hadoop to get out of bed for, in general. Though I would wonder why it was _that_ slow with Spark (or Hadoop, for that matter).
- sfifs 8y ago
  Because it was running on a laptop instead of on a cluster :-)
- mr_toad 8y ago
  Spark is going to try and ingest all the data, and it won’t fit in RAM. Wrong tool for the job basically.
  - rsynnott 8y ago
    Depends on exactly how you do it, I suppose, but it shouldn't necessarily. Most Hadoop-y work can also be accomplished in Spark without much fuss.
jzwinck 8y ago
cut solved your problem. Let's talk about why. cut is line oriented, like most Unix style filters. It needs to keep only one line in memory at most. If you say: pd.read_csv(f)[[x,y,z]] It has to read and parse the entire 28GB into memory (because it is not lazily evaluated; cf Julia). If you actually need to operate on three columns in memory and discard the rest, you should: pd.read_csv(f, usecols=[x,y,z]) Then you get exactly what you need, and avoid swapping. The lack of lazy evaluation does inhibit composition--just look at the myriad options in read_csv(), some of which are only there to enable eager evaluation to remain efficient.

jakosz 8y ago

You can get very good improvements over Spark too. I've been using GNU Parallel + redis + Cython workers to calculate distance pairs for a disambiguation problem. But then again, if it fits into a few X1 instances, it's not big data!

exelius 8y ago

I feel that we have spent 30 years replicating Bash across multiple computer systems. The further my clients move to the cloud, the more shell scripts they write at the exclusion of other languages. and just like this, I have clients who have ripped out expensive enterprise data streaming tools and replaced them with bash. The future of enterprise software is going to be a bloodbath.

hoffbrau99 8y ago
I want to believe, but instead it will be javascript-driven blockchain at the command line
- konradb 8y ago
  how else would you control your ML IoT cluster?

ram_rar 8y ago

I think much of this issue can be attributed to 2 most underrated things 1. Cache line misses. 2. So called definition of BigData. (if data can be easily fit into memory, then its not Big period! ) Many times, I have seen simple awk / grep commands will outperform Hadoop jobs. I personally feel, its lot better to spin up larger instances, compute your jobs and shut it down than bearing the operational overhead of managing hadoop cluster.

nisa 8y ago

See also the wonderful COST paper: https://www.usenix.org/conference/hotos15/workshop-program/presentation/mcsherry https://www.usenix.org/conference/hotos15/workshop-program/p... But the article is kind of wrong. It depends on your data size and problem - you can even use commandline-tools with Hadoop Map/Reduce and the Streaming API and Hadoop is still useful if you have a few terabytes of data that you can tackle with map and reduce algorithms and in that case multiple machines do help quite a lot. anything that fits on your local ssd/hdd probably does not need hadoop... however you can run the same unix commands from the article just fine on a 20tb dataset with Hadoop. Hadoop MapReduce/HDFS is a tool for a specific purpose not magic fairy dust. Google did build it for indexing and storing the web and probably not to calculate some big excel sheets...

insaneirish 8y ago

Something, something, Joyent Manta: https://apidocs.joyent.com/manta/job-patterns.html https://apidocs.joyent.com/manta/job-patterns.html

jgord 8y ago

anecdota - We used fast csv tools and custom node.js scripts to wrangle import of ~500GB of geo polygon data into a large single vol postgresql+postGIS host. We generate svg maps in psuedo-realtime from this data-set : 2MB maps render sub-second over the web, which feels 'responsive'. I only mention this as many marketing people will call 50Million rows or 1TB "Big Data" and therefore suggest big / expensive / complex solutions. Recent SSD hosts have pushed up the "Big Data" watermark, and offer superb performance for many data applications. [ yes, I know you can't beat magnetic disks for storing large videos, but thats a less common use-case ]

sitkack 8y ago
How do you partition your data and/or build your indexes?
- jgord 8y ago
  two kinds of indexes, for different purposes ... a) basically a preprocessed static inverted index on keywords [ using GIN / tsv_ etc ] b) geo location index on GIS geometry field - relying on postGIS to be performant over geo queries The data changes infrequently, so these are computed at time of import / regular update. We do have a fair amount of ram set aside for postgres [ circa 20GB ]

tluyben2 8y ago

I have rewritten incredibly overarchitected stuff, Cassandra, Hadoop, Kafka, Node, Mongo etc with a plethora of the ‘the latest cool programming languages’ running on big clusters of Amazon and Google to simple, but not sexy, c# and mysql or pgsql. Despite people commenting on the inefficiency of ORMs and unscalable nature of the solution I picked, it easily outperformes in every way for the real worldcases these systems were used for. Meaning; far easier achitecture, easier to read and far better peformance in both latency and throughput for workloads that will probably never happen. Also; one language, less engineers needed, less maintenance and easily swappable databases. I understand that all that other tech is in fact ‘learning new stuff’ for RDD, but it was costing these companies a lot of money with very little benefit. If I need something for very high traffic and huge data, I still do not know if I would opt for Cassandra or Hadoop; even with proper setup, sure they scale but at what cost? I had far better results with kdb+, which requires very little setup and very minimal overhead if you do it correctly. Then again, we will never have to mine petabytes, so maybe the use case works there: would love to hear from people who tried different solutions objectively.

dajonker 8y ago
kdb+ is very cool. Just a single binary, less than 1 MB and yet an extremely fast and powerful tool for analytics. There's a very nice talk on CMU's series of database videos if you want to know more. https://www.youtube.com/watch?v=AiGdfmxEP68 https://www.youtube.com/watch?v=AiGdfmxEP68 And it scores extremely well on the 1 billion taxi drives benchmarks (no cluster but some seriously big machine): http://tech.marksblogg.com/billion-nyc-taxi-kdb.html http://tech.marksblogg.com/billion-nyc-taxi-kdb.html
moray 8y ago
I have a very similar experience, in particular with my current project one of the users (a sysadmin actually) asked if I was using elasticsearch or similar because he noticed that searching a record and filtering was very fast. My response: nope, MySQL! (plus an ORM, a very hated programming language and very few config optimisations on the server side). This project DB has a couple of tables with thousands of records, not billions, and, for now, a few users (40-50).. a good schema and a few well done queries can do the trick. I guess some people are so used to see sluggish application that as soon as they see something that goes faster than average they think it must use some cool latest big tech.

xvilka 8y ago

Partially because of the bloated architecture of those Hadoop, Kafka, etc. And of course Java. Implementing modern and lighter alternative to those in C++, Go or Rust would a step forward.

jfoutz 8y ago
Wierd. I’d always thought of go as a slower Java with an shallower pool of good libraries and weak dev tools. Is the compiler and gc that good? Maybe I’ll have to give it another try.
- virmundi 8y ago
  I’m not saying don’t look at other languages, but don’t do so given what was said. Java still runs better than Go. Hadoop, et al won’t necessarily run better as C++ or Rust. Distributed system performance is often dominated by setup and other negotiating between the components in the system. Java still has better, well tested in prod libs. Shiny is not better.
- rufius 8y ago
  Anecdotally from my own use cases, I’ve found Go to be good for problems easily made parallel. This is especially true for things that commonly fall under “microservices.” On the other hand, for more intensive work, I’ve found it doesn’t perform as well as JVM-based applications. A lot of this is either JRuby or data models in Java doing interesting things with data structures. Finally, for sheer processing speed, we’ve started converting some of our components to Rust. This has a 3-4x speed advantage to Java and Go for similar implementations. Again - all anecdotes and I’m not really in a position to deeply explain how it all works (proprietary, etc). Just my two cents.
- Groxx 8y ago
  Depends on what you're targeting. For raw computation, Go's similar and sometimes noticeably faster, even after hotspot has its turn. For garbage collection pause time, Go's rather amazing, typically measuring less than a millisecond even for gigabytes of heap mem[1]. For bin startup time (e.g. for a CLI tool) try Go instead just because it'll be done way before the JVM even hands control to your code. For dev tools, oh hell yes. Stick to Java. Go comes with some nice ones out-of-the-box which is always appreciated, but the ecosystem of stuff for Java is among the very best, and even the standard "basic" stuff vastly out-does Go's builtins. [1]: https://twitter.com/brianhatfield/status/634166123605331968?lang=en https://twitter.com/brianhatfield/status/634166123605331968?... and there's also a blog/video? about these same charts and how it got there. pretty neat transformations.

ikeboy 8y ago

Recently was sorting a 10 million line CSV by the second field which was numerical. After an hour went by and it wasn't done, I poked around online and saw a suggestion to put the field sorted on first. One awk command later my file was flipped. Run same exact sort command on this but without specifying field. Completed in 12 seconds. Morals: 1. Small changes can have a 3+ orders of magnitude effect on performance 2. Use the Google, easier than understanding every tool on a deep enough level to figure this out yourself ;)

dorfsmay 8y ago
csv files are extremely easy to import in postgres, and 10 M rows (assuming not very large) isn't much to compute even in a 6 or 7 year old laptop. Keep it in mind if you've got something slightly more complicated to analyse.
- hobs 8y ago
  If SQL is your game but you dont want to get PG setup - try SQLite - wget "https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv" rows.csv sqlite3 .mode csv .import ./rows.csv newyorkdata SELECT * FROM newyorkdata ORDER BY `COUNT PARTICIPANTS`;
  - mxuribe 8y ago
    Nice! Talk about being a productive engineer! Kudos!
  - jillesvangurp 8y ago
    Or use csvkit ...
    - hobs 8y ago
      Oh nice, I hadn't seen this before, so a similar query would be a bit shorter! wget "https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv" csvsql --query "SELECT * FROM newyorkdata ORDER BY `COUNT PARTICIPANTS`" rows.csv > new.csv

0xcde4c3db 8y ago

See also: Scalability! But at what COST? [1] [2] > The COST of a given platform for a given problem is the hardware configuration required before the platform outperforms a competent single-threaded implementation. COST weighs a system’s scalability against the overheads introduced by the system, and indicates the actual performance gains of the system, without rewarding systems that bring substantial but parallelizable overheads. [1] http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf [2] https://news.ycombinator.com/item?id=11855594 https://news.ycombinator.com/item?id=11855594

nisa 8y ago
Keep in mind it's for graph processing - Hadoop/HDFS still shines for data-intensive streaming workloads like indexing a few hundred terabytes of data - where you can exploit the parallel disk io of all disks in the cluster - if you have 20 machines with 8 disks in a cluster that's 20 * 8 * 100mbyte/s = 16gbyte/s throughput - for 200 machines it's 160gbyte/s. However for iterative calculations like pagerank the overhead for distributing the problems is often not worth it.

jandrese 8y ago

Most people who think they have big data don't.

paulmd 8y ago
But we have millions of rows! -- my actual employer (the actual problem is our SAN is getting us a grand total of 10 iops. No, that's not wrong, 10. The "what the christ, just buy us a system that actually works" momentum has been building for 2 years now but hasn't managed to take over. Hardware comes out of a different budget than wasted time, yo.)
- mirceal 8y ago
  So you have big data. Now you just need the big hardware /s
- candiodari 8y ago
  A cynical person might point out that for empire-building managers the goals are : 1) minimize hardware spend at all costs 2) maximize lost time (if spent under the manager's control)
cortesoft 8y ago
I don’t know why these ‘you don’t have big data’ type articles bother me so much, but they really do. I know it isn’t saying NO ONE has big data, but I feel defensive anyway. Some of us DO work for companies that need to process millions of log lines a second. I know the article is not for me, but it still feels like a dismissal of our real, actual, big data problem.
- realusername 8y ago
  These articles are not targeted at you indeed but to the thousands of companies trying to setup a massive architecture to process 3GB of data per year. All these big data solutions are still necessary of course, it's just not for everyone.

dorfsmay 8y ago

If you need to do something more complicated where SQL would be really handy, and like here you're not going to update your data, give monetDB a try. Quite often when I'm about to break down and implore the hadoop gods, I remember about monetDB, and most of the time it's sufficient to solve my problem. MonrtDB: columnar DB for a single machine with a psql like interface: https://www.monetdb.org/ https://www.monetdb.org/

mirceal 8y ago

People are suckers for complex, over-engineered things. They associate mastering complexity with intelligence. Simple things that just work are boring / not sexy. I’ll be in my corner doing asking “do we really need this?”/“have you tested this?”

blablabla123 8y ago

I've heard this so often, especially from my boss - whoever this is. I come to the conclusion this is just a lame excuse because the people feel out of control and overwhelmed when using this kind of software. It is heavily Java based and to those who never touched javac, the Stack traces must look both intimidating and ridiculous. On the other hand, processing data over several steps with a homegrown solution needs a lot of programming discipline and reasoning, otherwise your software turns into an unmaintainable and unreliable mess. In fact this is the case where I work right now...

hans_castorp 8y ago

So there are 2197188 games in that file. Extracting only the lines with "[Result]" in them into new file (using grep) takes about 3 seconds. Importing that into a local Postgres database on my laptop takes about 1.5 seconds. Then running a simple: select result, count(*) from results group by result; Takes about 0.5 seconds. So the total process took only 5 seconds (and now I can run much more aggregation queries on that)

angel_j 8y ago

the hadoop solution may have taken more computer time, but that is far less valuable than a person's the command-line solution probably took that person a couple or more hours to perfect

raghava 8y ago

For many, the incentive from the gig is a) resume enrichment by way of buzzword addition b) huge budget grants and allocation, purportedly for lofty goals while management is really unaware of real technology needs/options Much has been talked about this already; sharing them again! [1] https://news.ycombinator.com/item?id=14401399 https://news.ycombinator.com/item?id=14401399 [2] https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html [3] http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra... [4] https://www.reddit.com/r/programming/comments/3z9rab/taco_bell_programming/ https://www.reddit.com/r/programming/comments/3z9rab/taco_be... [5] https://www.mikecr.it/ramblings/taco-bell-programming/ https://www.mikecr.it/ramblings/taco-bell-programming/

forinti 8y ago

This reminds me of the early EJBs which were really complicated and most people who used them didn't really need them.

verytrivial 8y ago

Anchoring the search would allow the regex to terminate faster for non-matching lines: ... | mawk '/^\[Result' ...

notacoward 8y ago

A lot of people are saying how they've worked on single-machine systems that performed far better than distributed alternatives. Yawn. So have I. So have thousands of others. It should almost be a prerequisite for working on those distributed systems, so that they can understand the real point of those systems. Sometimes it's about performance, and even then there's no "one size fits all" answer. Just as often it's about capacity. Seen any exabyte single machines on the market lately? Even more often that that, it's about redundancy and reliability. What happens when your single-machine wonder has a single hardware failure? Sure, a lot of tyros are working on distributed systems because it's cool or because it enhances their resumes, but there are also a lot of professionals working on distributed systems because they're the only way to meet requirements. Cherry-picking examples to favor your own limited skill set doesn't seem like engineering to me.

koomi 8y ago

It can be colder at night than outside.

icc97 8y ago

Relevant article from 2013: 'Don't use Hadoop - your data isn't that big' [0] and the most recent HN discussion [1]. [0]: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html [1]: https://news.ycombinator.com/item?id=14401399 https://news.ycombinator.com/item?id=14401399

libx 8y ago

Once, I saw a presentation of Unicage, a big data solution that had (perhaps still has) a free version https://youtu.be/h_C5GBblkH8?t=2566 https://youtu.be/h_C5GBblkH8?t=2566 It seems that it has evolved to a company now: http://www.unicage.com/products.html http://www.unicage.com/products.html Did anyone try the unicage solution?

dunk010 8y ago

'Nuff said: https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce

hajderr 8y ago

Ah yes this is so true!

rurban 8y ago

Esp. if you use multipipe-enhanced coreutils, like dgsh. https://www2.dmst.aueb.gr/dds/sw/dgsh/ https://www2.dmst.aueb.gr/dds/sw/dgsh/