13 ms·
What you need may be “pipeline +Unix commands” only
- beagle3 8y agoNot directly command line, but very relevant: https://www.frankmcsherry.org/assets/COST.pdf https://www.frankmcsherry.org/assets/COST.pdf "Scalability, but at what COST".
- TicklishTiger 8y agoif your data set can be disposed by an awk script, it should not be called “big data”. Why not? I don't see how awk is limited to a certain amount of data.
- gsich 8y agoIf your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.
- rzzzt 8y agoMedium data. Substantial data. Just-enough-data.
- anon1253 8y agoMy favorite term is "annoying sized" data, not enough to warrant clusters and HPC, but enough to make a decent laptop crawl to a halt. It's that uncomfortable in-between that makes up the bulk of the data I usually encounter.
- icelancer 8y agoThis is pretty much where we are at, especially with a lot of transforms and processing on high-speed video captures.
- zokier 8y agoThe one nice thing about AWS and its ilk is the ability to spin up big chunky VM for few hours/days for ad-hoc data processing for pretty small amount of money. That can in many cases shift the boundary of annoying sized enough, with the added bonus that your own workstation is not fully occupied by the processing. Of course such approach is not applicable to all scenarios, but it is a useful trick to keep in your sleeve.
- marmaduke 8y agoA while back on RDS we did our migrations by standing up a super fat instance, fail over, migrate, fail back and shutdown the fat instance. This was a great asset, and AWS made this easy.
- gizmo686 8y agoI thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework
- Aeolun 8y agoMost stream processing doesn’t rely too much on RAM, unless you literally need all the data in memory at the same time.
- groestl 8y agoIf it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.
- Aeolun 8y agoTrue, but I can stream process 20GB of data on my tiny 2GB RAM home server as well. That’s not really ‘big data’ in my opinion.
- gsich 8y agoThat is harder to define. Server mainboards can hold more RAM then consumer mainboards. So with 32GB per slot and 4 slots I would set a limit to 128GB? Also this would make so much more tasks "big data". Games with 50+GB are not big data, neither is e.g video conversion.
- groestl 8y agoI think the limit is more in the double digits TB range right now.
- 8y ago
- Aeolun 8y agoYou can still connect a fairly big NAS to a beefy server and do the processing, unless thoughput rate becomes an issue. Saturating a 10Gbit link means you can probably process up to 100TB a day.
- geocar 8y agoOne time I met a company who insisted they were sending tens of TB of data per day and would need multi-PB per year storage compressed. Took one look at the data: All json, all GUIDS and bools. If we just pre-parse it, the entire dataset for a year fits in a few 100s GB uncompressed -- literally could fit on a macbook air for most of the year. The funny thing about "big data" in my experience, is just how small it actually becomes when you start using the right tools. And yet so much energy goes into just getting the wrong tools to do more...
- dredmorbius 8y agoAll of the source text for all of Google+ Communities posts is a few hundred GB. This for 8.1 million communities and ~10 million active users. Add in images and the rest of the Web payload (800 KiB per page), and that swells to Petabyte range. But the actual scale of the user-entered text is stunningly small.
- tetha 8y ago> The funny thing about "big data" in my experience, is just how small it actually becomes when you start using the right tools. Rings way too true for me atm. My current workplace is currently struggling, because one of our application stores something like a combined 300G of analytics data in the database with the application data. Modifying the table causes hours of downtime because everyone claims that backwards compatible db changes are too hard. And everyone is scared because with more users there's "so much more analytics data" incoming. Yes, with 300Gb across 3-4 years. And I'm just wondering why it's not an option to just move all of that into one decently sized mysql/postgres instance. Give it SSDs, 30 - 60Gb of ram for the hot dataset (1-2 month) and it'll just solve our problems. But apparently, "that's too hard to do and takes too much time" without further reason.
- 0db532a0 8y agoRelevant reading: The GNU coreutils manual https://www.gnu.org/software/coreutils/manual/html_node/index.html https://www.gnu.org/software/coreutils/manual/html_node/inde...
- rofo1 8y agoI feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong that I am surprised that even at this inefficacy, it's such a viable and well-paid profession. It's almost as if we are creating insanely complex solutions that in turn require a large amount of developers to support them, whereas we could have chosen a much more practical solution which is self-sustaining.
- Aeolun 8y agoYou forget that using more developers means more headcount, and more headcount means I have more responsibility as a manager. These crazy complex solutions also look a lot more difficult on the slides than the simple 3 layer architecture that’s often shown to me. With something that simple, and requiring so few people, how am I ever going to convince my clients to pay me multiple millions of dollars for it.
- semi-extrinsic 8y agoI guess you should sell them the choice of either the big complex thing written by a large team for $2.3 MM, or the lean highly-optimized solution written by a handful of elite "10X" developers for $1.9 MM. (Of course the 10X developers are in high demand and very expensive.)
- atoav 8y agoAs a young programmer I certainly found some love for all these unix tools and learn new things every day. I think the problem is scale. Back in the day (before I was born), very few people were programmers and the resources they could use were limited. This means they didn’t need insanely complex solutions because they already needed complex solutions just to make it work on the limited hardware. People were trying to solve problems with computers. Nowaday you take a problem that could be solved by an microcontroller with three buttons and make it a cloud app with web server, web interface and all kind of other things like containers. We donlt really tend to ask the question what a good solution would look like. Often it is the case that you just use the technology the developer wants to learn
- angarg12 8y agoI agree with the sentiment that many solutions are over-engineered, but when you need to process billions of records a day, you do need more complex systems. Bottom line: when facing an engineering problem, start with the simplest, fastest to implement solution, and build complexity as necessary. The simple solution suffices most of the time.
- lstodd 8y agoYou know, "billions a day" is only on the order of 10K per second. A single machine can handle that.
- monsieurbanana 8y agoThat would be amazing, wouldn't it? It's not true though, the problem with dealing with billion of operations a day is the spikes, most of the time you don't get a nice homogene rate for 24 hours straight.
- fwip 8y agoDepends on how much latency matters. A lot of big data is batch processing, for which if the data is 3 hours old that's more than good enough.
- deleted 8y ago[deleted]
- pjc50 8y ago> billions of records a day A good GPU can do >1bn calculations per frame.
- dahart 8y agoA good GPU today can do > 1 Tn calculations per frame.
- wodenokoto 8y agoDoes this add anything to the Taco Bell post linked in TFA? I suggest changing the link to: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...
- sokoloff 8y agoTacoBellArticle> I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python. That’s cringe-worthy...
- lsiebert 8y agoYeah, I know some women that are way better at unix then me. Heck I learned to program, to the extent that it wasn't auto didactic, mostly from women. Anyway, it's poor form to put gender into the mix of what good coding looks like. Don't man up, do it hard core or bravely. Don't pussy out, chicken out. Don't try something ballsy, try something gutsy.
- EdwardDiego 8y agoDespite the gendered terms, what makes me cringe is the belief that X is somehow "better" than Y - if you know Python, and have access to Python, and express yourself in Python faster, then use Python. I use sed, awk, cut, Python, as needed - whatever lets me solve my problem faster. No flexing about how you use X needed.
- kurczynski 8y agoI hate to be that guy, but they're NOT "Unix" tools, as the name GNU literally states. The post makes a good point that I fully agree with, just doesn't explain it well enough.
- codetrotter 8y agoI and many others take “Unix” to mean any UNIX or UNIX-like operating system, whereas “UNIX” with all capital letters means only the ones that are certified UNIX.
- mrighele 8y agoMany of the GNU tools are reimplementation of already existing tools. For example the initial implementation of AWK was in 1977 [1], a few years before GNU even existed [2], so it _is_ a Unix tool. [1] https://en.wikipedia.org/wiki/AWK#History https://en.wikipedia.org/wiki/AWK#History [2] https://en.wikipedia.org/wiki/GNU#History https://en.wikipedia.org/wiki/GNU#History
- taejavu 8y agoawk predates GNU by at least 7 years. It is most certainly a UNIX tool.
- cyphar 8y agoThe GNU coreutils are a reimplementation of the Unix utilities. With some caveats, the same tools are available on BSD and Solaris derivatives which are both Unix. So while GNU is obviously an important project, it would not be correct to say "xargs" is not a Unix tool.
- ernesth 8y agoawk is in the POSIX norm. It is a Unix tool. GNU awk is one of the popular awk implementation (usually referred as gawk). I personnally prefer mawk. awk is not GNU.
- nik1aa5 8y agoI have been using the command line for all type of work for years now. The most satisfying is to realize that there is always more to learn. And once you grasped the basics, they fit together like LEGO bricks. While I think it's important to make that argument, the posted article and the one it refers to lack some guidance on how to reach "command line mastery". I recently came across this great resource here on HN: https://github.com/jlevy/the-art-of-command-line https://github.com/jlevy/the-art-of-command-line It gives great overview of the toolbox you have on the command line. Equipped with `man` you're ready to optimize your everyday work. And always remember to write everything down and ask yourself WHY something works the way it works. The interface of the standard tools is thought out very well. Getting comfortable with this mindset pays off.
- ambrop7 8y agoUnless your data contains spaces, tabs, or, god forbid, newlines. Unix pipeline tools lack any sort of useful data structuring capabilities, making them appropriate for one-off tasks at most.
- johncolanduoni 8y agoYeah I feel like there’s a pretty broad medium between Hadoop and awk. I generally find a python script to be a much clearer solution than an dense line of awk; the latter tends to turn into clever code golf pretty quickly.
- devnonymous 8y agoWhat are you on about? I can't think of one Unix command that manipulates structured data and doesn't offer a delimiter option. Furthermore the entire point (if you take a moment to think about it) of pipelines is, instead of all commands supporting your preferred delimiter, you need one command to translate (I don't know perhaps something named tr? ) in the pipeline to deal with such limitations. Do you seriously think non-whitespace separated structured records is a novel idea which the simpler times of Unix didn't have to deal with? Have you looked at the passwd file? /rant
- mrighele 8y agoSpaces, tab and newlines are not a problem. The fact that there are some standard tools available doesn't mean you are limited to that. If you have a CSV file with spaces and newlines use cvskit or a small python script importing the relevant library. If you have to parse JSON file you can use jq to pick the relevant fields regardless of how the document is formatted. You can even process binary data as long as the file format is understood by the tool.
- zimpenfish 8y agoTranscoding CSV to TSV using csvkit has been a life saver for me the last few months due to an old industry's insistence on CSV as an information delivery method. jq has previously been a lifesaver too.
- deleted 8y ago
- kissgyorgy 8y agoThis old article has the same topic with a more complex example and a surprising result about parsing 3.46Gb of data: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- m_sahaf 8y agoA few years ago I had a personal project where I needed to load some of Wikipedia's tables into MySQL on a cheap computer. On my first attempt to just load the downloaded SQL scripts, the computer churned for 3 days before I called it a quits on the attempt. This article inspired the alternative approach: strip the first few lines down until the first Insert collection of inserts, use sed to convert '),(' to \n, then use sed again to strip the first opening and the last closing parens leftover from the earlier operation. Now we have plain CSV. Import it into MySQL. The whole operation took about an hour or two, including importing into MySQL. Yes, I lost the indices, but it was a prototype, so nothing much was lost.
- aboutruby 8y agoMy favorite thing has been `| ruby -e "puts STDIN.to_a. ..."`, allows to run any kind of code on the standard input, much easier than remembering awk/sed various options and much more powerful. edit: same thing can be done with Python/Perl
- apabepa 8y agoYes, easier if you know ruby.. sed/awk is good for munching strings in small scripts and it is also always installed on a unix system (AFAIK).
- dredmorbius 8y agoYou'll find sed and awk in busybox, virtually always, even on minimal systems. Including embedded devices, routers, Android, etc.
- speedplane 8y agoThis article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are great up to a few GBs of data, Excel is even better if you're dealing with less than a few tens of MBs. But to deal with Terabytes of data quickly and efficiently, these tools totally break down.
- aidos 8y agoWell, the article does explicitly say that if you can do this then you don’t have “big data”. Maybe you see TB level processing a lot in your line of work, but most developers never will. Whenever I deal with anything a bit bigger, I break off the smallest section I need to deal with and work with that.
- speedplane 8y ago> Maybe you see TB level processing a lot in your line of work, but most developers never will. I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine). I don't know the percentage of developers that deal with this, but I do know that it's pretty common.
- dredmorbius 8y agoSampling or subetting your data is almost always the correct response, especially for analysis. If you're literally processing TB or PB of data, you'll want to parallelise. Though shell tools do this amazingly well.
- hashhar 8y agoI have a simple bash script composed of 6 sed commands piped together to convert >100GB csvs (table dumps of recommendation data) into Redis binary protocol which are then ingested into a Redis Cluster using redis-cli --pipe. It takes somewhere around 15 minutes running on a modest bare metal server.
- bibyte 8y agoThis submission seems weirdly relevant. https://news.ycombinator.com/item?id=19271135 https://news.ycombinator.com/item?id=19271135
- sizzzzlerz 8y agoI was going to comment on exactly this article except that I couldn't find it quickly. It isn't weirdly relevant; it's totally relevant. It demonstrates that, with sufficient knowledge of the command line, one can write the most amazing tools, quickly and succinctly. Here, knowledge doesn't necessarily meaning knowing everything immediately but also knowing what resources to reference to find out stuff. I've been programming for 40 years and using unix/linux since the 80's and in this little one-line script, I discovered two things that one can do with the appropriate arguments that I've never known. YMMV.
- hackerm0nkey 8y agoYup. Totally agree with OP. Early on in my career I had to generate on the fly reports for hundreds of GB of data and all it needed was to throw some *NIX commands around and eventually piping them to awk to do the final bit and it was blazing fast. These days, these are called big data. No it isn't...
- jimmy_ruska 8y ago* If it's simple transforms, use cli tools. * If it requires aggregation and it's small, use cli tools. * If this is data you're using over and over again then load it in the database and then do the cleaning, ELT. * If it's 2tb of data and under, still use bzip2, get splittable streams and pass it to gnu parallel. * If it requires massive aggregations or windows, use spark|flink|bleam. * If you need to repeatedly process the same giant dataset use spark|flink|bleam. * If the data is highly structured and you mainly need aggregations and filtering on a few columns use columnar DBs. I've been using Dlang with ldc a lot because of how fast its compile time regex is, and its built in json support. Python3+pandas is also a good choice if you don't want to use awk.
- jandrese 8y agoFor bzip2 why not just use pbzip2? Frankly, I wish distros would replace the stock bzip2 with pbzip2 (I think it's drop in compatible).
- hedora 8y agoBefore reaching for spark, etc: Sort is good for aggregations that fit on disk (TBs these days, I guess) Perl does well too if the output fits in a hashtable in DRAM, so 10’s (or maybe 100’s?) of GBs
- sn41 8y agoI use an awk interpreter called mawk.[1] This is noticeably faster than gawk or other standard variants. [1] https://invisible-island.net/mawk/ https://invisible-island.net/mawk/
- dredmorbius 8y agoOften, yes, though not always, and mawk has some omissions relative to gawk. Try multiple interpreters with timings. Gawk's profiler can be invaluable.
- dredmorbius 8y agoCase in point from my own recent work: I've been analysing characteristics of Google+ Communities, mostly looking for plausibly active good-faith instances. There are 8.1 million communities in total, and thanks to some friendly assistance, I'd identified slihtly more than 100,000 with both 100 or more members, and visible activity within the preceeding 31 days, as of early 2019. The task of Web scraping those 100k communities, parsing HTML to a set of characteristics of interest, and reducing that to a delimited dataset of about 16 MB, was all done via shell tools, and on very modest equipment. Most surprising was that parsing the HTML (using the HTML-XML utilities: http://www.w3.org/Tools/HTML-XML-utils/README http://www.w3.org/Tools/HTML-XML-utils/README) took longer than downloading the data. Creating the datafile was done with gawk, and most analysis subsequently in R, though quick-and-dirty summaries and queries can be run in gawk. Performance: downloading (curl): 16 hours, parsing (hxextract & hxselect) 48 hours, dataset preparation (gawk): 2 minutes, analysis (gawk / R), a few seconds for simple outputs. The parsing step is painfully long, the rest quite tractable.
- pdimitar 8y agoAre you planning on open-sourcing the downloader part? I'm very interested.
- dredmorbius 8y agoLiterally just a Bash while-read loop over community IDs. It's embarrassingly trivial. I'm planning on posting the data, probably to https://social.antefriguserat.de/ https://social.antefriguserat.de/ and will include procssing scripts. This is the fetch-script, which saves both the HTML and HEAD responses: #!/bin/bash sample_file=$1 comm_path='community-pages' base_url='https://plus.google.com/communities' i=0 time sed -e 's,^.*/,,' $sample_file | while read commid; do i=$((i+1)) echo -e "\n>>> $i $commid <<<" 1>&2; url="${base_url}/${commid}" commfile="${comm_path}/${commid}.html" commhead="${comm_path}/${commid}.head" echo "curl -s -o '${commfile}' -D '${commhead}' '${url}'" done The sample file is simply a list of G+ community IDs or URLs, e.g.: 100000056330101053659 100000310247038604843 100000355641542704509 100000408644688836681 100000537266485621548 100000813948204546252 100001055751908082772 100001158162744298957 100001173291703462139 100001193552641351693
- DanielBMarkham 8y agoFor my next book, I'm working through a concept I call "good enough programming. Good enough programming is something you code that provides something people want -- and you never look at the code again. Find the problem, solve the problem, walk away from the problem. That's not sexy. It's not going to get you an article to write for a famous magazine, but it's good enough. We have lost sight of "good enough" in programming, and without some kind of guardrails, we end up doing stuff we like or stuff that sounds good to other programmers. For instance, while I love cloud computing, I'm seeing "how-to" articles written about setting up a VPC for doing something like playing checkers. Yes, it was an oversimplified article, and you have to write that way, but without wisdom, how is the reader supposed to know that? What criteria do they use to determine whether it's a co-lo server, a lambda, or a world-wide distributed cloud? We're going like gangbusters selling programmers and companies on all kinds of new and complex ways of doing things. They like it. We like it. But it is in anybody's best interest over the long run? Recently I rewrote a pet project for the third time. First time it was C#, SQL Server, and an ORM. Then it was F#, MySQL, and linux. The last time it was pure FP in F# and microservices. Some of you may know where this is going. Just as I finished writing the app in a real microservices format, I realized. Holy cow! This whole thing was just a few Unix commands and some pipes. My thinking went from all kinds of concerns about transactions and ORM-fun to just some nix stuff in a small script. The problem stayed the same. Something else happened too. At each step, I did less and less maintenance. The last rewrite has had no maintenance required at all. In my spare time, I'm going to do the nix one using no servers at all on a static SPA. In a very meaningful way, there's no app, there's no server, and there's nothing to maintain. Yet I still get the functionality I need. And I never maintain it. Of course that's not possible for every app, but the key thing I learned wasn't the magic of serverless static SPAs or the joys of unix. It was that I didn't know whether or not it was possible or not until I did it. By thinking in a pure FP fashion and deploying in true microservices, the rest just "fell out" of the work. At first I was actually thinking in a way that would have only led to more and more complexity and maintenance requirements. My belief is that we get our thinking right first, use code budgets, and try for a simple unix solution. If it doesn't work, why? At least then we've made an effort to be good enough programmers. That beats most everybody else.
- MrQuincle 8y ago
- kschiffer 8y agoNice article, though I think you should work on your blog's typography. Using a bold typeface for body copy is not pleasant to read.
- nickjj 8y agoUnix commands definitely go a long ways. I've been freelancing for a long time but never automated invoicing people up until recently. So I combined grep, cut, paste and bc to parse a work log file to get how many hours I worked on that project, what amount I am owed and how many days I worked. I can run these analytics by just passing in the log file, a YYYY/MM date (this month's numbers), YYYY date (yearly numbers) or no date (lifetime). Long story short, the working prototype of it was 4 lines of Bash and took about 10 minutes to make. Now I never have to manually go through these work log files again and add up invoice amounts (which I always counted up manually 3 times in a row to avoid mistakes). If you're sending a bunch of invoices a month, this actually took kind of a long time and was always error prone.
- twic 8y agoI've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either because the input is small, or because it is highly repetitive. Made-up example, finding all the file extensions used in a directory tree: find /usr/lib -type f | sed -rn 's/^.*\.([^/]*)$/\1/p' | awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' That script is easily tweaked, eg to uniquify by a part of the string. Say you have a log file formatted like this: 2019-03-03T12:38:16Z hob: turned to 75% 2019-03-03T12:38:17Z frying_pan: moved to hob 2019-03-03T12:38:19Z frying_pan: added butter 2019-03-03T12:38:22Z batter: mixed 2019-03-03T12:38:27Z batter: poured in pan 2019-03-03T12:38:28Z frying_pan: tilted around 2019-03-03T12:39:09Z frying_pan: FLIPPED 2019-03-03T12:39:41Z frying_pan: FLIPPED 2019-03-03T12:39:46Z frying_pan: pancake removed If you want to see the first entry for each subsystem: awk '{ if (!($2 in seen)) print $0; seen[$2] = 1; }' Or the last (although this won't preserve input order): awk '{ seen[$2] = $0; } END { for (k in seen) print seen[k]; }' I don't think there's another simple tool in the unix toolkit that lets you do things like this. You could probably do it with sed, but it would involve some nightmarish abuse of the hold space as a database.
- mooreds 8y agoI did a lightning talk on awk last year and found this great article series from 2000 on all the powers of awk (including network access, but not yet email :) ). https://www.ibm.com/developerworks/library/l-awk1/index.html https://www.ibm.com/developerworks/library/l-awk1/index.html
- mobilemidget 8y agoSo I assume you would use some sort of ' grep XX | sort | uniq' (I still) to get a unique line output. Is this awk line now your default, or did you find yourself using both for convenience? Do you alias these awk commands on all machines you work on, or other way put, I did not find a nice way to keep my custom aliases 'in sync' over different machines, perhaps you have some recommendation or workflow that is really sweet? TIA!
- ChlorophZek 8y agoYes, buddy, the simplest tools are the most powerful, and the fastest. Time is money.
- tomcooks 8y agoThis is what you get by not answering RTFM to stupid questions.
- havkom 8y ago> BTW, if your data set can be disposed by an awk script, it should not be called “big data”. I think this statement is wrong. The popular meaning of the hype term “big data” can not be easily changed. Rather, awk, sed and other tools that can read from stdin and write to stdout are great tools for “big data” and often more efficient and suitable than larger and more hyped systems.
- 3pt14159 8y agoI really agree with aspects of this, and I think CLIs and Unix pipes are way more powerful than we treat them, but be forewarned that there are problems with doing everything with pipes. You need to code more defensively with them. For example, it is rare, but every so often a newline will be fail to be emitted. kinda\n likethis\n \n example\n There are many other gotchas, but that one is a doozy because if you're using, say, tab delimited data and cut you'll miss a line. It's one of the reasons I use line delimited JSON if at all possible. Also, this constant re-parsing of text does mean your string validation needs to be more paranoid. For example, some JSON parsers parse curly quotes as normal programming quotes. Horrible practice, I know, but it could have been avoided. Also, it's easy to accidentally do shit like this when you're in a rush. Some string matching tools handle the character matching of the different ways of creating, say, "ë" will also make matching quotes more relaxed. Anyway, all of this to say that I 100% agree with the posted and linked articles, but each method has its own security considerations and software folk should be aware of them before starting.
- AnIdiotOnTheNet 8y agoThis is one of the reasons I prefer PowerShell, it requires a lot fewer text parsing shenanigans. UNIX tools simply failed to evolve. Single io stream pipelining on raw ASCII was perfectly reasonable in the 1970s but it isn't the 1970s anymore. We should be composing tools with multiple typed io stream paths in GUIs (or TUIs I suppose), leveraging two or even three dimensional layouts. All our interfaces should be composed this way, allowing us to take them apart and modify them at will to fit our workflow. But that never happened. We never made a better hammer, we just try to squeeze all our problems into ASCII-processing nails instead.
- marmaduke 8y agoBut do you have similar tools as te core CLI tools on Linux like grep and sed? How do you discover functionality without man?
- 3pt14159 8y agoYou know it is so funny that you're mentioning this. I completely agree. It's to the point where I've been toying around with creating my own shell and faking typed IO streams via Postgres+DSL. It's tricky though. Sometimes I want pub-sub, other times I want event stream. Sometimes I want crash-on-failure, other times I don't. There is this problem in software that I can't really word precisely, but the closest I can come is "do it like this, except these cases here, except-except those cases there" and these things kinda keep stacking up until you have a program that has too much knowledge baked into it. Take, for example, emoji TLDs. Because emojis aren't consistent across platforms they can get coerced into different types. I didn't know that when I bought and used a couple emoji domains. When someone tried to click on a link in Android and was met with a 404, I was so confused. I wasn't even seeing the request come into nginx! After I figured it out, I realized that emoji domains won't work. The underlying assumption of TLDs is that there is one, and only one, way of encoding something and that these things aren't coerced. That assumption is wrong.
- nudpiedo 8y agoI don't get the motivation of this article, it links to the taco bell programming article which says exactly the same. I usually wouldn't write an article to repeat the same another article says, or if it is something that could have been just a comment in the original blog.
- nudpiedo 8y agoI totally agree that most of the *nix tools are most of the time the best ones, but things get trickier when there is a complex dynamic pipeline e.g. the input conditions the kind of processes and these are also dynamic based on other inputs.
- fwip 8y agoI've found Nextflow to be an excellent solution to parallelizing and adding extra logic to any cli pipeline. It also helps manage environment and track metrics.
- 0x445442 8y agoOh how I miss Ted Dziuba posts. He's was a treasure trove of pragmatism for the industry.
- JJMcJ 8y agoOne place installed a Hadoop cluster. Some of us suspected that replacing a regex monstrosity with Lex/Yacc might have given the speed needed to process the files. EDIT: Lex/Yacc that is some faster parser generator, I'm not too knowledgeable on that.
- Svoka 8y agoI had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run. Then I decided to run it on larger dataset (because I needed too). Like week of logs, not a day of logs. While it was running, I wrote rust CLI, which was working like `cat /*.log | logparser` and did one day in 12 seconds, and a week in a two minutes. And I gave up waiting on awk, btw. It is not always better to use command line. If you have gigabytes or tens of gigabytes of data, it would be easier to write some cli tool to help you out. Also, it was much easier to put significantly more complex logic into it because of type checking, and, you know, being actual high level programming language, not hack&slash awk script. EDIT: Looking back on my "one liner" vs "rust cli" I would not be able to make meaningful adjustments to one liner comprehension. It is, to my sorrow, write-only thing.
- draw_down 8y agoYeah. These tools are very convenient when they work, but there are lots of situations where they don’t quite cut it.
- unhammer 8y agoIf your awk script gets too long / unreadable you just put it in a file and use some whitespace and longer variable names. AWK scripts tend to be very readable (much more so than e.g. sed) as long as they stick to the "stateful filters" use-case as https://news.ycombinator.com/item?id=19294195 https://news.ycombinator.com/item?id=19294195 calls it, but yes they have their limits. If speed is a concern, you may want to try using mawk instead of GNU awk/gawk. I've had 4x speedups with mawk.
- DenseComet 8y agoBased on his timings, rust achieved a 20x speedup vs a 4x speedup if he used mawk.
- hedora 8y agoYep, and adding gnu parallel or xargs to the mix would give a # of cores speedup, which would make mawk faster than the single-threaded rust on a > 6 core machine (roughly).
- dahart 8y ago> Every item on the menu at Taco Bell is just a different configuration of roughly eight ingredients. HA! This has almost been my line for years regarding Mexican food. What I like to say is: it’s amazing how every possible permutation of 8 ingredients has been named. BTW I love Mexican food, lived in Mexico. > The post mentions a scenario which you may consider to use Hadoop to solve but actually xargs may be a simpler and better choice. I do feel like there’s a corollary to Knuth’s “premature optimization” quote regarding web scaling; premature scaling and using tools much bigger than necessary for the job at hand is pretty common.
- ims 8y agoSometimes there's a middle ground: make your "map" and "reduce" steps separate scripts. If you want to do the parsing in Python instead of awk, just make a tiny script that reads from stdin and writes to stdout - that way you can put it between xargs or parallel and whatever else is in the pipeline. The parallelization is a separate concern, so it doesn't need to be mixed in with the parsing (or whatever) concern. The downloading is a separate concern; use wget or requests in a Python script or whatever, it doesn't need to be mingled with the parsing.
- booleandilemma 8y agoAt my company what you can do is determined by which AWS services you can string together.
- dontbenebby 8y agoDoes anyone have suggestions on books to grow my scripting fu? (End of chapter exercises tend to be useful for me) I know bash, and know a lot of basic commands, but I'm not familiar with some more advanced things. I don't know awk or sed for example.
- 01100011 8y agoWhat really helped me is my desire to automate most aspects of my job. If I'm doing something more than once(and twice is more than once), I prefer to automate it. I'm really stupid, and I'm really, really lazy. I am not good at repetitive tasks, but computers are. If there's one thing I feel is important for every programmer to get, it's that our single reason for existence is to make computers do work for us. We should look for opportunities to do so, not always because it is the most efficient thing to do, but because it is what we do. So I end up writing a lot of bash scripts. A lot of small python scripts. Stupid scripts. Over time, you build up a library of tricks. You read threads like this where you pick up new tricks. It isn't something you'll learn once. Some of the tools take years to really get the feel of.
- photon_lines 8y agoIf you're looking for good books, I can vouch for 'A Practical Guide to Linux Commands, Editors, and Shell Programming' which has very through coverage and end of chapter exercises. For PowerShell, I'm currently reading the the free PowerShell Notes for Professionals ( https://books.goalkicker.com/PowerShellBook/ https://books.goalkicker.com/PowerShellBook/ ) and it's a great resource as well.
- brogrammer2018 8y agoThank you for sharing
- dontbenebby 8y agoThanks for the book suggestion, that looks like exactly the kind of thing I'm interested in.
- schoen 8y agoWhen I was in middle school, a friend and I had an internship with a physicist who wanted us to write some software to perform a simple transformation on data sets that he downloaded by FTP from some experiment. We spent about a week writing a program in QuickBASIC that successfully parsed the files and performed the transformation. Some years later, I realized that this would be a one-line awk script which I could now write in 20-30 seconds. (Probably someone comfortable with Excel could also perform the transformation in 20-30 seconds, although it might not scale as well to larger files.)
- jabl 8y agoThis is the Big Data version of "90% of Oracle instances could be replaced with sqlite". Pragmatism almost always loses to CV-padding and office politics.
- 01100011 8y agoIt's not just data processing. There are simple solutions to all sorts of tasks. I run 'motion' on my linux desktop at home to serve as a security camera when no one is home. For months I've been manually starting and stopping it, figuring I needed to setup an IoT system if I wanted to automate things. i.e. IFTTT on our phones, an MQTT server in the cloud, etc. Then I realized - I just need to start the camera when all of our phones are off the LAN. It took about 15 minutes to setup, and now I never have to worry about forgetting to stop or start the camera.
- AdieuToLogic 8y agoGUI's are the McDonald's of interfaces. The choices are limited and when you go there you know there's a good chance that you will end up wondering why what you got caused so much pain.