16 ms·
Problem solving with Unix commands
- dnet 8y agoRemoving leading zeroes doesn't require Python. One easy solution would be sed: $ echo -e '0001\n0010\n0002' | sed 's/^0*//' 1 10 2
- v3gas 8y agoOh nice. sed still scares me a bit.
- ianai 8y agoIt’s useful but highly cryptic in my usage.
- SomethingOrNot 8y agoI’m not terribly experienced with Unix tools but I reckon that it might be best to just use Perl instead. Then you just have to worry about PCRE instead of PCRE in addition to old-style regexps. Then again, Perl is even scarier.
- jabl 8y agoperl one-liners are pretty powerful and can replace awk/sed/cut/tr/etc. That being said, then you have to remember which command-line options you should give perl (was it -lane to do X, or was it -pie, or something else entirely?). But yeah, the perl rabbit hole goes as deep as you want to, and in some sense it makes it more difficult to say "screw it, I'm redoing this in a real language" as complexity rises. And then you end up with a thousand lines of line noise.
- SomethingOrNot 8y agoI haven’t coded Perl beyond one-liners. What attracts me to it are its regular expressions. So much of Unix scripting seems to involve regexps. So I figure Perl+utilities is the better option compared to utilities+Bash. I wouldn’t wanna use it for more than short scripts. Perl 6 might be fun, but it doesn’t seem to have a large enough community.
- b2gills 8y agoIf you want to improve your coding skills you should read “Higher Order Perl”. (made available for free online by the author) If you want to improve your Perl code read “Modern Perl”. (There is more than one version, and I know the first version was made freely available online) Perl is a better language for large codebases than most people give it credit for. That said it allows more creativity when it comes to your code. So you can make awful code just as easily as beautiful code. Perl6 makes the beautiful code easier to write and shorter, while making awful code a bit harder to write.
- chromatic 8y ago> There is more than one version, and I know the first version was made freely available online Good news! Every version is freely available online. Here's the most recent: http://modernperlbooks.com/books/modern_perl_2016/index.html http://modernperlbooks.com/books/modern_perl_2016/index.html
- b2gills 8y agoWell Perl is a real language, so of course it makes it more difficult to say “screw it, I'm redoing this in a real language”. That said the Perl that you write in a one-liner should be different than the Perl that you write when the complexity rises. If you have more than a few hundred lines, you probably need to move some of it out into a module. There should also be tests for that module. It might even make sense to structure it like a CPAN module so that you can use the existing tools to test and install it.
- jabl 8y agoI have to admit the last time I wrote anything larger than short scripts in perl was in the perl 4 period (on HP-UX to boot). I eagerly awaited the perl 5 version of the Camel book, but in the end I jumped ship to python before getting seriously into perl 5.
- dahart 8y agoDefinitely check out perl one liner patterns. Perl is less scary and more powerful and usually almost as short as sed commands. Perl can often replace pipelines that use both sed and awk. Perl one liners need judicious flags though, common patterns look like perl -ne, perl -pie, perl -lane ... these do very different things. Once you know them, it’s like a minor superpower.
- mshook 8y agoYeah plus seq can generate sequences with leading zeroes (something like seq -f %04.f 1 20). So instead of scripting, he could have generated a sorted list of numbers from the files he had. Created a file with the sequence of numbers for the range and diffed/commed the whole thing. Voilà...
- lelf 8y agoseq -f %04g 0 3 echo {0000..0003} # if bash
- v3gas 8y agoDid not know that. Thanks! That's much easier.
- pwg 8y agoThe seq provided in the GNU toolset has a -w flag to turn on "equal width" mode, so one can also get zero padded numbers out (from GNU seq) by turning on that mode and zero padding the input: $ seq -w 0001 0003 0001 0002 0003 $
- anc84 8y agoNice, this works automatically like this: $ seq -w 98 102 098 099 100 101 102
- neokantian 8y agoYeah, no need for Python. Even the following seems to work fine: $ printf "%d\n" 003 3
- scbrg 8y agoWell, that works up to a point. Fore some, that point might be considered a bit too close to zero... $ printf "%d\n" 009 sh: 1: printf: 009: not completely converted 0
- jandrese 8y ago0-prefix octal notation continues to be a mistake. I love when programs mysteriously fail when passed an IP address like 192.168.001.009. Thankfully modern tools have been moving away from supporting that notation and are less likely to explode all over unsuspecting users.
- neokantian 8y agoYes, you are right. This seems to work better: $ printf "%1.0f\n" 009 9 $ printf "%1.0f\n" 00125990 125990
- lelf 8y ago$ printf %d 010 8
- jfk13 8y agoIf one of the items is actually zero, this would delete it entirely, which probably isn't the desired result.
- leblancfg 8y agoYes. Or if you're going to whip out Python, might as well make it all in Python.
- masklinn 8y agoVery much my thinking, especially when there are no significant commands you need to shell out to: import sys, pathlib basedir = pathlib.Path(sys.argv[1]) for i in range(1, 501): if not (basedir / f'{i:04}_A.csv').is_file(): print(i)
- v3gas 8y agoNice one!
- hellabites 8y agoJust for fun some bash: for k in $(seq -f %04.f 1 501); do f="${k}_A.csv" ! [[ -f "$f" ]] && echo $f || : done or more succinctly, for k in {0001..0501}_A.csv; do ! [[ -f "$f" ]] && echo $f || : done and if you have GNU parallel installed: parallel -kj1 '! [[ -f "{}" ]] && echo {} || :' ::: {0001..0501}_A.csv
- jraph 8y ago$ echo -e '0001\n0010\n0002\n0' | bc 1 10 2 0
- hellabites 8y agoThis is better relative to the `sed` solution as it handles the `0000` case well.
- eMSF 8y agoGiven the situation in the article, you might as well do this: ls ????_A.csv | grep -o '[1-9][0-9]*'
- deleted 8y ago[deleted]
- ciucanu 8y agoI usually do text processing in Bash, Notepad++ and Excel. Each has its own pros and cons, that's why I usually combine them. Here you have the tools I use in Bash: grep, tail, head, cat, cut, less, awk, sed, sort, uniq, wc, xargs, watch ...
- james_s_tayler 8y agoAs an aside I once found out you can replace 'sort | uniq' entirely with an obscure awk command so long as you don't require the output to be sorted. Iirc it performs twice as fast. cat file.txt | awk '!x[$0]++'
- omaranto 8y agoThe awk commands prints the first occurrence of each line in the order they are found in the file. I can imagine that sometimes that might be even better than sorted order.
- pnutjam 8y agosort has a -u option on my linux... ------ -u, --unique with -c, check for strict ordering; without -c, output only the first of an equal run
- 2038AD 8y agoBash with Notepad++ and Excel? Do you use Wine or WSL?
- ciucanu 8y agoIt's kind of mandatory to use Windows in certain envs.
- AnIdiotOnTheNet 8y agoIf you're already in Windows land, you should consider leveraging PowerShell instead of bash. Pretty much all the same tooling is there, only with more descriptive names, tab completion on everything, passes typed object data instead of text parsing, etc.
- LogicX 8y agogiven the limited scope of files in the direcctory... not sure why it was necessary to use grep, instead of the built in glob? ls dataset-directory | egrep '\d\d\d\d_A.csv' which FWIW wouldn't even work, on multiple levels: you need -1 on ls and no files end with A.csv vs ls -1 dataset-directory/*_A?.csv ref: http://man7.org/linux/man-pages/man7/glob.7.html http://man7.org/linux/man-pages/man7/glob.7.html Update: apologies, apparently my client cached an older version of this page. at that time the files were named A1.csv and A2.csv
- deleted 8y ago[deleted]
- olog-hai 8y agoSome ls man pages state the following about the -1 option: "This is the default when the output is not directed to a terminal." I've never needed to use -1 when piping ls's output to another command.
- yuriko 8y agoIf you bother to write a python script to parse the integers, why not use python to solve the whole problem?
- ekianjo 8y agoyeah that was super weird. Why write an article about the merits of shell tools if you put some python in the mix...
- Retra 8y agoPython basically acts like a subshell with its own language... I don't see why anyone would think unix shell scripts are really that different from Python scripts, especially if you're not doing the subprocess control things that command shells are optimal for. Invoking python to do something doesn't seem any more awkward to me than invoking sed, awk, etc..
- ekianjo 8y agoPython has no bindings to the underlying OS (unless you import os modules, and even then it has its own way of working), and does not interact directly with environment variables either, and as such can not be seriously considered like a "shell" tool at the same level as the other ones. It just was never designed to act in this way. And in the embedded world you don't want to install a python interpreter if you can do without.
- Retra 8y agoIf you're just sorting numbers, you don't need the OS or environment variables.
- v3gas 8y agoFair point! I'd argue there's a difference in time spent in writing a python script to solve it all, and just parsing the ints as I did. Python was my first thought for how to parse the ints.
- wmu 8y agoEasier would be just use 'cat list_of_numbers | sort | uniq -u' to get the unique entries.
- phireal 8y agoShorter still: sort -u < list_of_numbers
- adtac 8y agohttp://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html now, I'll wait for someone to post a link to the "UUOC award" award
- aepiepaey 8y agoAnd if you're using cat because it keeps the filename out of the way when editing the pipeline, then just put the redirect before the command instead, so instead of e.g. cat file | grep pattern | sort -u you can write < file grep pattern | sort -u and the filename is out of the way compared to grep pattern file | sort -u
- deleted 8y ago[deleted]
- wmu 8y agoThis is not the same. For sequence [5,5,4,3,3,2,1,1] "sort -u" returns [1,2,3,4,5], while "sort | uniq -u" returns [2,4].
- phireal 8y agoHuh, I didn't know that! Thanks.
- almostarockstar 8y agoThis was a nice read and a good introduction to text processing with unix commands. I agree with the other user re python usage - that you may as well use it for the whole task if you're going to use it at all - but I don't think it's a major flaw. It worked for you right? I would suggest naming the python file a bit more descriptively though. Interesting to read the other suggestions about dealing with this without python.
- v3gas 8y agoThanks! Glad to hear!
- darrenf 8y agoAll the pipes and non-builtin commands (especially python!) look like overkill to me, I must say. for set in *_data.csv ; do num=${set/_*/} success=${set/data/A} if [ ! -e $success ] ; then echo $num ; fi done ETA: likely specific to bash, since I have no experience with other shells except for dalliances with ksh and csh in the mid-90s.
- phaemon 8y agoYup, I'd probably have gone with a `for` loop also. A bit shorter: for set in *_data.csv; do [[ -f "${set/data/A}" ]] || echo "${set%_data.csv}" done Edit: though I just write it out like this for formatting on HN. In real life, that would just be a one-liner: for set in *_data.csv; do [[ -f "${set/data/A}" ]] || echo "${set%_data.csv}"; done
- hellabites 8y agoJust because I like GNU parallel: parallel -kj1 'f="{}"; [[ -f "${f/data/A}" ]] || echo $f' ::: *_data.csv
- ben509 8y agoI've often done this, usually not for a large dataset, but it's sometimes helpful to pipe text through Unix commands in Emacs. C-u M-| sort, for instance, will run the selection through sort and replace it in place. If you're going the all python route, and even want to be able to run bash commands, and want something where you can feed the output into the input, I'd strongly recommend jupyter. (If you want to stay in a terminal, ipython is part of jupyter and heavily upgrades the built-in REPL and does 90% of what I'm mentioning here.) You can break out each step into its own cell, save variables (though cell 5 will be auto-saved as a variable named _5) but the nicest thing is you can move cells around (check the keyboard shortcuts) and restart the entire kernel and rerun all your operations, essentially what you're getting with a long pipeline, only spread out over parts. And there are shortcuts like func? to pop up help on a function or func?? to see the source. It's got some dependencies, so I'd recommend running it in a virtualenv via pipenv: pipenv install jupyter # setup new virtualenv and add package pipenv run jupyter notebook pipenv --rm # Blow away the virtualenv Also, look into pandas if you want to slurp a CSV and query it.
- omaranto 8y agoI doubt you'll find many Emacs users that would prefer "C-u M-| sort" over "M-x sort-lines".
- samwhiteUK 8y agoI thought this was a neat demo of building up a command with UNIX tools. The python inclusion was a bit odd, yes. I learned about sys.stdin in Python and cutting characters using the -c flag
- v3gas 8y agoThanks!
- sureaboutthis 8y ago> I am starting to realize that the Unix command-line toolbox can fix absolutely any problem related to text wrangling. Am I the only one who thought, "No shit, Sherlock"?. This is a fundamental of UNIX that many people don't seem to grasp.
- adtac 8y agoEverybody realises this at some point. Nobody ever thought "I can use this for anything" when they first saw a shell. It takes time.
- SomethingOrNot 8y agoHe’s an MS student. He’s just documenting and sharing his journey. As blogs do. https://www.xkcd.com/1053/ https://www.xkcd.com/1053/
- fforflo 8y agoThis [0] the most complete post I've read on the topic. Lays out all the relevant tools. Spending some time going through each tool's documentation/options, pays off tremendously. [0]: https://www.ibm.com/developerworks/aix/library/au-unixtext/index.html https://www.ibm.com/developerworks/aix/library/au-unixtext/i...
- inp 8y agoReally nice link! Thanks fforflo.
- JdeBP 8y agoFor actual text processing, I recommend a book. * https://oreilly.com/openbook/utp/ https://oreilly.com/openbook/utp/
- SomethingOrNot 8y agoSeems like a book specifically about text processing for the purpose of writing/formatting/typesetting documents.
- jihadjihad 8y agoWow, great find. Sad how hard it seems these days to come across an easy-to-follow primer on a topic without narrative fluff and/or ads everywhere. For those interested in a standalone copy there is a PDF of the content available here https://www.ibm.com/developerworks/aix/library/au-unixtext/au-unixtext-pdf.pdf https://www.ibm.com/developerworks/aix/library/au-unixtext/a...
- wglb 8y agoWriting clear tutorials is a fair amount of effort, more than I originally thought when I first did it.
- Sir_Cmpwn 8y agoMy main problem with this reference is that it encourages the use of non-portable utilities and flags. Double check against POSIX before writing any of this into your scripts: http://pubs.opengroup.org/onlinepubs/9699919799/ http://pubs.opengroup.org/onlinepubs/9699919799/ See also: http://shellhaters.org/ http://shellhaters.org/
- SomethingOrNot 8y ago> I am starting to realize that the Unix command-line toolbox can fix absolutely any problem related to text wrangling. How many problems related to text wrangling arise simply by working with Unix tools? “This philosophical framework will help you solve problems internal to philosophy.”
- skywhopper 8y agoWhat a useless comment. The OP is an interesting walkthrough of solving a highly specific problem in a clever way using a common but often poorly understood toolset. Then you come in and leave a snarkbomb trashing the idea that learning how to use this toolset is worthwhile without providing any reasoning or alternatives. Do you also trash posts about learning how to build your own furniture or troubleshooting car engines? What elevated domain do you operate in that only has perfectly elegant solutions to beautifully architected problems that use only tools perfectly crafted to solve those exact problems? Doesn’t sound like very interesting work to me.
- SomethingOrNot 8y agohttp://www.art.net/~hopkins/Don/unix-haters/handbook.html http://www.art.net/~hopkins/Don/unix-haters/handbook.html
- inp 8y agoInstead to create a script in Python to convert numbers in integers, you can use awk: "python3 parse.py" becomes "awk '{printf "%d\n", $0}'"
- darrenf 8y agoWhy even use awk rather than the shell's (well, bash's) builtin printf? $ printf '%d\n' "0005" 5
- inp 8y agoYou can apply the awk command on a pipe, and so it is applies on each line of the file/stream.
- darrenf 8y agoRight - though that's solvable with xargs: $ echo "0005" | xargs printf '%d\n' 5 That said, my suggestion doesn't work anyway since the leading 0 marks it as octal, d'oh (as mentioned elsewhere in the thread).
- jfk13 8y agoThat might not always do what a naïve user expects: $ printf '%d\n' "0025" 21
- deleted 8y ago[deleted]
- tyingq 8y agoNot sure I understand why it needs to even know there's numbers in the filename. The problem seems to boil down to: "Find all files with the pattern '[something]_data.csv' and report if '[something]_A.csv' doesn't exist" Unless I'm missing something, all the sorting and sequence generation isn't adding anything.
- sagartewari01 8y agoMy favourite one is 'pkill -9 java'. Fixes my laptop if it starts lagging.
- fxfan 8y agoDoes that kill electron instances too? ;)
- clanrebornsx 8y agoDon't do this your intellj idea will crash and fatal dataloss will occur.
- whynotminot 8y agoIt's always good sport to kill java. Warms my heart every time.
- oh5nxo 8y agoIs there a nice alternative for seq or jot ? Something neater than for-loop in awk ?
- jabl 8y agoIn bash, you can create sequences with {A..B}. E.g. echo {1..10} or to count backwards echo {10..0} boom!
- denniskane2 8y agoTime to start porting this brand of innovative thinking to the web... https://linuxontheweb.org/shell.os https://linuxontheweb.org/shell.os
- maratc 8y ago$ join -v 2 <(ls | grep _A | sort | cut -c-4) <(ls | grep -v _A | sort | cut -c-4) The shortest one I could come up with, no need to use python. `join -v 2` shows the entries in the second sorted stream that don't have match in the first sorted stream, the rest is self-explanatory I hope. Edit: $ join -v2 -t_ -j1 <(ls | grep _A | sort ) <(ls | grep -v _A | sort) Is even shorter, it takes first field (-j1) where fields are separated by '_' (-t_)
- v3gas 8y agoNice golf!
- zimpenfish 8y agoSlightly shorter: ls -v|cut -d_ -f1|uniq -c|awk '$1<2{print $2}' Tested by creating 500 sets of dual files and removing 10 `_A` randomly. for i in $(seq 1 500); do j=$(printf %04d $i); touch ${j}_data.csv; touch ${j}_A.csv; done for i in $(seq 1 10); do q=$((RANDOM % 500)); r=$(printf %04d $q); rm -v ${r}_A.csv; done removed '0438_A.csv' removed '0327_A.csv' removed '0150_A.csv' removed '0173_A.csv' removed '0460_A.csv' removed '0194_A.csv' removed '0073_A.csv' removed '0293_A.csv' removed '0404_A.csv' removed '0153_A.csv' And then using the code above to verify the missing files 0073 0150 0153 0173 0194 0293 0327 0404 0438 0460
- maratc 8y agoElegant and short! But unless I'm missing something, your script will print even the datasets that have _A but not the corresponding _data?
- zimpenfish 8y agoA fair point but I was working within the context of the original problem which seemed to be "there's always a _data but not always an _A". If I was trying to provide a robust generic solution, I wouldn't be code golfing it...
- 8y ago
- pletnes 8y agoUseless use of seq spotted. Seq does not exist on many systems. Bash has {0001..0500} instead. Nice writeup though.
- skywhopper 8y agoThe brilliant fun of working with the Unix CLI toolset is that there are millions of valid ways to solve a problem. I also thought of a “better” solution of my own that took an entirely different approach than most of the ones posted here. That’s not really the point. What’s great about this article is that it follows the process of solving the problem step by step. I find that lots of programmers I work with struggle with CLI problem solving, which I find a little surprising. But I think it all depends on how you think about problems like this. If you start from “how can I build a function to operate on this raw data?” or “what data structure would best express the relationship between these filenames?” then you will have a hard time. But if you think in terms of “how can I mutate this data to eliminate extraneous details?” and “what tools do I have handy that can solve problems on data like this given a bit of mungeing, and how can I accomplish that bit of mungeing?” and if you can accept taking several baby steps of small operations on every line of the full dataset rather than building and manipulating abstract logical structures, then you’re well on your way to making efficient use of this remarkable toolset to solve ad hoc problems like this one in minutes instead of hours.
- deleted 8y ago[deleted]
- stiff 8y agoFor learning to get things done with Unix, I recommend the two old books "Unix Programming Environment" and "The AWK Programming Language". There are many resources to learn the various commands etc., but there is still no better place than those books to learn the "unix philosophy". This series is also good: https://sanctum.geek.nz/arabesque/series/unix-as-ide/ https://sanctum.geek.nz/arabesque/series/unix-as-ide/
- pdkl95 8y agoGary Bernhardt[1] gave a great talk about practical problem solving with the unix shell: "The Unix Chainsaw"[2]. "Half-assed is OK when you only need half of an ass." In the talk, he gives several demonstrations a key aspect of why unix pipelines are so practically useful: you build them interactively. A complicated 4 line pipeline started as a single command that was gradually refined into something that actually solves a complicated problem. This talk demonstrates the part that isn't included in the the usual tutorials or "cool 1-line command" lists: the cycle of "Try something. Hit up to get the command back. Make one iterative change and try again." [1] You might know him from his other hilarious talks like "The Birth & Death of JavaScript" or "Wat". [2] https://www.youtube.com/watch?v=sCZJblyT_XM https://www.youtube.com/watch?v=sCZJblyT_XM
- SomethingOrNot 8y ago> In the talk, he gives several demonstrations a key aspect of why unix pipelines are so practically useful: you build them interactively. The standard Unix interface might have been interactive in the ’70s, back when hardware and peripherals were horribly non-interactive. But I don’t know why so many so-called millenial programmers (people my age) get excited about the alleged interactivity of the Unix that most people are familiar with. It doesn’t even have the cutting edge ’90s interactivity of Plan 9, what with mouse(!) selection of arbitrary text that can be piped to commands and so on. And every time someone comes up with a Unix-hosted tool that uses some kind of fold-up menu that informs you about what key combination you can type next (you know, like what all GUI programs have with Alt+x and the file|edit|view|… toolbar), people hail it as some kind of UX innovation.
- SomethingOrNot 8y agoOne interactive feature I like about Bash though (or shell or whatever) is C-x C-e to edit the current line in a text editor (FC). Great when I know how to transform some text in the editor easily but not on the command line.
- jraph 8y agoI think the interactivity you describe might be a different thing from what your parent is talking about. From what I understand, your parent talks about how the commands are built iteratively, with some kind of trial-error loop, which is a strength that is supposedly not emphasized enough. And I agree by the way. Nothing to do with how things are input.
- boomlinde 8y agoA change in structure might be helpful: $ ls data 0001.csv 0002.csv 0003.csv 0004.csv ... $ ls algorithm_a 0001.csv 0002.csv 0004.csv ... $ diff -q algorithm_a data |grep ^Only |sed 's/.*: //g' 0003.csv ...
- v3gas 8y agoExcellent point, haha!
- Upvoter33 8y agoawk one liner: ls | awk '{split($1,x,"_"); split(x[2],y,"."); a[x[1]]+=1} END {for (i in a) {if (a[i] < 2) {print i}}}'
- IronBacon 8y agoZsh one liner (probably works in Bash too): for a in {0001..0500}; do [[ ! -f ${a}_A.csv ]] && echo $((10#${a})); done The only trick I'm using is base transformation to remove padding in the echo...
- hellabites 8y agoI didn't realize Zsh (and Bash) was capable of removing zero padding in that way. Everybody has there own style, but I would prefer to print the missing file pattern and avoid loops. If you have GNU parallel installed (works in bash) parallel -kj1 '! [[ -f "{}" ]] && echo {} || :' ::: $(jot -w %04d_A.csv - 1 501) or if preferred parallel -kj1 '! [[ -f "{}" ]] && echo {} || :' ::: {0001..0500}_A.csv
- IronBacon 8y ago> I didn't realize Zsh (and Bash) was capable of removing zero padding in that way. Well, it's for transforming an integer in a different bases like octal, binary until base 24 (or more don't recall), but it can be abused to strip padding zeroes from variables. Using printf should probably be cleaner but usually I only recall the the C syntax... I think I have parallel installed but I tend to use xargs out of habit, mostly because I was forced to use xargs in locked out production systems. If the number of files wouldn't be so big, I'd simply expand them on ls and capture stderr: ls {0001..0500}_A.csv 1> /dev/null It's a little nosier with the error messages but it's fast. With 500 files I'm sure I'll exhaust the shell parse(?) buffer: (ls {0001..0500}_A.csv 2>&1 1> /dev/null) | awk -F\' '{print $2}' and too much complications to suppress stdout and pipe only stderr. ^__^;
- nickjj 8y agoI think the best part about using Unix tools is it forces you to break down the problem into tiny steps. You can see feedback every step of the way by removing and adding back new piped commands so you're never really dealing with more than 1 operation at a time which makes debugging and making progress a lot easier than trying to fit everything together at once.
- mercer 8y agoIt's basically functional programming. I find that my approach to writing code is very similar to how I work with the shell. The main difference, I guess, is that the command 'units' are slightly bigger, in the form of functions, but the way I iterate my solution to a problem is basically the same.
- omaranto 8y agoThe article solves the problem: for which numbers x between 1 and 500 is there no file x_A.csv? It looks like in this case it is equivalent to the easier problem: for which x_data.csv is there no corresponding x_A.csv? cd dataset-directory comm -23 <(ls *_data.csv | sed s/data/A/) <(ls *_A.csv)
- ebeip90 8y agoThis will fail for any filenames that contain newlines
- omaranto 8y agoCorrect. It is intended for the filenames in the article. More generally, I try to write all my shell code to silently produce hard to track down errors when a filename contains newlines, in order to punish me for my carelessness if I ever accidentally create such a filename.
- deleted 8y ago[deleted]
- jclay 8y agoAfter moving back to working on a Windows machine the last several years and being “forced” into using PowerShell, I now find myself using it for these sorts of tasks on Linux. I now use PowerShell for any tasks of equal or greater complexity than the article. It’s such a massive upgrade over struggling to recall the peculiar bash syntax every time and the benefits of piping typed objects around are vast. As a nice bonus, all of my PowerShell scripts run cross-platform without issue.
- nxrabl 8y agoI've dabbled in PowerShell before, but I've always found the objects you get from cmdlets to be so much more opaque than the plain text you get from Unix output, which makes it harder to use the iterative approach to development the article and other commenters describe. Do you have any tips for poking around in PowerShell objects / a workflow that works for you?
- jclay 8y agoI’ve tried to love it while using it as an interactive shell, but it’s hard for me to lose the Unix muscle memory and remember their verbose commands. For anything more than a single pipe, or anything that requires loops or control flow, I switch to Powershell in Visual Studio Code with the PowerShell extension which has intellisense and helps to poke around the methods on each object. From there you can select subsets of your script and run with F8 which helps me prototype with quick feedback.
- useerup 8y agoUse `gm` (alias for Get-Member) e.g like ps | gm Will tell you exactly the different object types and member methods and properties are returned from `ps`. ls | gm Will tell you that ls returns two different object types (directories and files).
- Dowwie 8y agoMore power to those who enjoy writing control flow in shell, but if I need anything beyond a single line I'm going with an interactive ipython session.
- jancsika 8y agoThe problem with this is that there isn't a standard format forced on the args that following the command name "cut". What makes it worse is that there are seemingly patterns of standard format that get violated by other patterns. It's often based on when the utility was first authored and whatever ideas were floating around during the time. So sometimes characters can "clump" together behind a flag, under the assumption that multi-character flags will get two hyphens. Then some utilities or programs use a single flag for multi-character flags. Plus many other inconsistencies-- if I learn the basic range syntax for cut do I know the basic range syntax for imagemagick? Those inconsistencies don't technically conflict since each only exists in the context of a particular utility. But it's a real pain to sanity to see those inconsistencies sitting on either side of a pipe, especially when one of them is wrong. (Or even when it's a single command you need but you use the wrong flag syntax.) That all adds to the cognitive load and can easily make a dev tired before its time to go to sleep. Oh, and that language switch from bash to python is a huge risk. If you're scripting with Python on a daily basis it probably doesn't seem like it. But for someone reading along, that language boundary is huge. Because the user is no longer limited to runtime errors and finicky arg formatting errors, but also language errors. If the command line barfs up an exception or syntax error at that boundary I'd bet most users would just give up and quit reading the rest of the blog. Edit: clarification
- gcommer 8y agoIt's true that this can be a pain; but this flexibility is also bash's greatest feature: a bash script can make use of almost any other program, regardless of the particular idioms that that program's author was partial too. This is exactly why bash has been so successful for so long and likely will continue to be so for a very long time. Any attempts to tighten this down would raise the barrier for entry and therefore reduce the ecosystem that bash can operate in. Also, it's a bit of a false dichotomy. Any other language is also susceptible to these sorts of inconsistencies. For example: Do I specify a range as [min, max] or as two separate parameters? Is it inclusive or exclusive? etc. At some point all programming interfaces come down to conventions, and if your language only supports one then you'll only be able to interop with the subset of the broader community that agrees with you.
- 8y ago
- iheartpotatoes 8y agoI got paid $175/hr as a data analyst contractor to basically run bash, grep, sed, awk, perl. The people that hired me weren't dumb, just non-programmers and became giddy as I explained regular expressions. The gig only lasted 3 months, but I taught myself out of a job: once they got the gist of it they didn't need me. Yay?
- iheartpotatoes 8y agoThe people that created the command line weren't L33T H4XOR NOOBS. They were brilliant PhD scientists. Let's not confuse the two.
- pixelbeat__ 8y agoSet operations are very useful. Here's a summary: http://https://www.pixelbeat.org/cmdline.html#sets http://https://www.pixelbeat.org/cmdline.html#sets
- pixelbeat__ 8y agohttps://www.pixelbeat.org/cmdline.html#sets https://www.pixelbeat.org/cmdline.html#sets
- redka 8y agols | rb 'group_by { |x| x[/\d+/] }.select { |_, y| y.one? }.keys' https://github.com/thisredone/rb https://github.com/thisredone/rb
- BentFranklin 8y agoFor heavier duty text processing, try emacs -e myfuns.el When it comes to mashing text, nothing beats emacs.
- dahfizz 8y agoYou could use one sed command to replace your grep, cut, and python. It feels cheap to use python do massage data in a post about Unix command line.
- kritixilithos 8y agoNicely done using Unix utils. You can have a pure sed solution (save the `ls` invocation) that is much simpler, albeit obscure, that hinges on the fact that every number has a `data.csv` file. Given a sorted list of these files (through `ls` or otherwise) the following sed code will print out the data files for which A did not succeed on them. /data/!N /A/d P;D This works on the fact that there exists a data file for all successful and unsuccessful runs on data, so sed simply prints the files for which there does not exist an `A` counterpart. If you want to only print out the numbers, you can add a substitution or two towards the end. /data/!N /A/d s/^0*\|_.*//g;P;D Edit: fixed the sed program
- kritixilithos 8y agoActually the following is even shorter /A/{N;d;} So all together this gives the following ls|sed '/A/{N;d;}'
- mklm 8y agoIf you don't mind "cd dataset-directory" beforehand, a shorter and possibly more correct version would be: comm -1 -3 <(ls *_A.csv | sed 's/_.*$//') <(seq -w 0500) | sed 's/^0*//' The OP's solution doesn't seem correct because of the different ordering of the two inputs of `comm': lexicographical (ls) and numeric (seq).
- mklm 8y agoAlthough -w is supported by both GNU and BSD versions of `seq', BSD's ignores leading zeros in input. Thus a more portable approach is: comm -1 -3 <(ls *_A.csv | sed 's/_.*$//') <(seq -f %04.f 500) | sed 's/^0*//'
- adamchainz 8y agoI learnt a lot from the book Data Science at the Command Line, now free and online at https://www.datascienceatthecommandline.com/ https://www.datascienceatthecommandline.com/
- ortekk 8y agoIf you are using python in your pipeline, might as well go all in! from pathlib import Path all_possible_filenames = {f'{i:04}A.csv' for i in range(1,10)} cur_dir_filenames = {Path('.').iterdir()} missing_filenames = all_possible_filenames - cur_dir_filenames print(*missing_filenames, sep='\n')
- js2 8y agoNot the most efficient solution but this is what springs to mind for me: seq 1000 | xargs printf '%04d_A.csv\n' | while read -r f; do test -f $f || echo $f; done
- jon49 8y agoUse F# with a TypeProvider. Of course, I imagine it would take some work learning F# but once you learn it the sky is the limit in what you can do with this data.