32 ms·

The most surprising Unix programs

539 points by vitplister 7y ago

smitty1e 7y ago
Hadn't heard of most of these. The peoples' names were more recognizable.
morelisp 7y ago
> struct - Brenda Baker undertook her Fortan-to-Ratfor converter against the advice of her department head--me. I thought it would likely produce an ad hoc reordering of the orginal, freed of statement numbers, but otherwise no more readable than a properly indented Fortran program. Brenda proved me wrong. She discovered that every Fortran program has a canonically structured form. Programmers preferred the canonicalized form to what they had originally written. We could've had prettier et al instead of style linters 40(+?) years ago. :(
- qubex 7y ago
  I had to look up ‘Ratfor’ because I’d never heard of it — apparently it’s a FORTRAN preprocessor that added control structures.
  - msla 7y ago
    https://en.wikipedia.org/wiki/Ratfor https://en.wikipedia.org/wiki/Ratfor The original Ratfor brings FORTRAN 66 nearly up to the level of a respectable programming language. It turns this: if (a > b) { max = a } else { max = b } Into this: IF(.NOT.(A.GT.B))GOTO 1 MAX = A GOTO 2 1 CONTINUE MAX = B 2 CONTINUE ... with proper columnization, of course. Going the opposite direction is pretty miraculous to me. Ratfiv is the follow-on, which did the same to FORTRAN 77. However, FORTRAN 77 had control structures beyond the conditional GOTO, so Ratfiv was somewhat less necessary. FORTRAN 77 would look like this: IF (A .GT. B) THEN MAX = A ELSE MAX = B ENDIF https://en.wikipedia.org/wiki/Ratfiv https://en.wikipedia.org/wiki/Ratfiv
  - davidwihl 7y ago
    One of the books that most influenced my coding was Software Tools by Brian W. Kernighan, P.J. Plauger [0]. Even though I never used Ratfor, the clear descriptions were immensely useful. [0] https://www.goodreads.com/book/show/515603.Software_Tools https://www.goodreads.com/book/show/515603.Software_Tools
    - qubex 7y ago
      I have that book. I have read snippets of it. Evidently I should read all the way through it.
- mhd 7y ago
  As usual, the original paper is paywalled, but it appears that this is about transforming ancient Fortran from GOTOs to structured control-flow (if-then, loops etc.). That has almost nothing to do with the spaces-and-braces nitpicking of prettier/gofmt etc.
  - morelisp 7y ago
    Not "almost nothing to do" - putting programs into a readable normal form seems the natural evolution of these tools. When I started programming in the 90s, "spaces-and-braces" checking - as you say, nitpicking - was basically all we had, along with limited automatic tools to fix them (all more or less as good as `M-x indent-region`). If you were lucky and in a widely-used language you could cobble together compiler warnings, lint, and a few other tools to also get warnings about legacy interfaces (gets), dangerous practices (ignoring error codes), and unusual structure (shadowed variables, loop conditions that seemed impossible). Today we finally have considerably better tools that don't just check if you match a style guide but do a full reformat (not nitpicking, but doing it for you) and linters that can enforce 'deeper' structural demands, sometimes with automatic fixes. But 40 years ago we had tools to completely restructure programs to a normalized form, and the practical experience to know programmers found this form preferable! And like so many things in our field, 10-20 years later we had to rediscover it, painfully, all over again. Probably because today's programmers think source-to-source Fortran/Ratfor translation has "almost nothing to do" with the challenges facing them today.
    - TheDesolate0 7y ago
      I _still_ do this today with new (to me) code bases! By doing this I really read the code, and really get to understand what the previous programmer was doing. Also something I took from Asimov's foundation series, code that doesn't look right, doesn't run right. I know, not really; compiler gives no fucks, but I'm not a compiler however, and GCC error messages (Clang too!) are still about as useful as a hot bikini wax is to a walrus. This was one of the features that made me really fall in love with Emacs back in the day! I could set it up to force my style requirements, then even yanked (pasted) code would be proper (mostly) and I couldn't fat finger my code to death. My only emacs complaint is lisp. I get it, I just don't like it. I'll take fortran 77 over lisp any day (not 66 or before, tho. I'm not that crazy). So, sorry mr(s) moar lisp.
tannhaeuser 7y ago
What's surprising about eqn, dc, and egrep? I'm using the latter two all the time, and have used eqn (+troff/groff and even tbl and pic) in the 1990's for manuals and as late as (early) 2000's to typeset math-heavy course material. Not nearly as feature-rich as TeX/LaTeX, but much more approachable for casual math, with DSLs for typesetting equations, tables, and diagrams/graphs. I was delighted to see that GNU had a full suite of roff/troff drop-in replacements (which I later learned was implemented by James Clark, of SGML and, recently, Ballerina fame).
- saagarjha 7y ago
  The algorithms behind them.
- Mediterraneo10 7y ago
  I had never heard of eqn and was surprised to find that the binary is still there on my Linux box. With regard to roff in general, when I got into Linux-based typesetting around the turn of the millennium, that was already seen as antiquated tech, superseded by LaTeX which was undergoing a frenzy of development and improvement around that time. So, anyone under the age of 30 will probably be hearing of such *roff stuff for the first time (and sadly even familiarity with LaTeX has waned).
  - tannhaeuser 7y ago
    Ok I'm probably showing my age here then :) Back in the 1980 and 1990s, the roff suite, and most definitely egrep and classic Thompson DFA construction and DFA->NFA conversion was definitely Unix folklore/taught in Uni. Manpages are still rendered using roff/groff today, so probably many of us are using it regularly. Whereas GNU's texinfo has matured less well I'd say, or wasn't even very useful in practice to begin with due to lack of content. I'm also using TeX/LaTex, but it's still a programming language whereas roff/eqn etc are non-Turing DSLs and renderers for particular narrow purposes. I get your point, but saying these are "antiquated" is like saying HTML is obsoleted by JavaScript.
    - 7y ago
saagarjha 7y ago
And people say theoretical computer science isn’t useful in “the real world”… I am curious about this one, though, has anyone used it? > The syntax diagnostics from the compiler made by Sue Graham's group at Berkeley were the mmost helpful I have ever seen--and they were generated automatically. At a syntax error the compiler would suggest a token that could be inserted that would allow parsing to proceed further. No attempt was made to explain what was wrong. On the surface it sounds a lot like it would produce error messages like “expected ‘;’” that most beginner programmers come to hate: was it any better than this, or was that the extent of its intelligence and everything else at the time was even worse?
- thaumasiotes 7y ago
  > On the surface it sounds a lot like it would produce error messages like “expected ‘;’” that most beginner programmers come to hate Do people really come to hate these? I'd expect the opposite -- that people would start off hating messages like "expected ';'", but fairly quickly become accustomed to what they almost always mean. As long as you can look at the message and have a good idea of what's wrong, it's not a bad message.
  - saagarjha 7y ago
    The issue is that the solution to errors like these is often not adding a semicolon, but something else like “the compiler has no idea what is going on in this line” and the actual problem can range from something like an unbalanced delimiter, a misspelled keyword, or even a “syntactically valid-looking” (that a stupid parser, like a code highlighter or formatter, would approve of) but ultimately subtly illegal construct. With practice is usually becomes fairly easy to figure out where the actual error is, but it’s certainly not a very good experience for beginners. (And they’ll “come to hate it” because they’ll see it more than a few times before understanding how to deal with them.)
    - dottedmag 7y ago
      Pascal grammar is sufficiently different from C-alikes, so it might explain why for Pascal source code these messages are more meaningful.
ruslan 7y ago
I would add bc to the list, very useful to make occasional calculations from command line using "human readable" syntax.
- bonzini 7y ago
  Fun fact, the first version of bc was just a frontend to dc. It converted the structured input to dc's stack-based form and let dc do the math.
  - ruslan 7y ago
    Did not know that, thanks. I searched for dc inside bc and could find a reference to /usr/bin/dc, so I think bc still is just a wrapper. % uname -a FreeBSD skyrocket 9.3-RELEASE FreeBSD 9.3-RELEASE #1: Fri Nov 27 20:28:19 UTC 2015
    - bonzini 7y ago
      GNU bc isn't though they share the bignum code, I am not surprised that the BSDs are following the older implementation more closely!
ur-whale 7y ago
The fact that dc does (or at least tries to) guarantee error bounds on the result is news to me. And if that does indeed work, that's pretty cool.
- nn3 7y ago
  I doubt the modern GNU or BSD versions of it that you are likely using do. Noone uses the original anymore.
  - swixmix 7y ago
    Is scale factor the same as error bounds in http://man.openbsd.org/dc http://man.openbsd.org/dc ?
    - nn3 7y ago
      I believe it's just the number of digits when the printing cuts off
- bloomer 7y ago
  The default Android calculator app by Hans Boehm (developer of the Boehm Garbage Collector as well) does this by using the computable real numbers. https://dl.acm.org/doi/10.1145/2911981 https://dl.acm.org/doi/10.1145/2911981 Provides a good overview of how it works and perms website has more information. What’s cool about the computable reals implementation is you can increase the precision after the fact and it will recalculate up to that precision. Basically it memoizes the steps of the calculation and how they affect the precision.
ur-whale 7y ago
First time I hear of typo ... it's not on my standard Linux install ... where can I find the source code?
- saagarjha 7y ago
  It’s not quite the original, but Rob Pike wrote an implementation in Go: https://github.com/robpike/typo https://github.com/robpike/typo
  - TheDesolate0 7y ago
    In go? This is a job for rust! ...there goes my weekend.
- tjalfi 7y ago
  Typo was added in Research Unix V5[0] and also present in V6[1]. It isn't in V7, my guess is that it was replaced by spell. I don't think it would be difficult to get it compile on a modern system. [0] https://github.com/dspinellis/unix-history-repo/blob/Research-V5-Snapshot-Development/usr/source/s2/typo.c https://github.com/dspinellis/unix-history-repo/blob/Researc... [1] https://github.com/dspinellis/unix-history-repo/blob/Research-V6-Snapshot-Development/usr/source/s2/typo.c https://github.com/dspinellis/unix-history-repo/blob/Researc...
chmaynard 7y ago
The author is THE Doug McIlroy. It's wonderful to learn that he's still around and spreading the good word. https://en.wikipedia.org/wiki/Douglas_McIlroy https://en.wikipedia.org/wiki/Douglas_McIlroy
znpy 7y ago
It's surprising that Doug McIlroy still reads and writes about UNIX. For those who don't know, Dough is the guy that invented pipes.
- deleted 7y ago
  [deleted]
- cmroanirgo 7y ago
  Also interesting: > Originators of nearly half the list--pascal, struct, parts, eqn--were women, well beyond women's demographic share of computer science.
  - indigochill 7y ago
    In the 40s, computing was seen as primarily women's work (similar to the stereotype of switchboard operators). Into the 60s, women still comprised up to half of the computing workforce. In 84, they peaked at 37%. So demographically speaking, the ratio was not as bad as it is today. (Source: https://en.wikipedia.org/wiki/Women_in_computing https://en.wikipedia.org/wiki/Women_in_computing)
    - mhd 7y ago
      "Workforce" can be a bit misleading though. If you look at Mad Men, the women were the majority of the ad agency workforce -- in the secretary pool. An interesting statistic would be the gender ratio for people who published papers.
    - perl4ever 7y ago
      In the 40s, I don't think computing was a thing like you're implying, kind of like nuclear reactors were a very very tiny area. The explosion of computer usage and programming was more like the late 50s, when Fortran came out. Even then, I don't think people even thought in terms of a "computing workforce". Nobody majored in computers, a programmer might be a math major or might not. Male engineers had female assistants that were more than typists, but not the same as a Google SWE today either. Nor the same personalities. My source is my own family history and it makes me annoyed at the anachronistic assumptions and framework that people shoehorn historical tidbits into when discussing this topic. By the way, how can you have 50% in the 60s and then a peak at 37% later?
ur-whale 7y ago
> Originators of nearly half the list--pascal, struct, parts, eqn--were women, well beyond women's demographic share of computer science. When part of the joy of a place is that gender doesn’t matter, it’s hard to write about that joy, because calling attention to gender is the opposite of that.
- DagAgren 7y ago
  Gender doesn't matter, as long as you're male.
mjw1007 7y ago
« Typo was as surprising inside as it was outside. Its similarity measure was based on trigram frequencies, which it counted in a 26x26x26 array. The small memory, which had barely room enough for 1-byte counters, spurred a scheme for squeezing large numbers into small counters. To avoid overflow, counters were updated probabilistically to maintain an estimate of the logarithm of the count. » This sounds like something from the same family as hyperloglog Wikipedia traces that back to the Flajolet–Martin algorithm in 1984. When would typo have been written?
- saagarjha 7y ago
  I believe that the paper backing the tool came out in the 70s, but if I ask IEEE for it it gives me back an awful PDF of one page that constrains a poor scan of the cover page of the paper and nothing else so I can’t confirm whether this idea was in it. Perhaps you might find more success: https://ieeexplore.ieee.org/abstract/document/6593963 https://ieeexplore.ieee.org/abstract/document/6593963
- aasasd 7y ago
  Seems to me like a variant of a counting Bloom filter.
  - morelisp 7y ago
    Nope - counting bloom filters store an exact count of approximate events. This stores approximate counts of exact events. If you count 5 "abc" and 5 "xyz" in a counting bloom filter, it will always say you had 10 events, but might say they were 10 of the same event. If you count the same in Morris's structure, it will never confuse the two different sets, but might say one occurred 4 times and the other 8. Of course, that means you can combine the two, for the benefits and downsides of both - storing very high (and inaccurate) counts of very sparse (and maybe misattributed) event sets.
- mci 7y ago
  You are confusing the approximate counting of distinct elements (done by ingenious algorithms like hyperloglog or Flajolet–Martin) with the approximate counting of each element from a manageable set (done by incrementing the counters less and less often as they grow).
mci 7y ago
> Hidden inside WWB (writer's workbench), Lorinda Cherry's Parts annotated English text with parts of speech, based on only a smidgen of English vocabulary, orthography, and grammar. Writer's Workbench was indeed a marvel of 1970's limited-space engineering. You can see it for yourself [1]: the generic part-of-speech rules are in end.l, the exceptions in edict.c and ydict.c, and the part-of-speech disambiguator in pscan.c. Such compact, rule-based NLP has fallen out of favor these days but (shameless plug alert!) Writer's Workbench inspired my 2018 IOCCC entry that highlights passive constructions in English texts [2]. [1] https://github.com/dspinellis/unix-history-repo/tree/BSD-4_1_snap-Snapshot-Development/.ref-BSD-4/usr/src/cmd/diction https://github.com/dspinellis/unix-history-repo/tree/BSD-4_1... [2] https://ioccc.org/2018/ciura/hint.html https://ioccc.org/2018/ciura/hint.html
- ivan_ah 7y ago
  This Writer's Workbench seems really cool. The wikipedia page indicates there were quite a few more programs in the suite: https://en.wikipedia.org/wiki/Writer%27s_Workbench#Package_contents https://en.wikipedia.org/wiki/Writer%27s_Workbench#Package_c... Do you know where I could be able to find the source for all of these? I'd be interested to "revive" these utils, possibly rewriting as python or bash for easy hacking. I have some basic scrips for that, and they are already proving to be useful even though they simply call grep https://github.com/ivanistheone/writing_scripts https://github.com/ivanistheone/writing_scripts
  - dbremner 7y ago
    I don't think it has all of them, but [0] is a tarball from Research Unix that has some Writer's Workbench source code. The files are in cmd/wwb. Other tarballs may have more Writer's Workbench code but I haven't looked at them. [0] https://www.tuhs.org/Archive/Distributions/Research/Dan_Cross_v10/v10src.tar.bz2 https://www.tuhs.org/Archive/Distributions/Research/Dan_Cros...
- ahoka 7y ago
  It's very interesting that the readme calls shell scripts "runcom", an archaic name coming from the Compatible Time Sharing System. This is also the origin for rc files.
tangue 7y ago
I didn't knew about typo. One surprising unix program I discovered this year is cal (or ncal). Having a calendar in your terminal is sometimes useful and I wish I knew earlier I could type things like ncal -w 2020
- aasasd 7y ago
  Personally I prefer using the Mac app Alfred for things like that—basically a graphical one-shot terminal with autocompletion for a bunch of frequently-used stuff, in the vein of Spotlight. I whipped me up a script in Lua just so the calendar is faster than a readymade one in Python. However, Alfred needs to be bent somewhat to output content like a calendar in its suggestions.
  - butterthebuddha 7y ago
    I would like to invite you to share your setup.
- hinkley 7y ago
  A similarly flavored one I’ve always appreciated is the man page for ascii, which shows the octal, decimal, and hex values for each character in the ASCII space. Most unixes have one, although the format differs.
  - dotancohen 7y ago
    How have I been googling ASCII codes for two decades with this right under my fingertips?!? Thank you!
- pvaldes 7y ago
  or cal -3 -m 3 2020
  - ja27 7y ago
    or cal -3 -m 9 1752
sn41 7y ago
One of the useful applications of trigram-based analysis I have done is the following: for a large web-based application form where about 200000 online applications were made, we had to filter out the dummy applications - often, people would try out the interface using "aaa" as a name, for example. Since the names were mostly Indian, we did not even have a standard database of names to test against. What we did was the following: go through the entire database of all applications, and build a trigram frequency table. Then, using that trigram table, do a second pass over the database of names to find names with anomalous trigrams - if the percentage of trigram frequency anomaly in a name was too high (if the name was long enough), or the absolute number of trigrams in name was too high (if the name was short), we flagged the application and examined it manually. Using this alone, we were able to filter out a large number of dummy application forms. Of course, it is not a comprehensive tool since what forms a valid name is very vague, but I think this kind of a tool is useful and culture-neutral.
- amelius 7y ago
  The problem with these methods is that you exclude everybody that deviates from the norm. Yes, it might make your life (as a developer) a little bit easier, but it makes the lives of some of the applicants a lot harder.
  - pjc50 7y ago
    I think the "manual review" phase makes this OK, in a way that simply autobanning Mr Null from your system isn't.
    - amelius 7y ago
      I'm not sure if that would work either. Why not send them an email to verify? Or use captcha tech developed by big companies that actually has some science behind it.
      - sn41 7y ago
        These are high school students in India. Many of them are from rural backgrounds and do not have personal email accounts. From our experience, the forms are many times filled up by employees at cyber cafes who fill their own email addresses and mobile numbers instead of the students. The only reliable means of communications back to students is by a government approved website, or newspapers, or official media. (The process also has to stand up in court in case some student says that (s)he did not get the communication, and newspaper ads are a documentable evidence of communications on a specified date.) (BTW, the forms do have captchas, the spurious forms are manually filled in by mischievous/malicious/curious applicants.)
adben 7y ago
How about GNU parallel? https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/
- nunoferreira 7y ago
  wow! You just saved the future me thousands of hours.
  - saagarjha 7y ago
    I hope you don’t mind the citation nags ;)
- deleted 7y ago
  [deleted]
nunoferreira 7y ago
What about "comm" - compare two sorted files line by line. You can easily get occurrences only in file 1, in both files, only in file 2. Super powerful and saved me hours of work.
- pimlottc 7y ago
  comm is a really useful tool, with one big caveat — you must make sure your input files are all sorted the exact same way. If not, you can get unexpected results, and worse, might not even realize it. This may seem obvious, but there are many tiny ways that sorts can differ between locales, operating systems and programs (e.g. Excel), especially when dealing with Unicode. It may look the same 99% of the time, and you may not realize until later that you’ve accidentally filtered out values.
  - nunoferreira 7y ago
    Absolutely! From my experience I only use with listings from the same source with the same sort tool (mostly unix sort).
  - TomNomNom 7y ago
    My advice is to sort the files just-in-time using the shell: comm <(sort fileA.txt) <(sort fileB.txt)
  - Hello71 7y ago
    GNU comm prints a warning if either file is not sorted, unless all input lines are pairable.
- zamadatix 7y ago
  Comm is perfect for scripting usage but you might find diff better for human usage. Added bonus diff also does binary. Plus diff was in part written by the author of the linked content :).
  - TheGrassyKnoll 7y ago
vladdoster 7y ago
Crabs seems likes a really cool program. Here is a paper from Bell Labs http://lucacardelli.name/Papers/Crabs.pdf http://lucacardelli.name/Papers/Crabs.pdf
londons_explore 7y ago
> The math library for Bob Morris's variable-precision desk calculator used backward error analysis to determine the precision necessary at each step to attain the user-specified precision of the result. I wonder if compilers could do this today? If you can bound values for floating point operations, you might be able to replace them with fixed point equivalents and get a big speedup. You might also be able to replace them with ints or smaller floats if you can detect the result is rounded to an int. CPU's also have the possibility to do this since they know (some of) the actual values at runtime, and could take shortcuts with floating point calculation in places where not needed for the result.
- pavlov 7y ago
  Replacing floats with fixed point isn’t usually a meaningful optimization on modern CPUs. The FPU runs in parallel to the integer units, so you can easily end up idling the FPU while the integer units are too busy doing both the math and the necessary state management (counters, pointer arithmetic etc.) This could make sense for SIMD however, but then the problem is getting the array data in the right format before the computation — if you’re converting from float to int and back within the loop, it destroys any performance gain.
  - londons_explore 7y ago
    Fixed point uses a lot less power though, and many use cases are effectively power limited rather than functional-unit limited, since if you really do fill all functional units on every cycle you'll soon need to throttle back your clock speed... Perhaps a good example of that is video encoding, which is mostly fixed point, despite it looking like a pretty close fit for floating point maths.
    - pavlov 7y ago
      A very good point. My worldview of performance is highly biased towards “full steam ahead” desktop graphics. Video encoding is a bit of a special case though because the common algorithms are carefully designed for hardware acceleration. For most rendering, it doesn’t make sense to go out of your way to avoid the FPU.
abetusk 7y ago
For me, the most surprising one was paste. paste allowed me to interleave to streams or to split out a single stream into two columns. I'd been writing custom scripting monstrosities before I discovered paste: $ paste <( echo -e 'foo\nbar' ) <( echo -e 'baz\nqux' ) foo baz bar qux $ echo -e 'foo\nbar\nbaz\nqux' | paste - - foo bar baz qux I wonder what other unix gems I've been missing...
- bloopernova 7y ago
  Paste is quite wonderful. It adds a lot of flexibility in output and should definitely be more widely known.
- loeg 7y ago
  The related join(1) and comm(1) are oft-missed and occasionally helpful.
  - cptnapalm 7y ago
    I'm going through the AWK book and join is in there. I was unhappily surprised to find that using a tab as a delimiter is painful with join. Variations to overcome this which I've bumped into: join -t $'\t' file1 file2 (BASH only, I think.) join -t '<CTRL+v><Tab>' file1 file2 join -t "`echo '\t'`" file1 file2 Why 'join -t '\t' file1 file2' is apparently beyond the pale has me mystified.
    - loeg 7y ago
      Given that the delimiter must be a single character and the intended use is in shells, I am equally mystified as to why control codes must be passed literally. Join is smart enough to reject 2 character delimiters, but could easily grok escapes if it so chose: $ join -t '\t' join: illegal tab character specification
    - JdeBP 7y ago
      See https://unix.stackexchange.com/a/65819/5132 https://unix.stackexchange.com/a/65819/5132 for some thinking on making every individual program have an escape sequence parser, and why that syntax is not specific nor original to the Bourne Again shell.
kmstout 7y ago
sl ``` ( ) (@@) ( ) (@) () @@ O @ O @ O (@@@) ( ) (@@@@) ( ) ==== ________ ___________ _D _| |_______/ \__I_I_____===__|_________| |(_)--- | H\________/ | | =|___ ___| _________________ / | | H | | | | ||_| |_|| _| \_____A | | | H |__--------------------| [___] | =| | | ________|___H__/__|_____/[][]~\_______| | -| | |/ | |-----------I_____I [][] [] D |=======|____|________________________|_ __/ =| o |=-O=====O=====O=====O \ ____Y___________|__|__________________________|_ |/-=|___|= || || || |_____/~\___/ |_D__D__D_| |_D__D__D_| \_/ \__/ \__/ \__/ \__/ \_/ \_/ \_/ \_/ \_/ ```
- elteto 7y ago
  During college my friend and I kept an innocent prank going for a couple of years: every time one of us left our laptops unlocked the other would jump in and type 'alias ls=sl' in the prompt and then clear the screen. Good times.
  - Izkata 7y ago
    Put it in their bashrc ;)
- brunoff 7y ago
  $ cowsay "hey dude" __________ < hey dude > ---------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
  - tomcatfish 7y ago
    In your .bashrc (or whatever config file is run whenever you open up a new terminal). fortune | cowsay And voilà, you have a little quote running in your cow friend whenever you open up your terminal. Also: Does anyone have any more good fortune files? I only have the fortunes that came preinstalled on Ubuntu but would love to have more.
jawilson 7y ago
I've written a few useful scripts that everyone should have. histogram - simply counts each occurrence of a line and then outputs from highest to lowest. I've implemented this program in several different languages for learning purposes. There are practical tricks that one can apply, such as hashing any line longer than the hash itself. unique - like uniq but doesn't need to have sorted input! again, one can simply hash very long lines to save memory. datetimes - looks for numbers that might be dates (seconds or milliseconds in certain reasonable ranges) and adds the human readable version of the date as comments to the end of the line they appear in. This is probably my most used script (I work with protocol buffers that often store dates as int64s). human - reformats numbers into either powers of 2 or powers of 10. inspired obviously by the -h and -H flags from df. I'm sure I have a few more but if I can't remember them from the top of my head, then they clearly aren't quite as generally useful. Anyone else have some useful scripts like these?
- nonesuchluck 7y ago
  I work with csv files a lot. I have a short awk script which truncates/pads each column to a fixed width which I can specify at runtime. It also repeats the top column (headers) every 20 rows in a different ANSI color. I pipe the output to less -SR for interactive use so I can scan delimited data in a scrollable grid, with all columns aligned and labeled. I understand there's vim plugins for this, but, ehh.
  - JdeBP 7y ago
    There's also the likes of console-flat-table-viewer . One would have to convert the comma-separated stuff into one of the table types, but that's what Miller is for. (-: * http://jdebp.uk./Softwares/nosh/guide/commands/console-flat-table-viewer.xml http://jdebp.uk./Softwares/nosh/guide/commands/console-flat-... * http://johnkerl.org/miller/ http://johnkerl.org/miller/
- jldugger 7y ago
  > histogram Is this much different than `alias histogram="sort $1 | uniq -c | uniq -nr"` Sidenote: I started https://github.com/jldugger/moarutils https://github.com/jldugger/moarutils as a means of publishing and sharing these, but it turns out I don't even have a lot of dumb ideas. Will probably end up bookmarking this HN post for "later."
beefbroccoli 7y ago
There's a very simple system tool that clicked on about 50 simultaneous lightbulbs in my brain after only 10 minutes of playing with it: mkfifo
- ganzuul 7y ago
  I learned about it through a plugin for irssi, which lets you have a list of users in an IRC channel in a tmux window partition.
- ric2b 7y ago
  The man page for it is awful, 0 explanation of what it actually does. It allows you to create pipes as files! So you can do: `echo 'hello world' > mypipe` on one terminal and `cat < mypipe` on another! Very neat, I'm sure I'll find uses for it in the future.
jhoechtl 7y ago
Doug McIlroy is regularly active in the groff mailing list https://lists.gnu.org/archive/html/groff/ https://lists.gnu.org/archive/html/groff/
TheDesolate0 7y ago
sed & awk for life
yegle 7y ago
killall5 is the most bizarre command that I learned recently. Read manpage before trying it.
- saagarjha 7y ago
  killall5 was our favorite way to log off the machine in high school, at least until the (clearly incompetent) lab administrator removed its execute permissions because it had “kill” in its name.
- nullc 7y ago
  Sys5 killall: Bane of all regular linux administrators that also sometimes administered solaris boxes. Once after blowing up an in production database server during the day, I suffered the unfortunate difficulty of having to explain why running a command "killall" on a critical server that killed everything was an innocent mistake and that I didn't have any reason to expect it to kill everything. It's extremely difficult to not sound like a moron when explaining that you didn't expect "killall" to "kill all".
- cat199 7y ago
  +1 - pgrep and pkill should be more widely 'taught' for this reason
- NikkiA 7y ago
  So it's functionally equivalent to `kill` with PID of -1, which is what we used to use back in the old days anyway. `kill -9 -1` should only kill your user processes if you're not root.
noisy_boy 7y ago
I didn't find egrep surprising - I use it quite often. The thing I didn't know about it was that it was Al Aho's creation. I only knew about him from awk.
pvaldes 7y ago
both rename and mmv are pretty handy
winrid 7y ago
I found GNU parallel to be very useful/cool.
Torwald 7y ago
What does he man by "record structure in the file system" in re to Multics?
- tenebrisalietum 7y ago
  Unix files are simply a stream of bytes and outsource concern of file structure to userland. There's nowhere to set/get a type, no mechanism to create schema in the file like fields, lengths, constraints, etc. You simply can seek to a place in the file (if it's seekable) and read/write the bytes. What they mean is up to the programs/user/convention. Earlier filesystems were trying much more to be like databases.
- unused0 7y ago
  http://bitsavers.trailing-edge.com/pdf/honeywell/multics/AN57_userRingIO_PLM_May77.pdf http://bitsavers.trailing-edge.com/pdf/honeywell/multics/AN5...
mkchoi212 7y ago
“To avoid overflow, counters were updated probabilistically to maintain an estimate of the logarithm of the count.” Stuff like this really makes me love what the pioneers of CS did in the past. In the past, they were counting every byte and every register while nowadays, programmers make things without considering the impact it will have on the HW.
hyperpallium 7y ago
xargs parallelizes with -Pn
katharine7 7y ago
sed awk tr egrep for processing making special greeting lol converting images all are so exciting!!
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
lcall 7y ago
I have found it useful to survey the existing unix utilities (maybe every several years). I'm no genius but I find things I will use. One way of course is simply to review the names in wherever your system stores manual pages, and read (or skim) those where you don't know what they do, trying out some things, or trying to remember at least where to look it up later when ready to use it. Another is by browsing to https://man.openbsd.org/ https://man.openbsd.org/ , then put a single period (".") in the search field, optionally choose a section (and/or other system, not sure how far the coverage goes), and click the apropos button.

539 points by vitplister 7y ago

Hadn't heard of most of these. The peoples' names were more recognizable.

> struct - Brenda Baker undertook her Fortan-to-Ratfor converter against the advice of her department head--me. I thought it would likely produce an ad hoc reordering of the orginal, freed of statement numbers, but otherwise no more readable than a properly indented Fortran program. Brenda proved me wrong. She discovered that every Fortran program has a canonically structured form. Programmers preferred the canonicalized form to what they had originally written. We could've had prettier et al instead of style linters 40(+?) years ago. :(

qubex 7y ago
I had to look up ‘Ratfor’ because I’d never heard of it — apparently it’s a FORTRAN preprocessor that added control structures.
- msla 7y ago
  https://en.wikipedia.org/wiki/Ratfor https://en.wikipedia.org/wiki/Ratfor The original Ratfor brings FORTRAN 66 nearly up to the level of a respectable programming language. It turns this: if (a > b) { max = a } else { max = b } Into this: IF(.NOT.(A.GT.B))GOTO 1 MAX = A GOTO 2 1 CONTINUE MAX = B 2 CONTINUE ... with proper columnization, of course. Going the opposite direction is pretty miraculous to me. Ratfiv is the follow-on, which did the same to FORTRAN 77. However, FORTRAN 77 had control structures beyond the conditional GOTO, so Ratfiv was somewhat less necessary. FORTRAN 77 would look like this: IF (A .GT. B) THEN MAX = A ELSE MAX = B ENDIF https://en.wikipedia.org/wiki/Ratfiv https://en.wikipedia.org/wiki/Ratfiv
- davidwihl 7y ago
  One of the books that most influenced my coding was Software Tools by Brian W. Kernighan, P.J. Plauger [0]. Even though I never used Ratfor, the clear descriptions were immensely useful. [0] https://www.goodreads.com/book/show/515603.Software_Tools https://www.goodreads.com/book/show/515603.Software_Tools
  - qubex 7y ago
    I have that book. I have read snippets of it. Evidently I should read all the way through it.
mhd 7y ago
As usual, the original paper is paywalled, but it appears that this is about transforming ancient Fortran from GOTOs to structured control-flow (if-then, loops etc.). That has almost nothing to do with the spaces-and-braces nitpicking of prettier/gofmt etc.
- morelisp 7y ago
  Not "almost nothing to do" - putting programs into a readable normal form seems the natural evolution of these tools. When I started programming in the 90s, "spaces-and-braces" checking - as you say, nitpicking - was basically all we had, along with limited automatic tools to fix them (all more or less as good as `M-x indent-region`). If you were lucky and in a widely-used language you could cobble together compiler warnings, lint, and a few other tools to also get warnings about legacy interfaces (gets), dangerous practices (ignoring error codes), and unusual structure (shadowed variables, loop conditions that seemed impossible). Today we finally have considerably better tools that don't just check if you match a style guide but do a full reformat (not nitpicking, but doing it for you) and linters that can enforce 'deeper' structural demands, sometimes with automatic fixes. But 40 years ago we had tools to completely restructure programs to a normalized form, and the practical experience to know programmers found this form preferable! And like so many things in our field, 10-20 years later we had to rediscover it, painfully, all over again. Probably because today's programmers think source-to-source Fortran/Ratfor translation has "almost nothing to do" with the challenges facing them today.
  - TheDesolate0 7y ago
    I _still_ do this today with new (to me) code bases! By doing this I really read the code, and really get to understand what the previous programmer was doing. Also something I took from Asimov's foundation series, code that doesn't look right, doesn't run right. I know, not really; compiler gives no fucks, but I'm not a compiler however, and GCC error messages (Clang too!) are still about as useful as a hot bikini wax is to a walrus. This was one of the features that made me really fall in love with Emacs back in the day! I could set it up to force my style requirements, then even yanked (pasted) code would be proper (mostly) and I couldn't fat finger my code to death. My only emacs complaint is lisp. I get it, I just don't like it. I'll take fortran 77 over lisp any day (not 66 or before, tho. I'm not that crazy). So, sorry mr(s) moar lisp.

tannhaeuser 7y ago

What's surprising about eqn, dc, and egrep? I'm using the latter two all the time, and have used eqn (+troff/groff and even tbl and pic) in the 1990's for manuals and as late as (early) 2000's to typeset math-heavy course material. Not nearly as feature-rich as TeX/LaTeX, but much more approachable for casual math, with DSLs for typesetting equations, tables, and diagrams/graphs. I was delighted to see that GNU had a full suite of roff/troff drop-in replacements (which I later learned was implemented by James Clark, of SGML and, recently, Ballerina fame).

saagarjha 7y ago
The algorithms behind them.
Mediterraneo10 7y ago
I had never heard of eqn and was surprised to find that the binary is still there on my Linux box. With regard to roff in general, when I got into Linux-based typesetting around the turn of the millennium, that was already seen as antiquated tech, superseded by LaTeX which was undergoing a frenzy of development and improvement around that time. So, anyone under the age of 30 will probably be hearing of such *roff stuff for the first time (and sadly even familiarity with LaTeX has waned).
- tannhaeuser 7y ago
  Ok I'm probably showing my age here then :) Back in the 1980 and 1990s, the roff suite, and most definitely egrep and classic Thompson DFA construction and DFA->NFA conversion was definitely Unix folklore/taught in Uni. Manpages are still rendered using roff/groff today, so probably many of us are using it regularly. Whereas GNU's texinfo has matured less well I'd say, or wasn't even very useful in practice to begin with due to lack of content. I'm also using TeX/LaTex, but it's still a programming language whereas roff/eqn etc are non-Turing DSLs and renderers for particular narrow purposes. I get your point, but saying these are "antiquated" is like saying HTML is obsoleted by JavaScript.
  - 7y ago

saagarjha 7y ago

And people say theoretical computer science isn’t useful in “the real world”… I am curious about this one, though, has anyone used it? > The syntax diagnostics from the compiler made by Sue Graham's group at Berkeley were the mmost helpful I have ever seen--and they were generated automatically. At a syntax error the compiler would suggest a token that could be inserted that would allow parsing to proceed further. No attempt was made to explain what was wrong. On the surface it sounds a lot like it would produce error messages like “expected ‘;’” that most beginner programmers come to hate: was it any better than this, or was that the extent of its intelligence and everything else at the time was even worse?

thaumasiotes 7y ago
> On the surface it sounds a lot like it would produce error messages like “expected ‘;’” that most beginner programmers come to hate Do people really come to hate these? I'd expect the opposite -- that people would start off hating messages like "expected ';'", but fairly quickly become accustomed to what they almost always mean. As long as you can look at the message and have a good idea of what's wrong, it's not a bad message.
- saagarjha 7y ago
  The issue is that the solution to errors like these is often not adding a semicolon, but something else like “the compiler has no idea what is going on in this line” and the actual problem can range from something like an unbalanced delimiter, a misspelled keyword, or even a “syntactically valid-looking” (that a stupid parser, like a code highlighter or formatter, would approve of) but ultimately subtly illegal construct. With practice is usually becomes fairly easy to figure out where the actual error is, but it’s certainly not a very good experience for beginners. (And they’ll “come to hate it” because they’ll see it more than a few times before understanding how to deal with them.)
  - dottedmag 7y ago
    Pascal grammar is sufficiently different from C-alikes, so it might explain why for Pascal source code these messages are more meaningful.

ruslan 7y ago

I would add bc to the list, very useful to make occasional calculations from command line using "human readable" syntax.

bonzini 7y ago
Fun fact, the first version of bc was just a frontend to dc. It converted the structured input to dc's stack-based form and let dc do the math.
- ruslan 7y ago
  Did not know that, thanks. I searched for dc inside bc and could find a reference to /usr/bin/dc, so I think bc still is just a wrapper. % uname -a FreeBSD skyrocket 9.3-RELEASE FreeBSD 9.3-RELEASE #1: Fri Nov 27 20:28:19 UTC 2015
  - bonzini 7y ago
    GNU bc isn't though they share the bignum code, I am not surprised that the BSDs are following the older implementation more closely!

ur-whale 7y ago

The fact that dc does (or at least tries to) guarantee error bounds on the result is news to me. And if that does indeed work, that's pretty cool.

nn3 7y ago
I doubt the modern GNU or BSD versions of it that you are likely using do. Noone uses the original anymore.
- swixmix 7y ago
  Is scale factor the same as error bounds in http://man.openbsd.org/dc http://man.openbsd.org/dc ?
  - nn3 7y ago
    I believe it's just the number of digits when the printing cuts off
bloomer 7y ago
The default Android calculator app by Hans Boehm (developer of the Boehm Garbage Collector as well) does this by using the computable real numbers. https://dl.acm.org/doi/10.1145/2911981 https://dl.acm.org/doi/10.1145/2911981 Provides a good overview of how it works and perms website has more information. What’s cool about the computable reals implementation is you can increase the precision after the fact and it will recalculate up to that precision. Basically it memoizes the steps of the calculation and how they affect the precision.

ur-whale 7y ago

First time I hear of typo ... it's not on my standard Linux install ... where can I find the source code?

saagarjha 7y ago
It’s not quite the original, but Rob Pike wrote an implementation in Go: https://github.com/robpike/typo https://github.com/robpike/typo
- TheDesolate0 7y ago
  In go? This is a job for rust! ...there goes my weekend.
tjalfi 7y ago
Typo was added in Research Unix V5[0] and also present in V6[1]. It isn't in V7, my guess is that it was replaced by spell. I don't think it would be difficult to get it compile on a modern system. [0] https://github.com/dspinellis/unix-history-repo/blob/Research-V5-Snapshot-Development/usr/source/s2/typo.c https://github.com/dspinellis/unix-history-repo/blob/Researc... [1] https://github.com/dspinellis/unix-history-repo/blob/Research-V6-Snapshot-Development/usr/source/s2/typo.c https://github.com/dspinellis/unix-history-repo/blob/Researc...

chmaynard 7y ago

The author is THE Doug McIlroy. It's wonderful to learn that he's still around and spreading the good word. https://en.wikipedia.org/wiki/Douglas_McIlroy https://en.wikipedia.org/wiki/Douglas_McIlroy

znpy 7y ago

It's surprising that Doug McIlroy still reads and writes about UNIX. For those who don't know, Dough is the guy that invented pipes.

deleted 7y ago
[deleted]
cmroanirgo 7y ago
Also interesting: > Originators of nearly half the list--pascal, struct, parts, eqn--were women, well beyond women's demographic share of computer science.
- indigochill 7y ago
  In the 40s, computing was seen as primarily women's work (similar to the stereotype of switchboard operators). Into the 60s, women still comprised up to half of the computing workforce. In 84, they peaked at 37%. So demographically speaking, the ratio was not as bad as it is today. (Source: https://en.wikipedia.org/wiki/Women_in_computing https://en.wikipedia.org/wiki/Women_in_computing)
  - mhd 7y ago
    "Workforce" can be a bit misleading though. If you look at Mad Men, the women were the majority of the ad agency workforce -- in the secretary pool. An interesting statistic would be the gender ratio for people who published papers.
  - perl4ever 7y ago
    In the 40s, I don't think computing was a thing like you're implying, kind of like nuclear reactors were a very very tiny area. The explosion of computer usage and programming was more like the late 50s, when Fortran came out. Even then, I don't think people even thought in terms of a "computing workforce". Nobody majored in computers, a programmer might be a math major or might not. Male engineers had female assistants that were more than typists, but not the same as a Google SWE today either. Nor the same personalities. My source is my own family history and it makes me annoyed at the anachronistic assumptions and framework that people shoehorn historical tidbits into when discussing this topic. By the way, how can you have 50% in the 60s and then a peak at 37% later?

ur-whale 7y ago

> Originators of nearly half the list--pascal, struct, parts, eqn--were women, well beyond women's demographic share of computer science. When part of the joy of a place is that gender doesn’t matter, it’s hard to write about that joy, because calling attention to gender is the opposite of that.

DagAgren 7y ago
Gender doesn't matter, as long as you're male.

mjw1007 7y ago

« Typo was as surprising inside as it was outside. Its similarity measure was based on trigram frequencies, which it counted in a 26x26x26 array. The small memory, which had barely room enough for 1-byte counters, spurred a scheme for squeezing large numbers into small counters. To avoid overflow, counters were updated probabilistically to maintain an estimate of the logarithm of the count. » This sounds like something from the same family as hyperloglog Wikipedia traces that back to the Flajolet–Martin algorithm in 1984. When would typo have been written?

saagarjha 7y ago
I believe that the paper backing the tool came out in the 70s, but if I ask IEEE for it it gives me back an awful PDF of one page that constrains a poor scan of the cover page of the paper and nothing else so I can’t confirm whether this idea was in it. Perhaps you might find more success: https://ieeexplore.ieee.org/abstract/document/6593963 https://ieeexplore.ieee.org/abstract/document/6593963
aasasd 7y ago
Seems to me like a variant of a counting Bloom filter.
- morelisp 7y ago
  Nope - counting bloom filters store an exact count of approximate events. This stores approximate counts of exact events. If you count 5 "abc" and 5 "xyz" in a counting bloom filter, it will always say you had 10 events, but might say they were 10 of the same event. If you count the same in Morris's structure, it will never confuse the two different sets, but might say one occurred 4 times and the other 8. Of course, that means you can combine the two, for the benefits and downsides of both - storing very high (and inaccurate) counts of very sparse (and maybe misattributed) event sets.
mci 7y ago
You are confusing the approximate counting of distinct elements (done by ingenious algorithms like hyperloglog or Flajolet–Martin) with the approximate counting of each element from a manageable set (done by incrementing the counters less and less often as they grow).

mci 7y ago

> Hidden inside WWB (writer's workbench), Lorinda Cherry's Parts annotated English text with parts of speech, based on only a smidgen of English vocabulary, orthography, and grammar. Writer's Workbench was indeed a marvel of 1970's limited-space engineering. You can see it for yourself [1]: the generic part-of-speech rules are in end.l, the exceptions in edict.c and ydict.c, and the part-of-speech disambiguator in pscan.c. Such compact, rule-based NLP has fallen out of favor these days but (shameless plug alert!) Writer's Workbench inspired my 2018 IOCCC entry that highlights passive constructions in English texts [2]. [1] https://github.com/dspinellis/unix-history-repo/tree/BSD-4_1_snap-Snapshot-Development/.ref-BSD-4/usr/src/cmd/diction https://github.com/dspinellis/unix-history-repo/tree/BSD-4_1... [2] https://ioccc.org/2018/ciura/hint.html https://ioccc.org/2018/ciura/hint.html

ivan_ah 7y ago
This Writer's Workbench seems really cool. The wikipedia page indicates there were quite a few more programs in the suite: https://en.wikipedia.org/wiki/Writer%27s_Workbench#Package_contents https://en.wikipedia.org/wiki/Writer%27s_Workbench#Package_c... Do you know where I could be able to find the source for all of these? I'd be interested to "revive" these utils, possibly rewriting as python or bash for easy hacking. I have some basic scrips for that, and they are already proving to be useful even though they simply call grep https://github.com/ivanistheone/writing_scripts https://github.com/ivanistheone/writing_scripts
- dbremner 7y ago
  I don't think it has all of them, but [0] is a tarball from Research Unix that has some Writer's Workbench source code. The files are in cmd/wwb. Other tarballs may have more Writer's Workbench code but I haven't looked at them. [0] https://www.tuhs.org/Archive/Distributions/Research/Dan_Cross_v10/v10src.tar.bz2 https://www.tuhs.org/Archive/Distributions/Research/Dan_Cros...
ahoka 7y ago
It's very interesting that the readme calls shell scripts "runcom", an archaic name coming from the Compatible Time Sharing System. This is also the origin for rc files.

tangue 7y ago

I didn't knew about typo. One surprising unix program I discovered this year is cal (or ncal). Having a calendar in your terminal is sometimes useful and I wish I knew earlier I could type things like ncal -w 2020

aasasd 7y ago
Personally I prefer using the Mac app Alfred for things like that—basically a graphical one-shot terminal with autocompletion for a bunch of frequently-used stuff, in the vein of Spotlight. I whipped me up a script in Lua just so the calendar is faster than a readymade one in Python. However, Alfred needs to be bent somewhat to output content like a calendar in its suggestions.
- butterthebuddha 7y ago
  I would like to invite you to share your setup.
hinkley 7y ago
A similarly flavored one I’ve always appreciated is the man page for ascii, which shows the octal, decimal, and hex values for each character in the ASCII space. Most unixes have one, although the format differs.
- dotancohen 7y ago
  How have I been googling ASCII codes for two decades with this right under my fingertips?!? Thank you!
pvaldes 7y ago
or cal -3 -m 3 2020
- ja27 7y ago
  or cal -3 -m 9 1752

sn41 7y ago

One of the useful applications of trigram-based analysis I have done is the following: for a large web-based application form where about 200000 online applications were made, we had to filter out the dummy applications - often, people would try out the interface using "aaa" as a name, for example. Since the names were mostly Indian, we did not even have a standard database of names to test against. What we did was the following: go through the entire database of all applications, and build a trigram frequency table. Then, using that trigram table, do a second pass over the database of names to find names with anomalous trigrams - if the percentage of trigram frequency anomaly in a name was too high (if the name was long enough), or the absolute number of trigrams in name was too high (if the name was short), we flagged the application and examined it manually. Using this alone, we were able to filter out a large number of dummy application forms. Of course, it is not a comprehensive tool since what forms a valid name is very vague, but I think this kind of a tool is useful and culture-neutral.

amelius 7y ago
The problem with these methods is that you exclude everybody that deviates from the norm. Yes, it might make your life (as a developer) a little bit easier, but it makes the lives of some of the applicants a lot harder.
- pjc50 7y ago
  I think the "manual review" phase makes this OK, in a way that simply autobanning Mr Null from your system isn't.
  - amelius 7y ago
    I'm not sure if that would work either. Why not send them an email to verify? Or use captcha tech developed by big companies that actually has some science behind it.
    - sn41 7y ago
      These are high school students in India. Many of them are from rural backgrounds and do not have personal email accounts. From our experience, the forms are many times filled up by employees at cyber cafes who fill their own email addresses and mobile numbers instead of the students. The only reliable means of communications back to students is by a government approved website, or newspapers, or official media. (The process also has to stand up in court in case some student says that (s)he did not get the communication, and newspaper ads are a documentable evidence of communications on a specified date.) (BTW, the forms do have captchas, the spurious forms are manually filled in by mischievous/malicious/curious applicants.)

adben 7y ago

How about GNU parallel? https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/

nunoferreira 7y ago
wow! You just saved the future me thousands of hours.
- saagarjha 7y ago
  I hope you don’t mind the citation nags ;)
deleted 7y ago
[deleted]

nunoferreira 7y ago

What about "comm" - compare two sorted files line by line. You can easily get occurrences only in file 1, in both files, only in file 2. Super powerful and saved me hours of work.

pimlottc 7y ago
comm is a really useful tool, with one big caveat — you must make sure your input files are all sorted the exact same way. If not, you can get unexpected results, and worse, might not even realize it. This may seem obvious, but there are many tiny ways that sorts can differ between locales, operating systems and programs (e.g. Excel), especially when dealing with Unicode. It may look the same 99% of the time, and you may not realize until later that you’ve accidentally filtered out values.
- nunoferreira 7y ago
  Absolutely! From my experience I only use with listings from the same source with the same sort tool (mostly unix sort).
- TomNomNom 7y ago
  My advice is to sort the files just-in-time using the shell: comm <(sort fileA.txt) <(sort fileB.txt)
- Hello71 7y ago
  GNU comm prints a warning if either file is not sorted, unless all input lines are pairable.
zamadatix 7y ago
Comm is perfect for scripting usage but you might find diff better for human usage. Added bonus diff also does binary. Plus diff was in part written by the author of the linked content :).
- TheGrassyKnoll 7y ago

vladdoster 7y ago

Crabs seems likes a really cool program. Here is a paper from Bell Labs http://lucacardelli.name/Papers/Crabs.pdf http://lucacardelli.name/Papers/Crabs.pdf

londons_explore 7y ago

> The math library for Bob Morris's variable-precision desk calculator used backward error analysis to determine the precision necessary at each step to attain the user-specified precision of the result. I wonder if compilers could do this today? If you can bound values for floating point operations, you might be able to replace them with fixed point equivalents and get a big speedup. You might also be able to replace them with ints or smaller floats if you can detect the result is rounded to an int. CPU's also have the possibility to do this since they know (some of) the actual values at runtime, and could take shortcuts with floating point calculation in places where not needed for the result.

pavlov 7y ago
Replacing floats with fixed point isn’t usually a meaningful optimization on modern CPUs. The FPU runs in parallel to the integer units, so you can easily end up idling the FPU while the integer units are too busy doing both the math and the necessary state management (counters, pointer arithmetic etc.) This could make sense for SIMD however, but then the problem is getting the array data in the right format before the computation — if you’re converting from float to int and back within the loop, it destroys any performance gain.
- londons_explore 7y ago
  Fixed point uses a lot less power though, and many use cases are effectively power limited rather than functional-unit limited, since if you really do fill all functional units on every cycle you'll soon need to throttle back your clock speed... Perhaps a good example of that is video encoding, which is mostly fixed point, despite it looking like a pretty close fit for floating point maths.
  - pavlov 7y ago
    A very good point. My worldview of performance is highly biased towards “full steam ahead” desktop graphics. Video encoding is a bit of a special case though because the common algorithms are carefully designed for hardware acceleration. For most rendering, it doesn’t make sense to go out of your way to avoid the FPU.

abetusk 7y ago

For me, the most surprising one was paste. paste allowed me to interleave to streams or to split out a single stream into two columns. I'd been writing custom scripting monstrosities before I discovered paste: $ paste <( echo -e 'foo\nbar' ) <( echo -e 'baz\nqux' ) foo baz bar qux $ echo -e 'foo\nbar\nbaz\nqux' | paste - - foo bar baz qux I wonder what other unix gems I've been missing...

bloopernova 7y ago
Paste is quite wonderful. It adds a lot of flexibility in output and should definitely be more widely known.
loeg 7y ago
The related join(1) and comm(1) are oft-missed and occasionally helpful.
- cptnapalm 7y ago
  I'm going through the AWK book and join is in there. I was unhappily surprised to find that using a tab as a delimiter is painful with join. Variations to overcome this which I've bumped into: join -t $'\t' file1 file2 (BASH only, I think.) join -t '<CTRL+v><Tab>' file1 file2 join -t "`echo '\t'`" file1 file2 Why 'join -t '\t' file1 file2' is apparently beyond the pale has me mystified.
  - loeg 7y ago
    Given that the delimiter must be a single character and the intended use is in shells, I am equally mystified as to why control codes must be passed literally. Join is smart enough to reject 2 character delimiters, but could easily grok escapes if it so chose: $ join -t '\t' join: illegal tab character specification
  - JdeBP 7y ago
    See https://unix.stackexchange.com/a/65819/5132 https://unix.stackexchange.com/a/65819/5132 for some thinking on making every individual program have an escape sequence parser, and why that syntax is not specific nor original to the Bourne Again shell.

kmstout 7y ago

sl ``` ( ) (@@) ( ) (@) () @@ O @ O @ O (@@@) ( ) (@@@@) ( ) ==== ________ ___________ _D _| |_______/ \__I_I_____===__|_________| |(_)--- | H\________/ | | =|___ ___| _________________ / | | H | | | | ||_| |_|| _| \_____A | | | H |__--------------------| [___] | =| | | ________|___H__/__|_____/[][]~\_______| | -| | |/ | |-----------I_____I [][] [] D |=======|____|________________________|_ __/ =| o |=-O=====O=====O=====O \ ____Y___________|__|__________________________|_ |/-=|___|= || || || |_____/~\___/ |_D__D__D_| |_D__D__D_| \_/ \__/ \__/ \__/ \__/ \_/ \_/ \_/ \_/ \_/ ```

elteto 7y ago
During college my friend and I kept an innocent prank going for a couple of years: every time one of us left our laptops unlocked the other would jump in and type 'alias ls=sl' in the prompt and then clear the screen. Good times.
- Izkata 7y ago
  Put it in their bashrc ;)
brunoff 7y ago
$ cowsay "hey dude" __________ < hey dude > ---------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
- tomcatfish 7y ago
  In your .bashrc (or whatever config file is run whenever you open up a new terminal). fortune | cowsay And voilà, you have a little quote running in your cow friend whenever you open up your terminal. Also: Does anyone have any more good fortune files? I only have the fortunes that came preinstalled on Ubuntu but would love to have more.

jawilson 7y ago

I've written a few useful scripts that everyone should have. histogram - simply counts each occurrence of a line and then outputs from highest to lowest. I've implemented this program in several different languages for learning purposes. There are practical tricks that one can apply, such as hashing any line longer than the hash itself. unique - like uniq but doesn't need to have sorted input! again, one can simply hash very long lines to save memory. datetimes - looks for numbers that might be dates (seconds or milliseconds in certain reasonable ranges) and adds the human readable version of the date as comments to the end of the line they appear in. This is probably my most used script (I work with protocol buffers that often store dates as int64s). human - reformats numbers into either powers of 2 or powers of 10. inspired obviously by the -h and -H flags from df. I'm sure I have a few more but if I can't remember them from the top of my head, then they clearly aren't quite as generally useful. Anyone else have some useful scripts like these?

nonesuchluck 7y ago
I work with csv files a lot. I have a short awk script which truncates/pads each column to a fixed width which I can specify at runtime. It also repeats the top column (headers) every 20 rows in a different ANSI color. I pipe the output to less -SR for interactive use so I can scan delimited data in a scrollable grid, with all columns aligned and labeled. I understand there's vim plugins for this, but, ehh.
- JdeBP 7y ago
  There's also the likes of console-flat-table-viewer . One would have to convert the comma-separated stuff into one of the table types, but that's what Miller is for. (-: * http://jdebp.uk./Softwares/nosh/guide/commands/console-flat-table-viewer.xml http://jdebp.uk./Softwares/nosh/guide/commands/console-flat-... * http://johnkerl.org/miller/ http://johnkerl.org/miller/
jldugger 7y ago
> histogram Is this much different than `alias histogram="sort $1 | uniq -c | uniq -nr"` Sidenote: I started https://github.com/jldugger/moarutils https://github.com/jldugger/moarutils as a means of publishing and sharing these, but it turns out I don't even have a lot of dumb ideas. Will probably end up bookmarking this HN post for "later."

beefbroccoli 7y ago

There's a very simple system tool that clicked on about 50 simultaneous lightbulbs in my brain after only 10 minutes of playing with it: mkfifo

ganzuul 7y ago
I learned about it through a plugin for irssi, which lets you have a list of users in an IRC channel in a tmux window partition.
ric2b 7y ago
The man page for it is awful, 0 explanation of what it actually does. It allows you to create pipes as files! So you can do: `echo 'hello world' > mypipe` on one terminal and `cat < mypipe` on another! Very neat, I'm sure I'll find uses for it in the future.

jhoechtl 7y ago

Doug McIlroy is regularly active in the groff mailing list https://lists.gnu.org/archive/html/groff/ https://lists.gnu.org/archive/html/groff/

TheDesolate0 7y ago

sed & awk for life

yegle 7y ago

killall5 is the most bizarre command that I learned recently. Read manpage before trying it.

saagarjha 7y ago
killall5 was our favorite way to log off the machine in high school, at least until the (clearly incompetent) lab administrator removed its execute permissions because it had “kill” in its name.
nullc 7y ago
Sys5 killall: Bane of all regular linux administrators that also sometimes administered solaris boxes. Once after blowing up an in production database server during the day, I suffered the unfortunate difficulty of having to explain why running a command "killall" on a critical server that killed everything was an innocent mistake and that I didn't have any reason to expect it to kill everything. It's extremely difficult to not sound like a moron when explaining that you didn't expect "killall" to "kill all".
cat199 7y ago
+1 - pgrep and pkill should be more widely 'taught' for this reason
NikkiA 7y ago
So it's functionally equivalent to `kill` with PID of -1, which is what we used to use back in the old days anyway. `kill -9 -1` should only kill your user processes if you're not root.

noisy_boy 7y ago

I didn't find egrep surprising - I use it quite often. The thing I didn't know about it was that it was Al Aho's creation. I only knew about him from awk.

pvaldes 7y ago

both rename and mmv are pretty handy

winrid 7y ago

I found GNU parallel to be very useful/cool.

Torwald 7y ago

What does he man by "record structure in the file system" in re to Multics?

tenebrisalietum 7y ago
Unix files are simply a stream of bytes and outsource concern of file structure to userland. There's nowhere to set/get a type, no mechanism to create schema in the file like fields, lengths, constraints, etc. You simply can seek to a place in the file (if it's seekable) and read/write the bytes. What they mean is up to the programs/user/convention. Earlier filesystems were trying much more to be like databases.
unused0 7y ago
http://bitsavers.trailing-edge.com/pdf/honeywell/multics/AN57_userRingIO_PLM_May77.pdf http://bitsavers.trailing-edge.com/pdf/honeywell/multics/AN5...

mkchoi212 7y ago

“To avoid overflow, counters were updated probabilistically to maintain an estimate of the logarithm of the count.” Stuff like this really makes me love what the pioneers of CS did in the past. In the past, they were counting every byte and every register while nowadays, programmers make things without considering the impact it will have on the HW.

hyperpallium 7y ago

xargs parallelizes with -Pn

katharine7 7y ago

sed awk tr egrep for processing making special greeting lol converting images all are so exciting!!

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

lcall 7y ago

I have found it useful to survey the existing unix utilities (maybe every several years). I'm no genius but I find things I will use. One way of course is simply to review the names in wherever your system stores manual pages, and read (or skim) those where you don't know what they do, trying out some things, or trying to remember at least where to look it up later when ready to use it. Another is by browsing to https://man.openbsd.org/ https://man.openbsd.org/ , then put a single period (".") in the search field, optionally choose a section (and/or other system, not sure how far the coverage goes), and click the apropos button.