17 ms·

Linux utils that you might not know

• umanwizard 9y ago
  As much as people make fun of him for caring so much about a seemingly minor point, I've come around to understanding RMS's frustration about people calling all of the GNU project "Linux". This sort of title drives home why: neither Linus nor anyone else on the Linux project had anything to do with any of this.
  • foolfoolz 9y ago
    and on the contrary: aside from the title, is anything in this article specific to Linux?
    • ekianjo 9y ago
      Actually not, since these GNU tools could be used on BSD as well.
      • ahoka 9y ago
        GNU tools? Original column and factor is older than the GNU project. Don't attribute everything to that bearded madman. GNU is mostly just a copycat of already free software.
        • cyphar 9y ago
          Many GNU utilities provide extensions above and beyond the incredibly limited POSIX spec or more featured BSD, Solaris, etc utilities. Though a better title for this particular post would be "POSIX" utilities. > Don't attribute everything to that bearded madman. And yet the bearded madman has done a lot more than you probably have.
  • kemitche 9y ago
    If someone wrote an article titled "Windows utils that you might not know," I would not expect it's content to be solely applications written by Microsoft.
    • flukus 9y ago
      With WSL (and cygwin) this could be windows utils you may not know.
  • snakeanus 9y ago
    I think a better title would probably be "Common Unix utils that you might not know" or "GNU Coreutils that you might not know" (as some of the things that he is using seem to be coreutils-only?)
• giis 9y ago
  There is also another command 'paste'. Example: Created two files (1.txt and 2.txt) [efg@fedori ~]$ cat 1.txt file1: line1 file1: line2 [efg@fedori ~]$ cat 2.txt file2: line1 file2: line2 In order to merge the lines of these two files we can use paste: [efg@fedori ~]$ paste 1.txt 2.txt file1: line1 file2: line1 file1: line2 file2: line2 Also check out 'man fold'
  • mynewtb 9y ago
    Also super handy to turn lines into columns. 'paste - - -' for example will take every three lines as single row.
• hellomynameis12 9y ago
  GNU utils
• MichaelBurge 9y ago
  * 'comm -3' is a quick way to do a set difference from the command line. * Newer versions of sort have an option for running sorts in parallel. If you're using an older sort, you can split the files, sort them individually with GNU parallel, and use --merge to combine them. * If you have scripts that read data files, process them, and output more files, consider using a Makefile. * tmux is a good way to leave a development session on a server that you come back to at the start of the day, or that you run long-running processes in. * Setting a soft ulimit system-wide is a good way to avoid accidentally running the machine out of memory, causing your C libraries to be paged out to disk, and usually requiring a reboot. Since it's soft, you can override it at any time. * strace and gdb can be run on just about anything to see what it's doing. If it's an interpreted language, you can often attach to a live process, call a function to invoke some interpreted code, and not have to kill & restart the process to fix a bug or oversight. Or you can attach to a Postgres worker process to get a stack trace, which often tells you why your query is slow if it runs so long you can't get an explain analyze. * Look in /proc to find tons of useful information about running processes and settings. For example, you can tell how many filehandles a process has open, and sometimes what file it corresponds to. Just yesterday, a grub update caused a server to boot the kernel with the command line replaced with 2 random characters and a newline, which would be harder to debug without /proc/cmdline. * pushd and popd are useful in shell scripts to temporarily change directories without forgetting where you were.
  • falsedan 9y ago
    > If you have scripts that read data files, process them, and output more files, consider using a Makefile. Dear sweet hypnotoad, don't use Makefiles for this. Scripts in a pipeline are perfectly well suited for ETL. They have the advantage of using the same language as the command language (Makefile is not shell, and when it differs it's a significant surprise). Plus, you can drop a script into a dir in your PATH and it will work in any project. Use make if you have a project which expensively computes assets and reusing partial outputs is possible. > pushd and popd are useful in shell scripts to temporarily change directories without forgetting where you were. I would start with cd - # go back to where I just came from Which handles most of the pushd/popd use cases (outside of writing a script). Really, there's a whole separate set of skills for effective interactive shell use & writing great bash scripts (e.g. parameter expansion and history manipulation).
• db48x 9y ago
  for extra fun try watch -t -n 1 "factor \$(date +%s)"
  • manojlds 9y ago
    I thought it started with the for for a second there.
    • shimon_e 9y ago
      for extra in fun do watch -t -n 1 "factor \$(date +%s)" done
  • qznc 9y ago
    alias primetime watch -t -n 1 "factor \$(date +%s)"
    • db48x 9y ago
      Now that's a good alias
• vertebrate 9y ago
  Why does one use `font-weight: 200` for code?
  • pacaro 9y ago
    It's to keep the page weight balanced. Prose is considered to have neutral buoyancy, but code is naturally heavier and will tend to drag the page down if that isn't compensated by a lighter font. FWIW poetry can be so light, that it can require a very heavy font weight to keep it tethered to the page.
    • vdm 9y ago
      <tips hat>
• cybersol 9y ago
  Even as a decades long 'nix user, I had never used join until recently, which joins lines of two files on a common field. I used to turn to fgrep or a Python script to solve a whole class of problems join can help solve on the command-line.
  • minhajuddin 9y ago
    I recently got to use `join`. However, without reading documentation on outer joins. I went ahead and wrote my own version: join -t$'\t' <(cat c <(comm -13 <(cut -f1 c) <(cut -f1 d) | sed -e 's/$/\t/') | sort -k1,1) <(cat d <(comm -23 <(cut -f1 c) <(cut -f1 d) | sed -e 's/$/\t/') | sort -k1,1) Then I stumbled on an easier version with using just join join -a1 -a2 -o auto f1 f2 There are really too many unexplored things on the linux command line for a typical dev.
• ivanstojic 9y ago
  It always pains me when I see people use "cat" left and right, even when they don't need it. This makes for good reading: http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html
  • partycoder 9y ago
    Many people chain ps and grep... pgrep does that
    • laumars 9y ago
      That's not a great example as `pgrep` is a little more nuanced than running `ps [options] | grep [string]`, eg you cannot use many of the same `ps` flags `pgrep` like you can with `ps`. At least with "in appropriate use of cat" (as some call it) you're literally just swapping the stdin file stream with a disk io file stream so there's no functional difference what-so-ever. I'm not saying I agree with the GP either though as most of the time complaints about "in appropriate use of cat" are just showboating. Using `cat` "inappropriately" is arguably more readable for less seasoned shell script developer and it's certainly a more logical program flow for a human to parse. ie "open file, grep for contents, do something else, etc". But it's still sometimes worth a reminder that many string processing tools can accept file input directly without the need for piping it via stdin (or the files can be redirected directly from the shell via the less than, `<`, token).
      • lathiat 9y ago
        ps | grep | grep -v grep amirite? pgrep often fails me I think mostly for apps the edit their cmdline and it needs to whole word match I think?
        • stevekemp 9y ago
          You can avoid literal matches like so: ps | grep [g]rep Though it is still a bit terrible.
      • 9y ago
• deleted 9y ago
  [deleted]
• smitherfield 9y ago
  One note: I believe `shred` won't work as expected on a journaling file system like ext4.
  • deleted 9y ago
    [deleted]
  • Scaevolus 9y ago
    No. Ext4 doesn't do data journaling by default. Even when enabled, what's written to the journal is blank data that's about to be written, not the current file's contents. 1) write to journal "I'm going to overwrite this file with this data (zeros)" 2) commit journal 3) write data to file This is typical for journaling filesystems-- step 3 can be interrupted by a crash and replayed later (by re-reading the journal). For filesystems with CoW data (ZFS, btrfs), the in-place data will probably not be overwritten.
    • LukeShu 9y ago
      Before evaluating the claims myself: The shred manual specifically claims that it is "not guaranteed to be effective" on "log-structured or journaled file systems", and specifically calls out ext3 in data=journal mode. I would assume that the concern with ext3/4 in data=journal mode is that shred does not guarantee that the records of previous writes are evicted from the journal.
      • dom0 9y ago
        In data=journal mode, data to be written is first written into the journal. Only after the journal is flushed it will be written out to the correct location. Therefore, a crash at any time is fixed by replaying the journal forwards. Note that the ext3/4 journal is a redo log, not an undo log. Old file contents are not copied into the journal on a write. Thus, I don't see why shred should be less effective in data=journal mode compared to the other journaling modes. CoW file systems are a different story. They don't allow you to overwrite physical file contents. You have to set the +C (FL_NOCOW) flag, which is, by principle, only effective for a file that does not have any contents yet. Thus, you can't set +C on an existing file and overwrite it's contents.
• brettproctor 9y ago
  Anyone here will probably enjoy checking out commandlinefu: http://commandlinefu.com http://commandlinefu.com Especially looking down the list of all time greats: http://www.commandlinefu.com/commands/browse/sort-by-votes http://www.commandlinefu.com/commands/browse/sort-by-votes
  • dabber 9y ago
    Thanks for the second link. I go there every few weeks but never thought to check out the top voted. ...In hindsight, of course... The gem I just plucked out is something I've been curious about for a while but never looked up: CTRL-X e The shell will take what you've written on the command line thus far and paste it into the editor specified by $EDITOR [then run it when saved] Similar to `fc` except you don't need to run the command before invoking the editor
    • vram22 9y ago
      Also works in vi mode (I think the default for bash is emacs mode - for editing commands, that is): Run: set -o vi once after you log in (for ksh / bash and compatible shells only, maybe, not sure about csh), or (better) put that line in your .bashrc or similar startup file, so it runs each time you log in. (I used to use "ksh -o vi" earlier, before I knew about "set -o vi" or before it existed, but in that case, it has to be the last line in your startup file, otherwise the other lines below will not run until you exit that (sub)shell.) Then, when typing a command at the command line, just press ESC then v ; it does the same as what you said. You can also do ESC :q! (in the editor, if it is vi) to quit without running the command you just edited, or save the command to another file for editing later at leisure, then quit without running it right now. In fact, "set -o vi" also enables limited editing in vi mode right on the command-line, after you press ESC - you can use the command-mode commands of vi (h, l, b, w, f, F, and more) to move around, change characters or words, can also overwrite or append or insert text, etc. You can even use / and ? and n and N to search backward and forwards in the commmand history to find (by substring) a previous command, to edit it. Once you find the right commmand, just press v. Great for productivity.
• RijilV 9y ago
  I was really happy about tac when I found out about it, same with paste and comm as already mentioned. Gotten a lot of mileage out of the various seldom used options of uniq, diff and cut too.
  • lucb1e 9y ago
    tac, but also rev. I can't think of an example right now, but that's been useful as well.
    • aidos 9y ago
      I often use rev if I need something from the end of log lines. Rev, cut field by delimiter, rev back
  • VSpike 9y ago
    I have a friend who uses `tac | tac` in pipelines to make the pipeline wait for all the data before continuing, which is an interesting hack.
    • cnvogel 9y ago
      If you install the "moreutils", there's sponge doing the same. https://joeyh.name/code/moreutils/ https://joeyh.name/code/moreutils/ https://linux.die.net/man/1/sponge https://linux.die.net/man/1/sponge
• jurgemaister 9y ago
  To shreds, you say?
• sabujp 9y ago
  factor is buggy : factor 100000000000000000000000000000000000000 100000000000000000000000000000000000000: 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 http://www.wolframalpha.com/input/?i=factor+100000000000000000000000000000000000000&dataset= http://www.wolframalpha.com/input/?i=factor+1000000000000000... also numfmt bug: $ numfmt --to=iec 999999999999999999939999999 828Y $ numfmt --to=iec 999999999999999999940000000 numfmt: value too large to be printed: '1e+27' (cannot handle values > 999Y) but 999999999999999999939999999 is ~1000YB : http://www.wolframalpha.com/input/?dataset=&i=999999999999999999939999999+bytes+to+yottabytes http://www.wolframalpha.com/input/?dataset=&i=99999999999999...
  • barrkel 9y ago
    You got the same answer from factor and Wolfram alpha; where is the bug?
    • sabujp 9y ago
      wa shows 76 prime factors, which is the not same as printing out a bunch of 2's and 5's
      • thehobgoblin 9y ago
        WolframAlpha lists the factors as 2^38 * 5^38, which is just a compact notation for 38 2s and 38 5s, totalling 76 factors. If you count the 2s and 5s in the output of factor, you get the same result of 38 each for a total of 76 factors.
      • godd2 9y ago
        100000000000000000000000000000000000000 == 10 ^ 38 == (2 * 5) ^ 38 == 2 ^ 38 * 5 ^ 38 == 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 2 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 * 5 Are 2 and 5 not prime?
• throwaway2048 9y ago
  should be noted the factor program on most systems is not accurate! https://marc.info/?l=openbsd-cvs&m=147272695616766&w=2 https://marc.info/?l=openbsd-cvs&m=147272695616766&w=2 https://marc.info/?l=openbsd-cvs&m=146826185625800&w=2 https://marc.info/?l=openbsd-cvs&m=146826185625800&w=2 Hey its part of bsdGAMES not bsdGETSHITDONE :)
• tduk 9y ago
  Don't forget the often overlooked "apropos"!
  • nerflad 9y ago
    "Google for manpages" as it's called... Especially helpful on a system without an internet connection :)
  • hawski 9y ago
    In my opinion it sucks. It is possible that I suck, but I almost never can find what I'm searching for with it. Now I found that full text search is man -K, but it searches through sources so it can also be quite useless. Is there a desktop full text search for man pages? It should use rendered man pages. It could be a fun project. Google doesn't count and I use it already.
  • dwe3000 9y ago
    My problem with apropos (and man, for that matter) is related to how at least one major distribution handles the man pages. Rather than bundling the man pages with the applications, the man pages for many applications are bundled together in one base package. When this is done regardless of whether or not the application is actually on the system, apropos and man continually return information on applications which are not on the system.
• lathiat 9y ago
  I needed column yesterday. TIL Also paste
• everybodyknows 9y ago
  Which xyz-file buried in the code base has that "abc" function I saw last month? locate /\*xyzGlobPattern | xargs grep -E '\<abc[(]'
  • sigi45 9y ago
    I'm using ack for that. Works very well.
  • vdm 9y ago
    ag -G [file pattern] [text pattern]
• sigi45 9y ago
  Show sorted list with human readable file/foldersize: du -hs * | sort -h
  • effie 9y ago
    Tha shows only some files. To list all files, use: du -hs * .* | sort -h
    • cyphar 9y ago
      Or if you (like me) hate depending on globbing semantics: $ find . -mindepth 1 -maxdepth 2 -print0 | xargs -0 -- du -hs Not that I would write it in a shell. ;)
• AstralStorm 9y ago
  And the columns line has the fan favourite useless use of cat. [0] http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html
• Tepix 9y ago
  "cal -3" is quite useful, I use it all the time. To show the week-of-year use "ncal -w". Add "-S" if you want ncal to start weeks on Sunday (crazy!).
  • kwhitefoot 9y ago
    Why crazy?
    • cyphar 9y ago
      A large proportion (if not the majority) of the world starts their calanders on Monday. While the ordering is arbitrary, I always thought of the weekend as a single block of days so it's odd to say that the week ends in the middle of the weekend (rather than at the end of the weekend).
      • coreyp_1 9y ago
        I always saw "weekend" in the same way as "book end", that is, that one is at the beginning of the week (one starting end), and one is at the conclusion of the week (the finishing end). Because our weeks come successively, the end of one is immediately followed by the beginning of the next, hence the "block" called the "weekend".
        • cyphar 9y ago
          I'd never thought of it that way. I still think it's odd because book ends aren't made of chapters of a book (and weekends are days like any other). All-in-all it's arbitrary anyway. :P https://xkcd.com/1073/ https://xkcd.com/1073/
        • kwhitefoot 9y ago
          Bookends usually come in pairs.
  • bartl 9y ago
    Use `cal -m` if you want a week to start on a Monday. AFAIK Sunday is the (US centric) default.
• Aissen 9y ago
  Just look at anything in the "moreutils" package. Just great stuff. Excerpt: - ifdata: do not parse the output of ifconfig/ip anymore. Just use this tool. - sponge: when you need to overwrite an input file at the end of pipe. sponge will wait for the pipe to end before overwriting, preventing any data loss - vidir: edit a given dir with your EDITOR. Awesome for mass renames/deletes. - ts: add timestamps to a command - parallel: C implementation of GNU parallel (in perl). Very small, very fast, just does the core (running commands in parallel), and does not try to take over xargs. There are others of course.
  • vog 9y ago
    Although these tools are great, I'm afraid these are of limited use, as these aren't preinstalled but must be explicitly installed. For personal use these are great. But if you write a shell script to be executed on different machines where you can't install anything, these tools won't be available to you. And if you don't have that requirement, i.e. you can install everything, just install appropriate Perl/Python/Ruby libraries and use a proper scripting language. (BTW, I tend to assume that Python is installed on almost all modern Unix systems by default, hence I prefer writing portable tools in Python instead of portable Shell code.)
    • malux85 9y ago
      Is that really a problem anymore? Not being able to install things was common in multi-user thin-client type systems, but these days with containerisation and VMs, if you're allowed to execute a script, you're probably allowed to install new packages No?
      • angry_octet 9y ago
        Sadly, no. It is often possible to install things locally, i.e. not using apt. The idea that everyone has sudo privs is unrealistic. A version of apt that added a per user subset would be very useful.
        • 9y ago
• jeroenjanssens 9y ago
  $ curl -s http://datascienceatthecommandline.com | pup '.sect3 > h3 text{}' | head alias awk aws bash bc bigmler body cat cd chmod
  • aktau 9y ago
    TIL about pup(1) [1], what a fantastic tool to grab some data from an HTML page. Written in Go too, so easy to deploy. That one's going in my toolchest. [1]: https://github.com/ericchiang/pup https://github.com/ericchiang/pup
• dkns 9y ago
  My favorite: basename Returns just the filename plus extensions e.g. basename a/b/c/d/foo.php returns foo.php Very useful for shell scripting.
  • deleted 9y ago
    [deleted]
  • chriswarbo 9y ago
    It can also strip off a given suffix, if it appears: $ basename a/b/c/d/foo.php .php foo The complement to basename is dirname: $ dirname a/b/c/d/foo.php a/b/c/d
  • fpoling 9y ago
    With shell one can just use "${path_var##*/}" to get the base name from the string stored in $path_var. This also works if $path_var is just a base name already without any slashes. EDIT: fixed the substitution.
    • josefx 9y ago
      > With shell one can just use "${path_var##*/}" The reason I use python whenever I need something "shell" like. Cryptic symbol salad that makes code golf fanatics green with envy.
• squarefoot 9y ago
  Mgen and Drec. No idea if they have been included in any modern distro, or there are more modern equivalents, anyway over 15 years ago I used them on Alpha hardware under both Linux, BSD and Tru64 to test a local network for missing UDP packets now and then in a strange regular pattern. By missing packets now and then I mean figures like 5 packets every 3 millions at full speed, it wasn't something one could hope to reproduce just by pinging around, therefore we needed something to leave during night hours running and writing huge logs. Turned out that it was an interrupt triggered by the video subsystem on the Alpha machines only under Tru64 that grabbed 100% of the CPU for a very short time, but enough to make the receiving task miss a few packets. Once identified the problem I was able to reproduce and predict it with 100% accuracy. https://www.nrl.navy.mil/itd/ncs/products/mgen https://www.nrl.navy.mil/itd/ncs/products/mgen ps. ...I know, I know, but sadly no way to make them change the protocol used.
• _pmf_ 9y ago
  My favorite unexpected tool: tsort for performing topological sorting (i.e. sorting a dependency graph by order of dependencies).
• alex_duf 9y ago
  I really like the command "join" as well
• xkxx 9y ago
  I like how this article's title doesn't claim that it's about utils "that you don't know". Yet I was expecting utils that have something to do with the Linux kernel, not generic *nix command-line tools.
• hprotagonist 9y ago
  afaik, `shred` doesn't do anything useful on SSDs. Beneath the OS, there's intentional indeterminacy on sector access, so instructions to write over the same sectors multiple times with garbage aren't guaranteed to actually do that.
  • sherincall 9y ago
    The reason shred writes "multiple times with garbage" is due to magnetic disks where even when overwritten, old bit values could still be extracted with advanced tools. This is not the case for SSDs, so a single pass of shred (either random or zero) is enough here. Multiple passes will just kill your SSD faster. The problem with sector access still stands, so you're not sure it's actually overwritten. You should use sfill(1) in addition to shred. sdmem(1) also, if you don't want to physically power down your machine.
    • DanBC 9y ago
      >The reason shred writes "multiple times with garbage" is due to magnetic disks where even when overwritten, old bit values could still be extracted with advanced tools. People thought that you needed multiple overwrites for traditional drives but in reality that data was gone after a single overwrite of 0. There were no "advanced techniques" that could recover the data. Some specs suggested multiple passes, but this was because they were using precautionary principle. > This is not the case for SSDs, so a single pass of shred (either random or zero) is enough here. Multiple passes will just kill your SSD faster. SSDs do weird things with the data, so if your data is important enough you should destroy the drives. There's some data tucked away in odd places.
      • marcosdumay 9y ago
        Hum, no. Some time ago you needed multiple uncorrelated overwrites to hide your data. Then as HDDs shrank you strted needing less and less passes up to today that you just need to clear it once. It's not about people thinking unrealistic things. The command was created for defending against a demonstrated attack.
• nbanks 9y ago
  Although I frequently use cal, the article mentioned cal -3 which will be useful near the end of each month. Previously I always displayed the calendar for the whole year or next month instead. Going through the core utils manual [1], I found a couple other useful looking commands I didn't know about. Previously I used awk in bash scripts primarily for printf, but apparently it's part of coreutils too. The timeout command also looks useful. 1. https://www.gnu.org/software/coreutils/manual/coreutils.html https://www.gnu.org/software/coreutils/manual/coreutils.html
• j_s 9y ago
  Tools like rdfind and rmlint breathed new life into my SSD by hardlinking duplicate files after video editing software helpfully duplicated huge videos on import. https://rdfind.pauldreik.se/ https://rdfind.pauldreik.se/ https://github.com/sahib/rmlint https://github.com/sahib/rmlint There are great GUI visualizers for disk space, like WizTree/Baobab; for the command line I've had success with ncdu. https://dev.yorhel.nl/ncdu https://dev.yorhel.nl/ncdu
• majewsky 9y ago
  Since the article mentions numfmt(1), I'm surprised that no one mentioned units(1) yet. I use it all the time to convert units. $ units '4123412312312 bytes' 'tebibytes' 4123412312312 bytes = 3.7502217 tebibytes 4123412312312 bytes = (1 / 0.26665091) tebibytes $ units "50 miles per gallon" "liters per 100 kilometers" reciprocal conversion 1 / 50 miles per gallon = 4.7042917 liters per 100 kilometers 1 / 50 miles per gallon = (1 / 0.21257185) liters per 100 kilometers
  • spleeder 9y ago
    I use it as a general purpose calculator.