30 ms·
Interesting commands for the Linux shell
- Tepix 9y ago> 6. List contents of tar.gz and extract only one file > tar -ztvf file.tgz > tar -zxvf file.tgz filename Tar no longer requires the modifier for the compression when extracting an archive. So no matter if it's a .tar.gz, .tar.Z, .tar.bz2, etc you can just use "tar xvf"
- kaushalmodi 9y agoAnd the same for compression.. tar caf foo.tar.xz foo/ The extension of the file after the `f` switch tells tar what compression to use. c - compress a - auto
- otherflavors 9y agoby tar you mean gnu tar
- xearl 9y agonice collection! upvoted for "15. `namei -l`" alone, which is far too little known. a better "20. Randomize lines in file": shuf file.txt instead of cat file.txt | sort -R (sort -R sorts by hash, which is not really randomisation.)
- victorlf 9y agoNice tip! I updated the article.
- NicoJuicy 9y agoCaching problems? :P
- trapperkeeper74 9y agoAnd prefer redirection to cat. whatever < file.txt not cat file.txt | whatever Also, there is no need for - or z on GNU tar in t or x modes: tar tf whatever.tgz tar tf whatever.tar.bz2 tar tf whatever.txz ... tar xf whatever.tar.gz tar xf whatever.tar.Z ... all work just fine
- TorKlingberg 9y agoI know whatever < file.txt is slightly more efficient, but there is value is keeping things going left to right with pipes in between. It makes it easy to insert a grep or sort, or swap out a part.
- veddan 9y ago<file.txt grep foo | sort
- joombaga 9y agoOh cool. Didn't know this was possible.
- BenjiWiebe 9y agoOk, if that actually works, that's amazing. Wow.
- vram22 9y agoIt works because of the general principle that I mentioned here: https://news.ycombinator.com/item?id=15249370 https://news.ycombinator.com/item?id=15249370 The shell (not the command) is the one expanding those metacharacters, so (within limits), in: cmd < file or < file cmd where you put that piece (< file) on the line does not matter, because the actual command cmd never sees the redirection at all - it is done [1] by the time the command starts. The cmd command will only see the command line arguments (other than redirections) and options (arguments that start with a dash). Even expansion of $ and other sigils (unless quoted) happens before the command starts. [1] I.e. the shell and kernel set up the standard input of the command to be connected to the file opened for reading (in the case of < file).
- lathiat 9y ago> namei -l /path/to/file This is fantastic, a game changer for me.. I often give people a command like ls -lad /path /path/to /path/to/file Thanks!
- schoen 9y ago> (sort -R sorts by hash, which is not really randomisation.) I looked at the source code for GNU sort and what they're doing is reading 16 bytes from the system CSPRNG and then initializing an MD5 digest object with those 16 bytes of input. Then the input lines are sorted according to the hash of each line with the 16 bytes prepended. Although they should no longer use MD5 for this, I don't think we know anything about the structure of MD5 that would even allow an adversary to have any advantage above chance in distinguishing between output created this way and an output created via a different randomization method. (Edit: or distinguishing between the distribution of output created this way and the distribution of output created via another method!) The output of sort -R is different on each output and ordinarily covers the whole range of possible permutations. $ for i in $(seq 10000); do seq 6 | sort -R | sha256sum; done | sort -u | wc -l 720
- xearl 9y agoSorting by hash will always cluster lines with equal content together, whereas true randomisation won't. Eg `(seq 3; seq 3; seq 3) | sort -R`.
- stiangrindvoll 9y agoDoesn't matter how long you've been using a GNU/Linux shell, you'll always learn something new every day. Thanks for this.
- hnlmorg 9y agoFunny enough I was saying this to a work colleague yesterday when I was looking at a shell script and scratching me head wondering how it was working when a variable named $RANDOM wasn't assigned a value. After a little investigation it turned out Bash (and possibly other $SHELL's?) returns a random number with each call of the $RANDOM variable. Handy little trick.
- vacri 9y agoThere is also the inbuilt variable $SECONDS, which seconds since the instance of bash was started. Makes it really easy to do 'time elapsed' at the end of a bash script. $ echo $SECONDS 83783 $ echo $SECONDS 83784 $ echo $SECONDS 83787
- monsieurbanana 9y agoThat's nifty. $ echo "$SECONDS / 60 / 60" | bc 362.38500000000000000000
- 1024core 9y ago"bc", without the "-l" option, will give you integers only. So the response of the command above will be "362", and not "362.38500000000000000000"
- mikegerwitz 9y agoecho "$(( SECONDS / 60 / 60 ))"
- vram22 9y agoecho "scale=2; $SECONDS / 60 / 60" | bc # will give 2 digits after the decimal point, instead of bc -l which gives more digits of precision after the decimal point, than we may usually want for this particular calculation.
- a3_nm 9y agoNice! Point 23 is a bit misleading though: comm only works on sorted input files. Also, "disown -h" in Point 21 works on bash but not on zsh. Also, in Point 22, "timeout" only works if the command does not trap SIGTERM (or you have to use -k).
- sirwitti 9y agoThanks, every once in a while these lists are useful!
- cutler 9y agoCentOS 7 specifies `man rename` : rename [options] expression replacement file... The rename example fails with "rename: not enough arguments".
- jwilk 9y agoUnfortunately, there are two incompatible rename programs in the wild. :( Debian ships this: https://metacpan.org/pod/distribution/File-Rename/rename.PL https://metacpan.org/pod/distribution/File-Rename/rename.PL Most(?) other distros ship the one from util-linux: http://man7.org/linux/man-pages/man1/rename.1.html http://man7.org/linux/man-pages/man1/rename.1.html
- loudmax 9y agoDon't read it like this: rename <expression> <replacement file> ... Read it like this: rename <expression> <replacement> <file(s)...> Argument 1 is a string (not a regex), argument 2 is string to replace the old string, and arguments 3+ is a file, or list of files, or a glob. The man page includes a warning about the lack of safeguards. It is unfortunate that rename doesn't have an equivalent of mv or cp's -i flag because it's so easy to overwrite files if your expressions aren't exactly correct. On some Ubuntu systems, I've seen another version of rename that's actually a Perl script that uses regular expressions. I think that version lets you set some safeguards, which are sorely lacking in the binary version of rename that ships with CentOS.
- wyoh 9y agoIt's available in Fedora under the name "prename". I've asked the maintener to make it available for EPEL 7, will this work for you?
- hnlmorg 9y agoSome handy tips there but I would recommend changing some of the `find` examples: find . -exec echo {} \; # One file by line You don't need to execute echo to do that as find will output by default anyway. There is also a `-print` flag if you wish to force `find` to output. find . -exec echo {} \+ # All in the same line This is think is a dangerous example because any files with spaces will look like two files instead of one delimited by space. Lastly, in both the above examples you're returning files and directories rather than just files. If you wanted to exclude directories then use the `-type f` flag to specify files: find . -type f ... (equally you could specify only directories with `-type d`) Other `find` tips I've found useful that might be worth adding: * You don't need to specify the source directory in GNU find (you do on FreeBSD et al) so if you're lazy then `find` will default to your working directory: find -type f -name "*.txt" * You can do case insensitive named matches with `-iname` (this is also GNU specific): find -type f -iname "invoice*"
- clarry 9y ago> * You can do case insensitive named matches with `-iname` (this is also GNU specific): > > find -type f -iname "invoice*" This was added to OpenBSD 17 years ago. Other BSDs soon followed. Solaris & IllumOS support it too. http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/find/option.c?rev=1.11&content-type=text/x-cvsweb-markup http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/find/op...
- hnlmorg 9y agoFor some reason I recalled -iname failing on FreeBSD but I've just logged onto some devs boxes (not that I didn't trust your post!) and it seems you're right as the option is there in the man pages. Apologies for this. It just goes to show how fallible the human memory is. :-/
- pixdamix 9y agoThe only forbidden characters in a Unix filenames are '/' and '\0' Want to mess with such a script? $ touch "$(echo -e -n 'lol\nposix')"
- joosters 9y agogrep -P "\t" Not criticising the author here, as grep -P is good, but you might not also know that you can enter tabs and other codes in bash by pressing ctrl-v. So you could also type: grep "[ctrl-v]TAB"
- flachsechs 9y agothis works in vim also.
- vram22 9y agoAnd Ctrl-V does not work in GVim on Windows, but Ctrl-Q does instead. E.g. if you have a text file with mixed Unix and DOS/Windows line endings, like Ctrl-J (line feed == LF == ASCII 10) and Ctrl-M (carriage return == CR == ASCII 13) + Ctrl-J, you can search for the Ctrl-Ms with /Ctrl-Q[Enter] and for tabs with /Ctrl-Q[Tab]. When you type the Ctrl-Q, it does not show as Ctrl-Q, but as a caret (^). And you can also use the same quoted Ctrl-M (Enter) pattern in a search and replace operation to remove them globally, by replacing them with a null pattern: :%s/Ctrl-Q[Enter]//g[RealEnter] or replace each tab with 4 spaces: :%s/Ctrl-Q[Tab]/[4 spaces]/g[RealEnter] where [RealEnter] means an unquoted Enter. This is mainly useful for making such changes in a single file (being edited) at a time. For batch operations of this kind, there are many other ways to do it, such as dostounix (and unixtodos), which come built-in in many Unixen, or tr, maybe sed and awk (need to check), a simple custom command-line utility like dostounix, which can easily be written in C, Perl, Python, Ruby or any other language that supports the command-line interface (command-line arguments, reading from standard input or filenames given as command line arguments, pipes, etc.).
- l0b0 9y agoThe first alternative is still preferable, since it's actually readable. You can also do $'\t' in at least Bash (and probably Zsh).
- joosters 9y agoOh, agreed. But the ctrl-v trick is useful in general anywhere where you'd like to put a special character in an input or command.
- aembleton 9y agoMMV is the latest interesting one I've found recently: https://ss64.com/bash/mmv.html https://ss64.com/bash/mmv.html For example, I can do the following: mmv "flight.*" "flight-new.#1" and this will rename all of my files that start with flight. to flight-new. preserving the file extension. So useful, when you've got a bunch of different files with the same name but with different extensions such as html, txt and sms.
- akx 9y agoI use this `rename` (available from Homebrew as `rename`): http://plasmasturm.org/code/rename/ http://plasmasturm.org/code/rename/
- pmoriarty 9y agoCheck out qmv from renameutils.[1] Using it you can use your $EDITOR to batch-rename files. It does sanity checking like making sure you don't rename two files to the same name and lets you fix such mistakes before trying again. [1] - http://www.nongnu.org/renameutils/ http://www.nongnu.org/renameutils/
- yodsanklai 9y agoI really like the "ag" command. Very convenient to grep in a bunch of files filtered by type. Example: ag --ocaml to_string Very fast and simple syntax.
- kaushalmodi 9y agoNow try rg (ripgrep). It has a much more robust file ignore handling than ag in my experience. When I last tried, ag couldn't handle this in .ignore file: !foo.1 foo.* That would ignore all foo.* files except foo.1 in rg. Also rg is a bit faster than ag for my use cases.
- VeejayRampay 9y agoIs it possible to register new file formats in ripgrep though (say for example slim files so that I could do rg -tslim PATTERN)? Last time I checked it was somehow possible but way too complicated for so common a task.
- kaushalmodi 9y agoFrom its README: rg --type-add 'foo:*.{foo,foobar}' -tfoo bar --- You simply make the below into an alias: rg --type-add 'foo:*.{foo,foobar}' Or, if the type you need is generic that other users can also use, submit a PR. The developer was kind to except mine.. https://github.com/BurntSushi/ripgrep/pull/107/files https://github.com/BurntSushi/ripgrep/pull/107/files .. it's a trivial 1-line PR.
- ptero 9y ago> If a program eats too much memory, the swap can get filled with the rest of the memory and when you go back to normal, everything is slow. Just restart the swap partition to fix it: sudo swapoff -a; sudo swapon -a Is this true? Not impossible, but I am surprised. If true, what is this fixing? In my naive view (never studied swap management), if at the current time a page is swapped out (and by now we have more memory -- we can kill swap completely and do fine), it should get swapped in when needed next time. As there is more memory now it should not, in general, be swapped out again. If true we are exchanging a number of short delays later for a longer delay now, which to me hardly looks like a win.
- bratch 9y agoIt is useful in the case that you want a predictable and expected delay immediately, rather than unpredictable delays for an unknown length of time to come.
- ptero 9y agoI get that part. I was wondering if by doing it all at once you somehow gain significantly over doing it normally. I doubt it, but prepared to learn otherwise. If it is much quicker to swap everything in maybe it is worthwhile to expose this functionality directly instead of doing hacks like swap off/on.
- joosters 9y agoIt depends on when you next need to use the system. If it is immediately after the memory hogging code exits, there's probably not much of a win. But if you run the memory hogging program, then go to lunch, if the swapoff/swapon is triggered before you get back, you will be avoiding the delays entirely.
- BenjiWiebe 9y agoIn my experience, it makes a big difference. My swap is on a hard drive, which can handle the sequential reading of the swap quite quickly. Whereas the random accessing of it in normal use is much slower per byte.
- disconnected 9y agoDisappointingly, this list treats "yes" like a toy that just prints things over and over, and doesn't mention actual useful uses for "yes", like accepting all defaults without having to press enter. Practical example: when you are doing "make oldconfig" on the kernel, and you don't care about all those questions: yes "" | make oldconfig Or, if you prefer answering no instead: yes "n" | yourcommand Also, the author refers to watch as a "supervisor" ("supervise command" - his words). That is bad terminology. Process supervision has well defined meaning in this context, and watch isn't even close to doing it. Examples of actual supervisors are supervisord, runit, monit, s6, and of course systemd (which also does service management and, er, too much other stuff, honestly).
- skocznymroczny 9y agoUnfortunately it doesn't work when trying to connect via ssh and warning about unknown key pops up.
- hnlmorg 9y agoThat is intentional behavior on the part of `ssh`. You can still disable that prompt within `ssh` (though it's not recommended): ssh -oStrictHostKeyChecking=no 192.168.0.100 ...or by adding rules in /etc/ssh/ssh_config or ~/.ssh/config Host 192.168.0.* StrictHostKeyChecking no
- feelin_googley 9y ago"...("supervise command" - his words). That is bad terminology." "Examples of actual supervisors are..." 1999-2001 http://cr.yp.to/daemontools/supervise.html http://cr.yp.to/daemontools/supervise.html runit and s6 are copies of daemontools. Otherwise I liked your comment about use of yes.
- tych0 9y agoThe kernel also recently (4.8-ish?) introduce an `olddefconfig` command, so you don't have to mess around with yes.
- hathym 9y agocached version: https://webcache.googleusercontent.com/search?q=cache:JQ7qBa09bekJ:https://www.lopezferrando.com/30-interesting-shell-commands/+&cd=1&hl=en&ct=clnk&gl=fr https://webcache.googleusercontent.com/search?q=cache:JQ7qBa...
- abhirag 9y agoAs long as we are on the topic of Linux shell commands I would like to share a tip which has helped me, whenever you are using wildcards with rm you can first test them with ls and see the files that will be deleted and then replace the ls with rm. This along with -i and -I flags makes rm just a tad bit less scary for me. Kinda basic but hopefully somebody finds it helpful :)
- mbrock 9y agoOr echo instead of ls.
- crazybigdan 9y agoYou can also use M-? to list all the files. So typing 'rm foo* ' and pressing Alt-? (on non-OSX computers) will give you a list of what your wildcard would expand to. M-* does the same thing but actually expands the wildcards, instead of just showing what that expansion would be.
- jcranberry 9y agoStill works with ESC on OSX
- sethrin 9y agoUsing find to do that is a much better idea. The author shows this as the first examples in #9.
- vram22 9y agoAs mbrock says, you can also use echo instead of ls for that. And "echo *" serves, in a pinch, as a rudimentary ls, when the ls command has been deleted and you are trying to do some recovery of a Unix system. Similarly dd can be used to cat a file if cat is deleted: dd < file And in fact whenever you want to do some command like: cmd some_flags_and_args_and_metacharacters ... you can just replace cmd with echo first to see what that full command will expand to, before you actually run it, so you know you will get the result you want (or not). This works because the $foo and its many variants and other metacharacters are all expanded by the shell, not by the individual commands. However watch out for > file and >> file which will overwrite or append to the given file with the output of the echo command.
- sillysaurus3 9y agoThrow this in your ~/bin as a script named math: #!/bin/sh scale=4 # results will print to the 4th decimal echo "scale=$scale; $@" | bc -l Now you can do math. $ math '1+1' 2 $ math '2/3' .6666 This is especially useful in shell scripts with interpolated variables: x=10 x=`math $x - 1`
- xelxebar 9y agoAlternatively, you could use an alias: # .bashrc alias bc='bc --mathlib' and a .bcrc file: # .bcrc scale = 4 Actually, this is what my .bcrc looks like: scale = 39 k_c = 299792458 # Speed of Light k_g = 6.67384 * 10^-11 # Gravitation k_atm = 100325 # Atmospheric pressure k_h = 6.62606957 * 10^-34 # Planck's constant k_hbar = 1.054571726 * 10^-34 # H Bar k_mu = 1.256637061 * 10^-6 # Vacuum permeability k_ep = 8.854187817 * 10^-12 # Vacuum permittivity k_epsilon = 8.854187817 * 10^-12 # Vacuum permittivity k_e = 1.602176565 * 10^-19 # Elementary charge k_coulomb = 8.987551787 * 10^9 # Coulomb's constant k_me = 9.10938294 * 10^-31 # Rest mass of an electron k_mp = 1.672621777 * 10^-27 # Rest mass of a proton k_n = 6.02214129 * 10^23 # Avogadro's number k_b = 1.3806488 * 10^-23 # Boltzmann's constant k_r = 8.3144621 # Ideal gas constant k_si = 5.670373 * 10^-8 # Stefan-Boltzmann constant k_sigma = 5.670373 * 10^-8 # Stefan-Boltzmann constant k_mt = 5.97219^24 # Mass of Earth (Tierra) k_rt = 6.371 * 10^6 # Mean radius of Earth (Tierra) pi = 3.1415926535897932384626433832795028841968 # requires --mathlib define t(x) { return s(x)/c(x); } define as(x) { return 2*a(x/(1+sqrt(1-x^2))); } define ac(x) { return 2*a(sqrt(1-x^2)/(1+x)); } define at(x) { return a(x); } define csc(x) { return 1/s(x); } define sec(x) { return 1/c(x); } define cot(x) { return c(x)/s(x); }
- schoen 9y agoIt's not nearly as flexible as bc overall, but GNU units has lots more constants built-in and also trig functions. I checked and it has all of the ones that you mentioned, sometimes under slightly different names. I was surprised that e is defined the elementary charge rather than Euler's constant!
- Exuma 9y agoHere are some that I use in Pet: Description: Remove executable recursively for all files Command: chmod -x $(find . -type f) Description: List files with permissions number Command: ls -l | awk '{k=0;for(i=0;i<=8;i++)k+=((substr($1,i+2,1)~/[rwx]/) *2^(8-i));if(k)printf("%0o ",k);print}' Description: Output primary terminal colors Command: for i in {0..16}; do echo -e "\e[38;05;${i}m\\\e[38;05;${i}m"; done | column -c 80 -s ' '; echo -e "\e[m" Description: NPM list with top-level only Command: npm list --depth=0 2>/dev/null Description: Show active connections Command: netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | sed -e 's/::ffff://' |cut -d: -f1 | sort | uniq -c | sort -rn | head
- xelxebar 9y agoWhile we're here sharing random cli-fu, you might like `find`'s native ability to spawn processes. Check out the `-exec` flag: find . -type f -exec chmod -x {} \+
- schoen 9y agoThis is more correct than the previous poster's approach, because it will work with files that contain space characters, while the previous poster's version breaks in this situation. Another approach that works is find . type f -print0 | xargs -0 chmod -x although find's built-in -exec can be easier to use than xargs for constructing some kinds of command lines.
- Exuma 9y ago
- sethrin 9y agoThe ones I get a lot of use out of are: curly brace substitution: $ mkdir -p new_project/{img,js,css} mkdir -p new_project/img new_project/js new_project/css $ mv some_file.txt{,.old} mv some_file.txt some_file.txt.old Caret substitution: # systemctl status mysql.service -- snip output -- # ^status^restart systemctl restart mysql.service Global substitution (and history shortcut): $ echo "We're all mad here. I'm mad. You're mad." we're all mad here. I'm mad. You're mad. $ !!:gs/mad/HN/ we're all HN here. I'm HN. You're HN. I have a (WIP) ebook with more such tricks on GitHub if anyone is interested: https://tenebrousedge.github.io/shell_guide/ https://tenebrousedge.github.io/shell_guide/
- godelski 9y agoCurly brace expansion is one of the most useful and overlooked topic in these kind of posts. And I'm always surprised it isn't mentioned with compiling. g++ -o foo{,.cpp}
- tych0 9y agoWorth noting that for the particular case you've cited, you can just use make, even without a makefile: ~ cat test.cpp #include <iostream> using namespace std; int main() { cout << "hi\n"; return 0; } ~ stat Makefile stat: cannot stat 'Makefile': No such file or directory ~ 1 make test g++ test.cpp -o test ~ ./test hi
- skykooler 9y ago
- kleff 9y agoNever knew about readlink for files, but I use pwd -P all the time to get the real full path (no symlinks) of the current directory. Really easy to remember as well, Print Working Directory.
- kazgul 9y agoThat is not the same use case. You could use this to get the current location of the script being run (not the location where it runs). readlink -f $0
- michaelcampbell 9y agoI guess I'm just overly sensitive (and maybe English is not the poster's first language), but I cringe at "the Linux shell"... I have used ksh, bash, (a little) csh, and zsh over the years, and love the architecture that makes "the shell" just another non-special binary that's not unique to the OS. And yes, a lot of the things mentioned work in a lot of shells, but some don't, or act differently.
- testestx 9y agoI would find it a little if it were called the GNU or the GNU/Linux default shell
- leephillips 9y agoI probably should be embarrassed to admit I didn't know about much of the stuff on this page about parameter expansion: http://wiki.bash-hackers.org/syntax/pe http://wiki.bash-hackers.org/syntax/pe Bash is insane.
- chubot 9y agoIt is a horrible language design, which was impressed upon me after implementing it for my shell Oil. Here are some pathological cases I pointed out: ${####}: http://www.oilshell.org/blog/2016/10/28.html http://www.oilshell.org/blog/2016/10/28.html Each # means a different thing. ${foo//z//}: http://www.oilshell.org/blog/2016/10/29.html http://www.oilshell.org/blog/2016/10/29.html There are three different meanings of / there. "${a[@]}" -- http://www.oilshell.org/blog/2016/11/06.html http://www.oilshell.org/blog/2016/11/06.html You need 8 punctuation chars in addition to the array name to correctly interpolate it. Every other way gives you word splitting issues.
- Pete_D 9y agoNot a command as such, but I recommend skimming the readline man page[0] and trying some of the bindings out to see if you're missing out on anything. I went years without knowing about ctrl+R (search command history). [0] http://man7.org/linux/man-pages/man3/readline.3.html#EDITING_COMMANDS http://man7.org/linux/man-pages/man3/readline.3.html#EDITING...
- ghotli 9y agoctrl-r for fuzzy searching your history is all you really need to know.
- jheriko 9y ago.... or just use a modern os where all this stuff is intuitive and easy :P
- yellowapple 9y agoYou'll definitely want to check `free -m` before calling `swapoff` to make sure you really have enough memory for everything in swap, lest you want to invoke the wrath of the OOM killer.
- husamia 9y agoRelated Resource: http://www.commandlinefu.com/commands/browse http://www.commandlinefu.com/commands/browse
- vram22 9y agoNot really a single command, it's a one-liner - a pipeline, but might be interesting, not only for the one-liner but for the comments about Unix processes that ensued on my blog: UNIX one-liner to kill a hanging Firefox process: https://jugad2.blogspot.in/2008/09/unix-one-liner-to-kill-hanging-firefox.html https://jugad2.blogspot.in/2008/09/unix-one-liner-to-kill-ha...