20 ms·
Adventures in /usr/bin and the likes
- minxomat 10y agoPipes are not exclusive to *nix based systems. Windows e.g. has had them for quite some time, dating back to at least DOS 7[0] from Win95. [0] - http://www.lagmonster.org/docs/DOS7/pipes.html http://www.lagmonster.org/docs/DOS7/pipes.html
- rascul 10y agoNo reference, but I'm pretty sure I remember it from DOS 5.
- LukeShu 10y agoIn early versions, DOS pipes weren't real pipes. It would run the first command and save the output to a temporary file; then once that was finished, run the second command with the file as input. I'm not sure when it gained real pipes.
- flukus 10y agoAround windows nt/95 when multi-tasking was added would be my bet.
- pjc50 10y agoWindows pipes appear to still be second-class citizens. Look at this mess: http://stackoverflow.com/questions/34504970/non-blocking-read-on-os-pipe-on-windows?noredirect=1&lq=1 http://stackoverflow.com/questions/34504970/non-blocking-rea... The eventual solution using SetNamedPipeHandleState isn't really supported: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365787(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... "Note that nonblocking mode is supported for compatibility with Microsoft LAN Manager version 2.0 and should not be used to achieve asynchronous input and output (I/O) with named pipes."
- binarycrusader 10y agoBut pipes were original and first to *nix based systems, first appearing in Version 3 of UNIX: https://en.wikipedia.org/wiki/Pipeline_(Unix)#History https://en.wikipedia.org/wiki/Pipeline_(Unix)#History
- minxomat 10y agoSo? The article mentions pipes as a distinctive feature of *nix systems. Neither it nor I claimed who had it first. The truth is piping from devices, programs and files is pretty much the same in Windows :)
- binarycrusader 10y agoI never said anyone was wrong; only pointing out the origins. I think one might be able to successfully argue that NIX-like systems and the programs executed on them are distinctively* biased towards pipes. Regardless, as you said, the point is moot.
- WildUtah 10y agoThese are named pipes, not command line pipes. Windows NT was the first version of DOS that had them and they were copied from UNIX.
- minxomat 10y agoIn Windows, specifically using |, the pipe is anonymous[0]. Named pipes can be created, but named pipes (fifos) on Windows are substantially different from those on unix. Named pipes are not relevant to the pipes my reference above introduces in Windows 95's DOS system. [0] - https://en.wikipedia.org/wiki/Anonymous_pipe https://en.wikipedia.org/wiki/Anonymous_pipe [1] - https://en.wikipedia.org/wiki/Named_pipe https://en.wikipedia.org/wiki/Named_pipe
- JdeBP 10y agoWindows NT was not a version of MS-DOS at all. This is why it had true pipes, of course. Their semantics largely came from OS/2, not from Unix.
- camtarn 10y agoWorth it just for learning that there's a "replace" command, for multi-file search-and-replace without having to remember sed syntax. Can't believe I've been running Linux for 16 years and I didn't know about that one.
- gizmo686 10y agoOn my system (Ubuntu 16.04) replace is not installed by default. It appears to be provided by the following packages: mysql-server-5.7 mariadb-server-10.0 percona-server-server-5.6 percona-xtradb-cluster-server-5.6 I downloaded mysql to check, and it appears that the replace provided is as described in the article; I assume the others are as well.
- voltagex_ 10y agoI'm not sure how I feel about installing something as big as mysql-server just to get one utility. Unfortunately it seems to pull in some MySQL/MariaDB specific code - https://github.com/MariaDB/server/blob/bb2c1a52c61706dde8c525a8887f2d364c0db1eb/extra/replace.c https://github.com/MariaDB/server/blob/bb2c1a52c61706dde8c52...
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- Sophira 10y agoIt's also worth noting that in MySQL, the replace command is deprecated in 5.7.18 and removed in MySQL 8. [0] I don't know its status in MariaDB. [0] https://dev.mysql.com/doc/refman/5.7/en/replace-utility.html https://dev.mysql.com/doc/refman/5.7/en/replace-utility.html
- kevhito 10y agoI was tempted to poke a little fun at you for having trouble with sed -- among my peers, 's/foo/bar/g' used pretty much daily, even just in informal emails/conversations. But I'm glad I looked at the list first, because I found this wonder: `lstopo --of txt`. I'm going to use this in class next week, I can't believe I've never seen that before.
- pjungwir 10y agoWow this list is tremendous! It's fun to still discover new commands after all these years. I actually used `taskset -c` just recently, when I was running lots of sidekiq processes and wanted to make sure they all used different cores. It helped me get full utilization out of the box, whereas before I would frequently see some sidekiqs competing for a core while other cores sat idle.
- rdtsc 10y agoBe careful with taskset and chrt, assigning realtime priority to processes and pinning on them on CPUs might lead to un-expected behavior, priority inversion or your system locking down (say if you pgrep some kernel tasks/threads in there as well by accident, I've done before).
- pjungwir 10y agoThank you for the warning! I can see how that could happen with chrt. I can't work out how it could happen with just taskset. Are you saying to be careful when combining them? That would make sense to me. Also I can't figure out what pgrap is. Do you mean pgrep? I think you must mean something else, because I don't see what pgrep has to do with priority inversion. But it shares a manpage with pkill. . . . I don't do real-time work, but I've always been curious to know more about it. And priority inversion seems like an interesting and challenging problem.
- rdtsc 10y ago> Also I can't figure out what pgrap is. Do you mean pgrep? Sorry meant pgrep. > I don't see what pgrep has to do with priority inversion. Grepping processes doesn't have anything to do with priority inversions and locking your system. But grepping processes then setting them to realtime priority sched_fifo 99 or could wreck havok > I don't do real-time work, but I've always been curious to know more about it. And priority inversion seems like an interesting and challenging problem. It can be fun. It is kind of a separate world on its own. Interestingly most of the popular OS-es by default are tuned for throughput not low latency. So configuring and getting everything just right is an interesting challenge (sometimes involving applying kernel patches, although lately that might not be necessary as some have been mainlined).
- gravypod 10y agoAfter reading this I'm reminded that if an IDE just supported replacing highlighted text after it was piped through a command it would have a lot going for it. A lot of these are features some that IDEs have been lacking for years yet they've been hiding away in /bin.
- tambourine_man 10y agoFWIW, you can do that in Vim. Select text then: :'<,'>!command
- my_ghola 10y agoAnd `%!command` to replace the whole file. That's how editing hexadecimal in vim works. `%!xxd` to turn it, then `%!xxd -r` to turn it back. Another useful one is `%!jq .` to pretty print a json file.
- susam 10y agoI use this often. I usually enter `:%!xxd -g1` to show each byte value separately instead of showing two bytes together. $ printf "\x00\x01\x02\x03" | xxd 0000000: 0001 0203 $ printf "\x00\x01\x02\x03" | xxd -g1 0000000: 00 01 02 03 While editing binary files in Vim, one needs to be careful about setting the binary option before editing a binary file or opening the binary file with `-b` option. For example, printf "\x00\x01\x02\x03" > a.bin vim a.bin :%!xxd -g1 would display 0000000: 00 01 02 03 0a ..... (The `printf` and `vim` commands are entered in the shell. The `:%!xxd` command is entered in Vim.) In the above example, a newline has been added by Vim where none existed in the binary file. In fact, if you then run `%!xxd -r` followed by `:w`, this additional newline would be written back to the file. Here is the right way to preserve the binary data in a binary file. printf "\x00\x01\x02\x03" > a.bin vim -b a.bin :%!xxd -g1 would display 0000000: 00 01 02 03 .... Only the four bytes in the file appear now. Alternatively, one may enter `:set bin` in Vim before editing a binary file.
- noisy_boy 10y ago
- z1mm32m4n 10y agoWhat a list of gems! Also, I noticed you can rewrite this function: # Copy an ip address from a domain # Usage cip domain.com function cip() { ip=$(resolveip -s $1) echo $ip | xsel -b echo $ip } as a oneliner without needing process substitution or echo: function cip() { resolveip -s $1 | tee /dev/fd/2 | xsel -b }
- akerl_ 10y agoI'm never 100% sure I understand FD redirection, but I think you can shorten that even further to: function cip() { resolveip -s $1 1>&2 | xsel -b } Since this'll copy stdout to stderr, and then stdout will get eaten by xsel
- tomsmeding 10y agoIt won't copy stdout to stderr; it'll move stdout to stderr. Stdout will be empty after the redirect, so no, thos won't work :)
- akerl_ 10y agoHave you tried it out? It worked on my system.
- IsmaOlvey 10y agoThe parent is correct, the given command redirects stdout to stderr, it's not copied (so stdout will be empty).
- akerl_ 10y agoAgain, have you tried it? # echo "foo" 1>&2 | sed 's/foo/bar/' foo bar Edit: I've done some more testing, and discovered that the above works on zsh, but not in bash 2nd Edit: Ahha! http://www.cs.elte.hu/zsh-manual/zsh_7.html http://www.cs.elte.hu/zsh-manual/zsh_7.html . So this is because zsh w/ the stock config (MULTIOS option enabled) will open as many outputs as you give it. So it can both copy FD 1's contents to FD 2 and to the pipe'd command.
- shakna 10y agoGoing off a comment on the article, using mainly awk to find your commands, I expanded it a little bit, as most commands I use take a first argument that changes how it works. (Aliasing this command might not be a bad idea.) history | awk '{print $1 "__-__" $2}' | sort | uniq -c | sort -rn | head -30 | awk '{gsub("__-__", " ", $2); $2} !max{max=$1;}{r="";i=s=100*$1/max;while(i-->0)r=r"#";printf "%50s %5d %s %s", $2, $1, r, "\n";}' It produces a histogram of the 30 most frequent used commands. Unsurprisingly for me, `git commit` was first. Kinda surprisingly, `apt-get` was third. Caveat: I convert spaces to the symbol sequence: __-__ and back. If your command includes that symbol sequence, it'll break something.
- zoidb 10y agoA lot of really cool stuff here! shameless plug I have a small side-project site that goes through a lot of common activities on the command line https://cmdchallenge.com https://cmdchallenge.com. Some great ideas that I think I will incorporate.
- inopinatus 10y agoISTR it was Æleen Frisch (author of Essential System Administration) who recommended that all nix admins take one day a year to read through the entire manual sections (1) and (8), which I implemented thus: $ cd /usr/bin ; man (and repeat for other bindirs) You will be amazed at what you've forgotten, and what your system can do.
- l0b0 10y ago$ cd /usr/bin ; man What manual page do you want? Is there a command to get all the man pages? I tried `man *` and the internal help, but no luck. Edit: q then Return did it. D'oh. The obvious follow-up question is how do I go back to the _previous_ man page?
- felipebueno 10y ago$ man man ;)
- l0b0 10y agoActually, I could not find anything relevant by searching it for "file" and "next". I guess it's purely the responsibility of the pager, which presumably is not part of man itself.
- deleted 10y ago[deleted]
- l0b0 10y agoHere ya go; all section 1 (user commands) man pages: man $(man -s 1 -k . | cut -d' ' -f 1 | sort | tr '\n' ' ')
- inopinatus 10y agoAh sorry that got escaped, was supposed to be cd /usr/bin ; man *
- squarefoot 10y agoI was thinking about something along this: get a random number (say dd'ing from /dev/urandom count=1 bs=2 or using shuf), then get the modulo M with number of entries in /usr/bin, then find the Mth entry of the ls, and print the man page. A file containing titles of already printed man pages can be used to exclude doubles.
- jwilk 10y agoSigh. /tmp is not your personal playground. Forget that it exists, unless you're a security expert.
- __Joker 10y agoNovice question. Why the temp folder is problematic? Some cursory google search shows up this http://security.stackexchange.com/questions/11606/what-are-the-dangers-of-storing-webserver-temp-files-in-the-tmp-folder http://security.stackexchange.com/questions/11606/what-are-t... But generally what kind of attacks /tmp folder enables ?
- JdeBP 10y agoConsider the headlined article, for starters. A simple ln -s /etc/passwd /tmp/test_at.log which any unprivileged user can quietly do, before running its examples for at, will ruin the system administrator's day just after 21:26. This is the basis for many of the problems with files in /tmp : predictable filenames written-to without care by privileged processes. It is a widespread disease that is the reason that the systemd people arrange to run many programs with PrivateTmp=true . In this particular case, root's own home directory would have made a much better private playground for the test_at.log file.
- JoachimS 10y agofwiw wc -l in macOS Sierra 10.12.3 reports 1059 entries. There are quite a few programs that are Apple specific (macbinary, appletviewer). But many of the ones talked about in the article are here too.
- jwilk 10y agoApparently "resolveip" is part of... MySQL. You may want to use "getent ahosts" instead: $ getent ahosts en.wikipedia.org 91.198.174.192 STREAM en.wikipedia.org 91.198.174.192 DGRAM 91.198.174.192 RAW
- Elrac 10y agoor simply "host" !
- floatboth 10y agoThat's an interesting command! (note: on FreeBSD, just "hosts", there's no "ahosts") I usually use drill(1) (from unbound, which is part of base on FreeBSD), or dig(1) on Linux boxes that don't have unbound. Also turns out that that replace command is also part of MySQL. LOL WTF MYSQL?!
- cel1ne 10y agoA command I often use for a nice overview of running stuff: $ netstat --numeric-ports -elp46 | sort Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name tcp 0 0 0.0.0.0:22223 0.0.0.0:* LISTEN root 12159 3370/sshd tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN root 781140 4940/nginx -g daemo tcp 0 0 0.0.0.0:4000 0.0.0.0:* LISTEN postgres 780857 4916/postgrest tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN root 13126 3537/master tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN postgres 12526 3406/postgres tcp6 0 0 ::1:5432 :::* LISTEN postgres 12525 3406/postgres tcp6 0 0 :::22223 :::* LISTEN root 12168 3370/sshd udp 0 0 0.0.0.0:123 0.0.0.0:* root 12479 3400/ntpd udp 0 0 127.0.0.1:123 0.0.0.0:* root 12485 3400/ntpd udp6 0 0 ::1:123 :::* root 12487 3400/ntpd udp6 0 0 :::123 :::* root 12480 3400/ntpd
- digi_owl 10y agoApparently netstat is one of the commands being depreciated on Linux, to be replaced by ss.
- floatboth 10y ago> date > /tmp/test_at.log will be executed at 9.26 PM on the same day well that's not guaranteed on all systems :D On FreeBSD: > at is implemented through the cron(8) daemon by calling atrun(8) every five minutes
- noisy_boy 10y agoReminds me of my sysadmin days when I was trying to learn shell scripting and would lookup man pages of random binaries in /usr/bin. Before we could bring down a box for nightly maintenance/backup, we would send a shutdown broadcast notice like: banner "Shutdown"|wall; banner "in 5 mins"|wall I found out "write" command and setup a hacky little one-to-one chat script - it was terrible and geeky but I really liked it. I'm probably forgetting some more obscure commands that I enjoyed :) PS: forgot to mention that when I told my boss about the "write" based scripts, he gloated that he had setup a quiz program which supported multiple participants using named pipes. Awesome guy :)