7 ms·

Just look at anything in the "moreutils" package. Just great stuff. Excerpt: - ifdata: do not parse the output of ifconfig/ip anymore. Just use this tool. - s

by Aissen 9y ago

Just look at anything in the "moreutils" package. Just great stuff. Excerpt: - ifdata: do not parse the output of ifconfig/ip anymore. Just use this tool. - sponge: when you need to overwrite an input file at the end of pipe. sponge will wait for the pipe to end before overwriting, preventing any data loss - vidir: edit a given dir with your EDITOR. Awesome for mass renames/deletes. - ts: add timestamps to a command - parallel: C implementation of GNU parallel (in perl). Very small, very fast, just does the core (running commands in parallel), and does not try to take over xargs. There are others of course.

vog 9y ago
Although these tools are great, I'm afraid these are of limited use, as these aren't preinstalled but must be explicitly installed. For personal use these are great. But if you write a shell script to be executed on different machines where you can't install anything, these tools won't be available to you. And if you don't have that requirement, i.e. you can install everything, just install appropriate Perl/Python/Ruby libraries and use a proper scripting language. (BTW, I tend to assume that Python is installed on almost all modern Unix systems by default, hence I prefer writing portable tools in Python instead of portable Shell code.)
- malux85 9y ago
  Is that really a problem anymore? Not being able to install things was common in multi-user thin-client type systems, but these days with containerisation and VMs, if you're allowed to execute a script, you're probably allowed to install new packages No?
  - angry_octet 9y ago
    Sadly, no. It is often possible to install things locally, i.e. not using apt. The idea that everyone has sudo privs is unrealistic. A version of apt that added a per user subset would be very useful.
    - cyphar 9y ago
      This is actually one of the things that made me work on rootless containers (https://rootlesscontaine.rs https://rootlesscontaine.rs). It is a pity that everyone assumes you have sudo access to all of the machines you want to run code on.
    - Aissen 9y ago
      There are other ways. fakeroot/fakechroot for a poor man's "container" (I cringe inside when I say that). Cross compilation + copying binaries. Or compiling on target with an appropriate ./configure --prefix.
    - ymse 9y ago
      Nix and Guix allow unprivileged package operations. I regularly run "one-off" stuff in a `guix environment` that includes all the required packages for that command, without polluting my user (or system) profile. For example, I recently wanted to transfer a file over HTTP from one machine to another, and did not have the "python" executable in PATH. So I started it in a `guix environment` and "containerized" it (using user namespaces) just for show: guix environment --container --network --ad-hoc python -- python3 -m http.server That starts the HTTP server from the current directory, but the process can not see anything else from the "real" system.
- angry_octet 9y ago
  Why assume python? It's huge compared to bash or perl. Lots of minimal configuration operational systems won't have python. More likely than go/ruby/node though.
  - vog 9y ago
    I had the impression that more an more system/initialization scripts are written in Python, at places where shell and perl were used in the past. And having a system using Python on startup is a pretty good indicator that Python will be preinstalled. However, I see that the situation may be different in specialized distros.
  - kefka 9y ago
    If you're on RH derived systems, you're going to have a python 2.6 installed for rpm and yum to work. This bit us in our ass with centos 6.8 , when another package required python 2.7... If you install 2.7 directly, your system fails with a thousand cuts. A chroot is needed for that, unfortunately.
    - sigjuice 9y ago
      chroot is unnecessary. You just install python 2.7 in its own directory and run it from there. The system-provided python 2.6 and your desired version of python should co-exist. As a general rule, never try to replace any program or library supplied by your distribution.
    - Pete_D 9y ago
      I had this problem recently. The solution was to install Python 2.7 as a software collection: https://www.softwarecollections.org/en/scls/rhscl/python27/ https://www.softwarecollections.org/en/scls/rhscl/python27/ This installs it to /opt/rh, and you can run commands that need them with "scl enable python27 mything" or by sourcing /opt/rh/python27/enable in a script to set up PATH, etc.
  - chrisper 9y ago
    You don't need to install go. It has the runtime thing built into the compiled executable.
- Aissen 9y ago
  Most of the time, I try not to care. It's packaged everywhere, so either install it or ask the script users to do so. When you're using Python/Perl/Ruby you'll often have to use pip/cpan/rvm for dependencies, and you're back to square one.
- deleted 9y ago
  [deleted]
discreditable 9y ago
One of my favorite moreutils tools is combine[1]. It allows you to compare the contents of text files with boolean operations. For example, want to know what lines are in file1 but not file2? Use: combine file1 not file2 1. http://manpages.ubuntu.com/manpages/zesty/en/man1/combine.1.html http://manpages.ubuntu.com/manpages/zesty/en/man1/combine.1....
- bbrazil 9y ago
  You can use comm from coreutils for that too: `comm -1 file1 file2`
  - Aissen 9y ago
    Yep, but you'll have to read the man everytime to figure out if you need -1, -2 or -3, or a combination of those. combine gives you a better interface with boolean operators you know.
    - xkxx 9y ago
      The 1st column is the lines unique to the 1st file, the 2nd column is the lines unique to the 2nd file, and the 3rd one is the lines that are not unique. -1, -2, -3 allow to disable those columns. Pretty easy to remember. Ah, HN downvoting neutral comments again and I don't like it, but I have a policy of not upvoting anything out of pity.
      - Aissen 9y ago
        Ahaha, then I guess I don't use it that often. I'll try to remember this mnemonic. PS: do not worry about karma ^^
  - discreditable 9y ago
    The comm manpage[1] says that the files have to be sorted. In combine, they do not. 1. http://manpages.ubuntu.com/manpages/zesty/en/man1/comm.1.html http://manpages.ubuntu.com/manpages/zesty/en/man1/comm.1.htm...

by Aissen 9y ago

vog 9y ago

Although these tools are great, I'm afraid these are of limited use, as these aren't preinstalled but must be explicitly installed. For personal use these are great. But if you write a shell script to be executed on different machines where you can't install anything, these tools won't be available to you. And if you don't have that requirement, i.e. you can install everything, just install appropriate Perl/Python/Ruby libraries and use a proper scripting language. (BTW, I tend to assume that Python is installed on almost all modern Unix systems by default, hence I prefer writing portable tools in Python instead of portable Shell code.)

malux85 9y ago
Is that really a problem anymore? Not being able to install things was common in multi-user thin-client type systems, but these days with containerisation and VMs, if you're allowed to execute a script, you're probably allowed to install new packages No?
- angry_octet 9y ago
  Sadly, no. It is often possible to install things locally, i.e. not using apt. The idea that everyone has sudo privs is unrealistic. A version of apt that added a per user subset would be very useful.
  - cyphar 9y ago
    This is actually one of the things that made me work on rootless containers (https://rootlesscontaine.rs https://rootlesscontaine.rs). It is a pity that everyone assumes you have sudo access to all of the machines you want to run code on.
  - Aissen 9y ago
    There are other ways. fakeroot/fakechroot for a poor man's "container" (I cringe inside when I say that). Cross compilation + copying binaries. Or compiling on target with an appropriate ./configure --prefix.
  - ymse 9y ago
    Nix and Guix allow unprivileged package operations. I regularly run "one-off" stuff in a `guix environment` that includes all the required packages for that command, without polluting my user (or system) profile. For example, I recently wanted to transfer a file over HTTP from one machine to another, and did not have the "python" executable in PATH. So I started it in a `guix environment` and "containerized" it (using user namespaces) just for show: guix environment --container --network --ad-hoc python -- python3 -m http.server That starts the HTTP server from the current directory, but the process can not see anything else from the "real" system.
angry_octet 9y ago
Why assume python? It's huge compared to bash or perl. Lots of minimal configuration operational systems won't have python. More likely than go/ruby/node though.
- vog 9y ago
  I had the impression that more an more system/initialization scripts are written in Python, at places where shell and perl were used in the past. And having a system using Python on startup is a pretty good indicator that Python will be preinstalled. However, I see that the situation may be different in specialized distros.
- kefka 9y ago
  If you're on RH derived systems, you're going to have a python 2.6 installed for rpm and yum to work. This bit us in our ass with centos 6.8 , when another package required python 2.7... If you install 2.7 directly, your system fails with a thousand cuts. A chroot is needed for that, unfortunately.
  - sigjuice 9y ago
    chroot is unnecessary. You just install python 2.7 in its own directory and run it from there. The system-provided python 2.6 and your desired version of python should co-exist. As a general rule, never try to replace any program or library supplied by your distribution.
  - Pete_D 9y ago
    I had this problem recently. The solution was to install Python 2.7 as a software collection: https://www.softwarecollections.org/en/scls/rhscl/python27/ https://www.softwarecollections.org/en/scls/rhscl/python27/ This installs it to /opt/rh, and you can run commands that need them with "scl enable python27 mything" or by sourcing /opt/rh/python27/enable in a script to set up PATH, etc.
- chrisper 9y ago
  You don't need to install go. It has the runtime thing built into the compiled executable.
Aissen 9y ago
Most of the time, I try not to care. It's packaged everywhere, so either install it or ask the script users to do so. When you're using Python/Perl/Ruby you'll often have to use pip/cpan/rvm for dependencies, and you're back to square one.
deleted 9y ago
[deleted]

discreditable 9y ago

One of my favorite moreutils tools is combine[1]. It allows you to compare the contents of text files with boolean operations. For example, want to know what lines are in file1 but not file2? Use: combine file1 not file2 1. http://manpages.ubuntu.com/manpages/zesty/en/man1/combine.1.html http://manpages.ubuntu.com/manpages/zesty/en/man1/combine.1....

bbrazil 9y ago
You can use comm from coreutils for that too: `comm -1 file1 file2`
- Aissen 9y ago
  Yep, but you'll have to read the man everytime to figure out if you need -1, -2 or -3, or a combination of those. combine gives you a better interface with boolean operators you know.
  - xkxx 9y ago
    The 1st column is the lines unique to the 1st file, the 2nd column is the lines unique to the 2nd file, and the 3rd one is the lines that are not unique. -1, -2, -3 allow to disable those columns. Pretty easy to remember. Ah, HN downvoting neutral comments again and I don't like it, but I have a policy of not upvoting anything out of pity.
    - Aissen 9y ago
      Ahaha, then I guess I don't use it that often. I'll try to remember this mnemonic. PS: do not worry about karma ^^
- discreditable 9y ago
  The comm manpage[1] says that the files have to be sorted. In combine, they do not. 1. http://manpages.ubuntu.com/manpages/zesty/en/man1/comm.1.html http://manpages.ubuntu.com/manpages/zesty/en/man1/comm.1.htm...