9 ms·

GNU Parallel is the first thing I usually install on top of a standard Unix userland. It's almost a superset of xargs, with many interesting features. It depend

by nextos 8y ago

GNU Parallel is the first thing I usually install on top of a standard Unix userland. It's almost a superset of xargs, with many interesting features. It depends on perl, though. htop is also pretty much a great replacement for top. And ripgrep a great replacement for the find | xargs grep pattern. Aside from that, I'm pretty content with the Unix userland. It's remarkable how well tools have aged, thanks to being composable: doing one thing and communicating via plain text. I'm less happy with the modern CLI-ncurses userland. Tools like e.g. mutt are little silos and don't compose that well. I've migrated to emacs, where the userland is much more composable.

AnIdiotOnTheNet 8y ago
I don't feel the plain text portion has aged well at all in regards to composability. It leads to a lot of headache as the complexity of the task grows because of the in-band signaling and lack of universal format. I think it is high time the standard unix tools were replaced with modern equivalents that had a consistent naming scheme and pipelined typed object data instead of plain text.
- merlincorey 8y ago
  I think you should check out Powershell.
  - alexeldeib 8y ago
    This made me chuckle, because it's incredibly accurate. I started with bash and am still naturally more comfortable there for general purpose work. PowerShell is annoyingly verbose sometimes and has its own WTF moments, but there are waaay fewer surprises when working with complex scripts and variables.
    - brirec 8y ago
      It’s not included anywhere, but PowerShell is open source and you can install it on *nix now! Obviously the things that integrate with Windows aren’t there, but the object-oriented pipelining sure is.
      - jedieaston 8y ago
        I believe the AD and Azure tools work, and the other ones needed for remotely managing Windows Server, although I could be wrong.
  - mwj 8y ago
    Is that a joke? The syntax is worse than PHP and it doesn't run on Linux
    - CupOfJava 8y ago
      It runs on linux: https://azure.microsoft.com/en-ca/blog/powershell-is-open-sourced-and-is-available-on-linux/ https://azure.microsoft.com/en-ca/blog/powershell-is-open-so... https://docs.microsoft.com/en-us/powershell/scripting/setup/installing-powershell-core-on-linux?view=powershell-6 https://docs.microsoft.com/en-us/powershell/scripting/setup/...
  - AnIdiotOnTheNet 8y ago
    I'm actually a fan of Powershell, but unix people take it as a personal insult if you try and tell them their 70s-era tooling is inferior in some way to something designed with 30 years of hindsight.
    - h1d 8y ago
      Used PowerShell to code a small script and simply making md5 of a string is enough to make you feel nuts. The UNIX style. echo -n 'string' | md5 Compared to that obvious command, this is utter madness. You can't write a single thing without googling. $string = "string" $md5 = new-object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider $utf8 = new-object -TypeName System.Text.UTF8Encoding $hash = [System.BitConverter]::ToString($md5.ComputeHash($utf8.GetBytes($string))) $hash = $hash.ToLower() -replace '-', '' No shit the Bash on Windows is a godsend.
      - pjmlp 8y ago
        Of course, because someone wrote a md5 executable for you to call. You can do the same on PowerShell and write a pipeline just the same way.
        h1d 8y ago
        You're right that a shell alone can't calculate md5 but a separate md5 binary does it for you, but the question still stands when the common answer on the internet seems to be to write that cryptic code as that's probably the easiest way provided on PowerShell. The best alternative I could find is some community maintained PowerShell extension (with just 177 GitHub stars now), which is far better but the lack of interest in making PowerShell act more straightforward is weird. "string" | Get-Hash -Algorithm MD5 (https://github.com/Pscx/Pscx https://github.com/Pscx/Pscx) Still feels funny to read the GGP's comment. > 70s-era tooling is inferior in some way to something designed with 30 years of hindsight
        TeMPOraL 8y ago
        Still, all that stops PS from being a better shell in linuxland is some people writing PS-equivalents of all the small executables that ship with your linux distribution.
- skissane 8y ago
  If Unix pipes gained support for exchanging some kind of out-of-band signalling messages, then CLI apps could tag their output as being in a particular format, or even the two ends of a pipe could negotiate about what format to use. If sending/receiving out-of-band messages was by some new API, then it could be done in a backwards compatible way. (e.g. if other end starts reading/writing/selecting/polling/etc without trying to send/receive a control message first, then the send/receive control message API returns some error code "other end doesn't support control messages") I have suggested this before: https://news.ycombinator.com/item?id=14675847 https://news.ycombinator.com/item?id=14675847 (But I don't really care enough about the idea to try to implement it... it would need kernel changes plus enhancements to the user space tools to use it... but, hypothetically, if PTYs got this support as well as pipes, your CLI tool could mark its output as 'text/html', and then your terminal could embed a web browser right in the middle of your terminal window to display it.)
  - vram22 8y ago
    >your CLI tool could mark its output as 'text/html', and then your terminal could embed a web browser right in the middle of your terminal window to display it. Ha ha, I had dreamed up something like this in one of my wilder imaginings, a while ago: A command-line shell at which you can type pipelines, involving some regular CLI commands, but also GUI commands as components, and when the pipeline is run, those GUIs will pop up in the middle of the pipeline, allow you to interact with them, and then any data output from them will go to the next component in the pipeline :) Don't actually know if the idea makes sense or would be useful.
    - v_lisivka 8y ago
      http://www.xml.com/pub/2000/06/07/xmlterm/ http://www.xml.com/pub/2000/06/07/xmlterm/
    - chriswarbo 8y ago
      > A command-line shell at which you can type pipelines, involving some regular CLI commands, but also GUI commands as components, and when the pipeline is run, those GUIs will pop up in the middle of the pipeline, allow you to interact with them, and then any data output from them will go to the next component in the pipeline There seems to be some precedent for this sort of thing. For example, DVTM can invoke a text editor as a "filter", where the editor UI is drawn on stderr and result saved to stdout.
- jolmg 8y ago
  Plain text was chosen to interface the various programs of Unix OSes because it's the least common denominator all languages share. It also forces all tools to be composable with each other. You can take output text that was obviously not formatted for easy consumption by another program and still use all the information it outputs for input into another program. Programs that were only thought to have users handling its input and output (ncurses apps) can also be forced to be used by programs through things like the expect TCL program or Ruby's expect library. If programs used typed data, they'd still need the option to output text to present results in a format the user can understand. To do this, a negotiation protocol could be established, like skissane said. This, in my opinion, is BAD, because then there's the possibility or probability that they'll be differences in the information conveyed in the different formats. I believe that the use of plain text as the universal format of communication between programs is one of the greatest design decisions for Unix and CLI.
  - oblio 8y ago
    How hard is it to come up with a object format (more like a data structure format since I wouldn't want logic/code being passed around) and then come up with a standard text serializer for it? Not that hard, in my opinion. You'd standardize it once via an RFC and you'd be done with it. I don't think your problem is as big as you say it is. The real problem is that this pile of code we already have kind of works and it's already making trillions for its users. Changing the whole ecosystems would cost millions and millions, for only a very long term and unclear benefit. In other words, worse is better.
    - dullgiulio 8y ago
      It's not about backwards compatibility. It's about the fact that text is what we read as humans and if commands parse the same format there is only one output format to implement.
      - oblio 8y ago
        I don't believe that. Text for human consumption, in a well designed UI (and I mean even a CLI one!), should be different from text for machine consumption. Human consumption generally optimizes for characteristics almost diametrically opposed from machine consumption. Of course, who am I kidding, in real life we have some sort of crappy text interface which is half-baked both for humans and for machines. But we've been using it for almost half a century and it's too widespread to redo, so there we are, plowing through it daily.
- deleted 8y ago
  [deleted]
walterstucco 8y ago
Why not grep -r or rgrep?
CupOfJava 8y ago
I had issues using gnu parallel in the past because of --will-cite. I ended up not using it.
- ethelward 8y ago
  There is a Rust clone[0], that works seamlessly for my -- arguably simple -- use cases. [0] https://github.com/mmstick/parallel https://github.com/mmstick/parallel
- mechanical_jane 8y ago
  What problems arose? I usually just run --citation once on a new system, and never see the notice again, and I have never had the notice cause any problems as it only shows if the output is to the screen.
JoshMnem 8y ago
> Tools like e.g. mutt are little silos and don't compose that well. I've migrated to emacs, where the userland is much more composable. What do you use for reading email in Emacs?
- nextos 8y ago
  Notmuch, which is IMHO superb and totally underrated. I like it a lot due to its clever architecture. It never ever touches your email. It operates on a separate tag database. Then, it's the task of a backend to translate tag changes into maildir actions before and after syncing email. Keeping tag to actions decoupled from the GUI is extremely clever, because it allows implementing basically any email workflow you can imagine. For simple workflows, calling a one liner notmuch command is sufficient. You don't really need to implement anything. Mu4e is an alternative client to Notmuch. Quite similar to Mutt. Gnus is the other big alternative. It's quite old, and complex to configure. Besides, the codebase is overcomplicated as it tries to do email in a news-like fashion. Still, it has lots of great ideas on how to deal with email from many sources. E.g. using predictive scoring.
  - JoshMnem 8y ago
    Thanks, I'm in the process of migrating from Thunderbird to mutt/neomutt and am wondering if emacs might have advantages. I already use Emacs for Org Mode. I'm also looking for a scriptable, text-based RSS reader: https://community.codeselfstudy.com/t/text-based-rss-atom-feed-readers/992 https://community.codeselfstudy.com/t/text-based-rss-atom-fe...
    - TeMPOraL 8y ago
      > am wondering if emacs might have advantages Fresh mu/mu4e user here. The advantages over mutt/neomutt are that your e-mail now becomes the part of the same consistent UX of Emacs, and then every improvement to any of your workflows in Emacs is automatically inherited by your e-mail workflow. This depends on how much you like customizing Emacs and/or writing Emacs Lisp to solve your problems. Some practical examples include: - org-mu4e & org-notmuch will let you link directly to your e-mail messages from your org-mode files; potentially useful if you're using org-capture to quickly add TODOs and notes. - it's trivial to add any kind of template responses, template subresponses, etc. - you can make Emacs automatically do things in response to particular e-mails, or you can compose/send e-mails directly from any elisp code Emacs is a fully programmable environment with lots of existing software packages and the best interoperability story I've ever seen.
qalmakka 8y ago
Is the Perl dependency actually relevant? I've never seen a Linux distro that doesn't install Perl 5 by default, and it's also installed by default on macOS and OpenBSD. The other BSDs all have Perl in their ports, and you almost always end up installing it anyway due to the sheer amount of stuff that depends on it.

by nextos 8y ago

AnIdiotOnTheNet 8y ago

I don't feel the plain text portion has aged well at all in regards to composability. It leads to a lot of headache as the complexity of the task grows because of the in-band signaling and lack of universal format. I think it is high time the standard unix tools were replaced with modern equivalents that had a consistent naming scheme and pipelined typed object data instead of plain text.

merlincorey 8y ago
I think you should check out Powershell.
- alexeldeib 8y ago
  This made me chuckle, because it's incredibly accurate. I started with bash and am still naturally more comfortable there for general purpose work. PowerShell is annoyingly verbose sometimes and has its own WTF moments, but there are waaay fewer surprises when working with complex scripts and variables.
  - brirec 8y ago
    It’s not included anywhere, but PowerShell is open source and you can install it on *nix now! Obviously the things that integrate with Windows aren’t there, but the object-oriented pipelining sure is.
    - jedieaston 8y ago
      I believe the AD and Azure tools work, and the other ones needed for remotely managing Windows Server, although I could be wrong.
- mwj 8y ago
  Is that a joke? The syntax is worse than PHP and it doesn't run on Linux
  - CupOfJava 8y ago
    It runs on linux: https://azure.microsoft.com/en-ca/blog/powershell-is-open-sourced-and-is-available-on-linux/ https://azure.microsoft.com/en-ca/blog/powershell-is-open-so... https://docs.microsoft.com/en-us/powershell/scripting/setup/installing-powershell-core-on-linux?view=powershell-6 https://docs.microsoft.com/en-us/powershell/scripting/setup/...
- AnIdiotOnTheNet 8y ago
  I'm actually a fan of Powershell, but unix people take it as a personal insult if you try and tell them their 70s-era tooling is inferior in some way to something designed with 30 years of hindsight.
  - h1d 8y ago
    Used PowerShell to code a small script and simply making md5 of a string is enough to make you feel nuts. The UNIX style. echo -n 'string' | md5 Compared to that obvious command, this is utter madness. You can't write a single thing without googling. $string = "string" $md5 = new-object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider $utf8 = new-object -TypeName System.Text.UTF8Encoding $hash = [System.BitConverter]::ToString($md5.ComputeHash($utf8.GetBytes($string))) $hash = $hash.ToLower() -replace '-', '' No shit the Bash on Windows is a godsend.
    - pjmlp 8y ago
      Of course, because someone wrote a md5 executable for you to call. You can do the same on PowerShell and write a pipeline just the same way.
      - h1d 8y ago
        You're right that a shell alone can't calculate md5 but a separate md5 binary does it for you, but the question still stands when the common answer on the internet seems to be to write that cryptic code as that's probably the easiest way provided on PowerShell. The best alternative I could find is some community maintained PowerShell extension (with just 177 GitHub stars now), which is far better but the lack of interest in making PowerShell act more straightforward is weird. "string" | Get-Hash -Algorithm MD5 (https://github.com/Pscx/Pscx https://github.com/Pscx/Pscx) Still feels funny to read the GGP's comment. > 70s-era tooling is inferior in some way to something designed with 30 years of hindsight
        TeMPOraL 8y ago
        Still, all that stops PS from being a better shell in linuxland is some people writing PS-equivalents of all the small executables that ship with your linux distribution.
skissane 8y ago
If Unix pipes gained support for exchanging some kind of out-of-band signalling messages, then CLI apps could tag their output as being in a particular format, or even the two ends of a pipe could negotiate about what format to use. If sending/receiving out-of-band messages was by some new API, then it could be done in a backwards compatible way. (e.g. if other end starts reading/writing/selecting/polling/etc without trying to send/receive a control message first, then the send/receive control message API returns some error code "other end doesn't support control messages") I have suggested this before: https://news.ycombinator.com/item?id=14675847 https://news.ycombinator.com/item?id=14675847 (But I don't really care enough about the idea to try to implement it... it would need kernel changes plus enhancements to the user space tools to use it... but, hypothetically, if PTYs got this support as well as pipes, your CLI tool could mark its output as 'text/html', and then your terminal could embed a web browser right in the middle of your terminal window to display it.)
- vram22 8y ago
  >your CLI tool could mark its output as 'text/html', and then your terminal could embed a web browser right in the middle of your terminal window to display it. Ha ha, I had dreamed up something like this in one of my wilder imaginings, a while ago: A command-line shell at which you can type pipelines, involving some regular CLI commands, but also GUI commands as components, and when the pipeline is run, those GUIs will pop up in the middle of the pipeline, allow you to interact with them, and then any data output from them will go to the next component in the pipeline :) Don't actually know if the idea makes sense or would be useful.
  - v_lisivka 8y ago
    http://www.xml.com/pub/2000/06/07/xmlterm/ http://www.xml.com/pub/2000/06/07/xmlterm/
  - chriswarbo 8y ago
    > A command-line shell at which you can type pipelines, involving some regular CLI commands, but also GUI commands as components, and when the pipeline is run, those GUIs will pop up in the middle of the pipeline, allow you to interact with them, and then any data output from them will go to the next component in the pipeline There seems to be some precedent for this sort of thing. For example, DVTM can invoke a text editor as a "filter", where the editor UI is drawn on stderr and result saved to stdout.
jolmg 8y ago
Plain text was chosen to interface the various programs of Unix OSes because it's the least common denominator all languages share. It also forces all tools to be composable with each other. You can take output text that was obviously not formatted for easy consumption by another program and still use all the information it outputs for input into another program. Programs that were only thought to have users handling its input and output (ncurses apps) can also be forced to be used by programs through things like the expect TCL program or Ruby's expect library. If programs used typed data, they'd still need the option to output text to present results in a format the user can understand. To do this, a negotiation protocol could be established, like skissane said. This, in my opinion, is BAD, because then there's the possibility or probability that they'll be differences in the information conveyed in the different formats. I believe that the use of plain text as the universal format of communication between programs is one of the greatest design decisions for Unix and CLI.
- oblio 8y ago
  How hard is it to come up with a object format (more like a data structure format since I wouldn't want logic/code being passed around) and then come up with a standard text serializer for it? Not that hard, in my opinion. You'd standardize it once via an RFC and you'd be done with it. I don't think your problem is as big as you say it is. The real problem is that this pile of code we already have kind of works and it's already making trillions for its users. Changing the whole ecosystems would cost millions and millions, for only a very long term and unclear benefit. In other words, worse is better.
  - dullgiulio 8y ago
    It's not about backwards compatibility. It's about the fact that text is what we read as humans and if commands parse the same format there is only one output format to implement.
    - oblio 8y ago
      I don't believe that. Text for human consumption, in a well designed UI (and I mean even a CLI one!), should be different from text for machine consumption. Human consumption generally optimizes for characteristics almost diametrically opposed from machine consumption. Of course, who am I kidding, in real life we have some sort of crappy text interface which is half-baked both for humans and for machines. But we've been using it for almost half a century and it's too widespread to redo, so there we are, plowing through it daily.
deleted 8y ago
[deleted]

walterstucco 8y ago

Why not grep -r or rgrep?

CupOfJava 8y ago

I had issues using gnu parallel in the past because of --will-cite. I ended up not using it.

ethelward 8y ago
There is a Rust clone[0], that works seamlessly for my -- arguably simple -- use cases. [0] https://github.com/mmstick/parallel https://github.com/mmstick/parallel
mechanical_jane 8y ago
What problems arose? I usually just run --citation once on a new system, and never see the notice again, and I have never had the notice cause any problems as it only shows if the output is to the screen.

JoshMnem 8y ago

> Tools like e.g. mutt are little silos and don't compose that well. I've migrated to emacs, where the userland is much more composable. What do you use for reading email in Emacs?

nextos 8y ago
Notmuch, which is IMHO superb and totally underrated. I like it a lot due to its clever architecture. It never ever touches your email. It operates on a separate tag database. Then, it's the task of a backend to translate tag changes into maildir actions before and after syncing email. Keeping tag to actions decoupled from the GUI is extremely clever, because it allows implementing basically any email workflow you can imagine. For simple workflows, calling a one liner notmuch command is sufficient. You don't really need to implement anything. Mu4e is an alternative client to Notmuch. Quite similar to Mutt. Gnus is the other big alternative. It's quite old, and complex to configure. Besides, the codebase is overcomplicated as it tries to do email in a news-like fashion. Still, it has lots of great ideas on how to deal with email from many sources. E.g. using predictive scoring.
- JoshMnem 8y ago
  Thanks, I'm in the process of migrating from Thunderbird to mutt/neomutt and am wondering if emacs might have advantages. I already use Emacs for Org Mode. I'm also looking for a scriptable, text-based RSS reader: https://community.codeselfstudy.com/t/text-based-rss-atom-feed-readers/992 https://community.codeselfstudy.com/t/text-based-rss-atom-fe...
  - TeMPOraL 8y ago
    > am wondering if emacs might have advantages Fresh mu/mu4e user here. The advantages over mutt/neomutt are that your e-mail now becomes the part of the same consistent UX of Emacs, and then every improvement to any of your workflows in Emacs is automatically inherited by your e-mail workflow. This depends on how much you like customizing Emacs and/or writing Emacs Lisp to solve your problems. Some practical examples include: - org-mu4e & org-notmuch will let you link directly to your e-mail messages from your org-mode files; potentially useful if you're using org-capture to quickly add TODOs and notes. - it's trivial to add any kind of template responses, template subresponses, etc. - you can make Emacs automatically do things in response to particular e-mails, or you can compose/send e-mails directly from any elisp code Emacs is a fully programmable environment with lots of existing software packages and the best interoperability story I've ever seen.

qalmakka 8y ago

Is the Perl dependency actually relevant? I've never seen a Linux distro that doesn't install Perl 5 by default, and it's also installed by default on macOS and OpenBSD. The other BSDs all have Perl in their ports, and you almost always end up installing it anyway due to the sheer amount of stuff that depends on it.