75 ms·

There's a great opportunity to not just reimplement the GNU coreutils but to actually rethink them. The way I install a Linux system these days and then immedia

There's a great opportunity to not just reimplement the GNU coreutils but to actually rethink them. The way I install a Linux system these days and then immediately download ripgrep or fd reminds me of how the first thing I used to do on a Solaris system was install bash and the GNU coreutils. There's a rich history of doing this - BSD Unix started as a bunch of utilities and patches to the AT&T version, it only became a freestanding distro with its own kernel later on. And the GNU coreutils go way back before the Linux kernel - you just installed them on a commercial Unix to make it more usable. In both cases they went beyond just drop-in replacements and started adding lots of new features and flags. Things have been pretty static for a long time so it's good to see some innovations coming.

est31 6y ago
The GNU coreutils are the lowest common denominator of most GNU systems, and thus a lot of software targets them including the GNU extensions. Replacements like rg or fd are often not even compatible to the unix originals let alone the GNU extensions. Yes, some of the GNU tools are badly in need of UI improvements, but you'd have to abandon compatibility with a great deal of scripts. I think both a coreutils rewrite as well as some end user facing software like rg has its place on modern unix systems. I'm a very happy user of rg! But I'd like some more respect for tradition by some of those tools. For example, "fd" in a unix setting refers to file descriptors. They should rename IMO.
- adrianN 6y ago
  Not being compatible is often the point of tools like ripgrep.
  - steerablesafe 6y ago
    It's not the point of ripgrep. If ripgrep were only marginally better at common grepping tasks then it could hardly justify not being compatible. It aims to be much better though, so it can justify it well. But not being compatible is not the point or the goal of the project, I believe.
    - burntsushi 6y ago
      Yes, this is correct in some sense. I didn't set out to create an "incompatible grep." I set out to build a "better" grep, where "better" could include things like being incompatible. Of course, ripgrep is still ultimately a lot more similar to grep than it is different. Where possible, I used the same flag names and prescribed similar behavior---if it made sense. Because it's good to lean on existing experiences. It makes it easier for folks to migrate. So it's often a balancing act. But yes, POSIX compatibility is certainly a non-goal of ripgrep.
      - jftuga 6y ago
        I would love it if --crlf was enabled by default with the Windows version. This would make using ^ and $ easier.
        jpitz 6y ago
        Make a config file for ripgrep and enable it. https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#configuration-file https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#c...
      - coldtea 6y ago
        That said, thanks for making ripgrep so good. I use it 100s of times every day for 3-4 years now... (and iirc, it's the default engine in VS Code as well).
      - orra 6y ago
        Not having to specify "-rHInE" on every invocation is one killer reason to use ripgrep, over grep. It's a concrete advantage of you not having 100% compatibility as a goal. (The other killer reason is the blitzing speed.)
        ses1984 6y ago
        You can solve `-rHInE` with just an alias though.
  - thesuperbigfrog 6y ago
    If a Rust rewrite of Coreutils is not backward compatible, it is not a replacement and the current GNU Coreutils would still be needed for backward compatibility. There is a long history of newer, better tools supporting backward compatibility to be a replacement for their predecessors: - bash and zsh support backward compatibility to be replacements for sh. - vim has compatibility mode to be a replacement for vi. If the Rust implementation is not backward compatible, it should not be called "Coreutils".
    - GoblinSlayer 6y ago
      What you call "Coreutils" should be called GNU/Coreutils.
      - thesuperbigfrog 6y ago
        The Rust implementation that we are discussing is MIT licensed and as far as I know not part of GNU. I do not believe that "Coreutils" is trademarked by GNU. There are other projects that use the name "Coreutils" that are not part of GNU: https://github.com/DiegoMagdaleno/BSDCoreUtils https://github.com/DiegoMagdaleno/BSDCoreUtils
        entropicdrifter 6y ago
        I think the person you're replying to was just memeing on the Richard Stallman essay that has the line, "What you call Linux should really be called GNU/Linux"
        audience_mem 6y ago
        I was sad to find out that rms apparently didn't even say it.
        GoblinSlayer 6y ago
        >I do not believe that "Coreutils" is trademarked by GNU. That's exactly why GNU sort of coreutils should be called GNU/Coreutils. The comment said that GNU/Coreutils are equivalent to Coreutils, I said they aren't.
- throwaway894345 6y ago
  > you'd have to abandon compatibility with a great deal of scripts. Having tried to reproducibly package various software, the C layer (and especially the coreutils layer) are the absolute worst, and I wouldn't shed a tear if we started afresh with something more holistically designed.
  - asguy 6y ago
    Absolute worst compared to what? Have you tried packaging software on Windows? If you have something better in mind, please implement it. It’d get used.
    - throwaway894345 6y ago
      "Worst" compared to packaging software in other languages. > If you have something better in mind, please implement it. It’d get used. Nonsense. It's not a dearth of better options that causes C folks to clutch their autotools and bash scripts and punting-altogether-on-dependency-management; we've had better options for decades. Change will happen, but it will take decades as individual packages become less popular in favor of newer alternatives with better features--build hygiene will improve because these newer projects are much more likely to be built in Rust or "disciplined C" by developers who are increasingly of a younger generation, less wed to the Old Ways of building software.
      - Spivak 6y ago
        Look I loathe autotools as a developer but it has the advantage of being extremely battle-tested and comes with lots of tooling to package it out of the box. In RPM spec files any autotools project can be packaged with more or less a single macro.
      - asguy 6y ago
        > It's not a dearth of better options that causes C folks to clutch their autotools and bash scripts and punting-altogether-on-dependency-management; we've had better options for decades. I build and package golang/Python/rust/C/C++ binaries using GNU make, bash, and Debian's tooling. I have dependency management, parallel builds, and things are reproducible. I do it that way, because I need a glue that will deploy to VMs, containers, bare metal, whatever. I don't use it because I'm scared of other tooling. I use it because I haven't seen a better environment for getting software I work on, into heterogeneous environments. I'm not attached to the old way; I'm attached to ways that I can be productive with.
- vmchale 6y ago
  fd is better IMO, plus it is more cross-platform. Same flags on Linux and Mac
  - dralley 6y ago
    fd doesn't do the same thing at all. fd matches against filenames, rg matches against file contents.
- alerighi 6y ago
  Another thing to mention: there is not only the compatibility at the side of software (scripts), that to me is easy to maintain (just keep the old software with the new compatible one and you are done). The most important thing is compatibility with humans. The main reason because I tend to use a pretty standard setup is because I know that these tools are the standard, I need to ssh into a server to resolve some issue, well I have a system that I'm familiar with. If for example on my system I have fancy tools then I would stop using the classic UNIX tools, and then every time I would have to connect to another system (quite often) I would type wrong commands because they are different or totally not present or have to install the new fancy tools on the new system (and it's not always the case, for example if we talk about embedded devices with 8Mb of flash everything more than busybox is not possible). To me the GNU tools have the same problem, they got you used to non POSIX standard things (like putting flags after positional arguments) that hits you when you have to do some work on systems without them. And yes, there still exist system without the GNU tools, MacOS for example, or embedded devices, some old UNIX server that is still running, etc. Last thing, if we need to look at the future... better change everything. Let's be honest, if I could chose to switch entirely on a new shell and tools, and magically have it installed on every system that I use, it would probably be PowerShell. For the simple reason that we are no longer in the '80 where everything was a stream of text, and a shell that can manipulate complex objects is fantastic.
  - pdimitar 6y ago
    > Last thing, if we need to look at the future... better change everything. Absolutely. We had decades to work with a fairly stable set of tools and they are not going anywhere. Whoever needs them, they are there and likely will be for several more decades. I am gradually migrating all my everyday DevOps workflows (I am a senior programmer and being good with tooling is practically mandatory for my work) to various Rust tools: ls => exa, find => fd, grep => rg, and more and more are joining each month. I am very happy about it! They are usually faster and work more predictably and (AFAICT) have no hidden surprises depending on the OS you use them in. > we are no longer in the '80 where everything was a stream of text, and a shell that can manipulate complex objects is fantastic. Absolutely (again). We need a modern cross-platform shell that does that. There are few interesting projects out there but so far it seems that the community is unwilling to adopt them. I am personally also guilty of this: I like zsh fine even though the scripting language is veeeeery far from what I'd want to use in a shell. Not sure how that particular innovation will explode but IMO something has to give at one point. Pretending everything is a text and wasting trillions of CPU hours in constantly parsing stuff is just irresponsible in so many ways (ecological included).
simias 6y ago
I think both approaches are interesting and valuable. Yes ripgrep is great for interactive usage, but in scripts I'll still be using grep for the foreseeable future. I think the situation with GNU coreutils on Solaris/BSDs is a bit different: a lot of the time the BSD/Solaris tools on one hand and the GNU tools on the other were incompatible in some ways. Some flags would be different (GNU ls vs. FreeBSD ls for instance have pretty significant flag differences) and then you have things like Make who have pretty profound syntax differences between flavours. As a result if you needed to write portable scripts you either had to go through a painful amount of testing and special casing for the various dialects, or you'd just mandate that people installed the GNU version of the tools on the system and use that. That's why you can pretty much assume that any BSD system in the wild these days has at the very least gmake and bash installed, just because it's used by third party packages. So IMO people used GNU on non-GNU systems mainly for portability or because they came from GNU-world and they were used to them. I know that there's a common idea that GNU tools were more fully featured or ergonomic than BSD equivalents, but I'm not entirely sold on that. I think it's mostly because people these days learn the GNU tools first, and they're likely to notice when a BSD equivalent doesn't support a particular feature while they're unlikely to notice the opposite because they simply won't use BSD-isms. For instance for a long time I was frustrated with GNU tar because it wouldn't automatically figure out when a compression algorithm was used (bz, gz or xz in general) and automatically Do The Right Thing. FreeBSD tar however did it just fine. Similarly I can get FreeBSD ls to do pretty much everything GNU ls can do, but you have to use different flags sometimes. If you don't take the time to learn the BSDisms you'll just think that the program is more limited or doesn't support all the features of the GNU version. An other example out of the top of my head is "fstat" which I find massively more usable than lsof, which is a lot more clunky with worse defaults IMO. It's mainly that lsof mixes disk and network resources by default, while fstat is dedicated to local resources and you use other programs like netstat to monitor network resources. Since I rarely want to monitor both at the same time when I'm debugging something, I find that it makes more sense to split those apart.
- mprovost 6y ago
  You're right, there were a lot of compatibility problems in the past because tools reused the same names but didn't behave the same. Sometimes they were completely different like ps - the BSD and System V versions are totally incompatible. There were attempts to unify some tools with something new, for example pax (which was supposed to replace the incompatible tar and cpio, but which basically nobody uses). These days most tools are getting new names (ripgrep, exa) where in the past you'd just call it the same thing. I think that decision has freed people up from having to maintain backwards compatibility with the GBU tools - which weren't always backwards compatible with the tools of the same name that they replaced, as you point out. Once you lose the requirement to be compatible you can actually rethink how the tool should be used and not worry about a decision that was made by someone in the 70s.

est31 6y ago

The GNU coreutils are the lowest common denominator of most GNU systems, and thus a lot of software targets them including the GNU extensions. Replacements like rg or fd are often not even compatible to the unix originals let alone the GNU extensions. Yes, some of the GNU tools are badly in need of UI improvements, but you'd have to abandon compatibility with a great deal of scripts. I think both a coreutils rewrite as well as some end user facing software like rg has its place on modern unix systems. I'm a very happy user of rg! But I'd like some more respect for tradition by some of those tools. For example, "fd" in a unix setting refers to file descriptors. They should rename IMO.

adrianN 6y ago
Not being compatible is often the point of tools like ripgrep.
- steerablesafe 6y ago
  It's not the point of ripgrep. If ripgrep were only marginally better at common grepping tasks then it could hardly justify not being compatible. It aims to be much better though, so it can justify it well. But not being compatible is not the point or the goal of the project, I believe.
  - burntsushi 6y ago
    Yes, this is correct in some sense. I didn't set out to create an "incompatible grep." I set out to build a "better" grep, where "better" could include things like being incompatible. Of course, ripgrep is still ultimately a lot more similar to grep than it is different. Where possible, I used the same flag names and prescribed similar behavior---if it made sense. Because it's good to lean on existing experiences. It makes it easier for folks to migrate. So it's often a balancing act. But yes, POSIX compatibility is certainly a non-goal of ripgrep.
    - jftuga 6y ago
      I would love it if --crlf was enabled by default with the Windows version. This would make using ^ and $ easier.
      - jpitz 6y ago
        Make a config file for ripgrep and enable it. https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#configuration-file https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#c...
    - coldtea 6y ago
      That said, thanks for making ripgrep so good. I use it 100s of times every day for 3-4 years now... (and iirc, it's the default engine in VS Code as well).
    - orra 6y ago
      Not having to specify "-rHInE" on every invocation is one killer reason to use ripgrep, over grep. It's a concrete advantage of you not having 100% compatibility as a goal. (The other killer reason is the blitzing speed.)
      - ses1984 6y ago
        You can solve `-rHInE` with just an alias though.
- thesuperbigfrog 6y ago
  If a Rust rewrite of Coreutils is not backward compatible, it is not a replacement and the current GNU Coreutils would still be needed for backward compatibility. There is a long history of newer, better tools supporting backward compatibility to be a replacement for their predecessors: - bash and zsh support backward compatibility to be replacements for sh. - vim has compatibility mode to be a replacement for vi. If the Rust implementation is not backward compatible, it should not be called "Coreutils".
  - GoblinSlayer 6y ago
    What you call "Coreutils" should be called GNU/Coreutils.
    - thesuperbigfrog 6y ago
      The Rust implementation that we are discussing is MIT licensed and as far as I know not part of GNU. I do not believe that "Coreutils" is trademarked by GNU. There are other projects that use the name "Coreutils" that are not part of GNU: https://github.com/DiegoMagdaleno/BSDCoreUtils https://github.com/DiegoMagdaleno/BSDCoreUtils
      - entropicdrifter 6y ago
        I think the person you're replying to was just memeing on the Richard Stallman essay that has the line, "What you call Linux should really be called GNU/Linux"
        audience_mem 6y ago
        I was sad to find out that rms apparently didn't even say it.
      - GoblinSlayer 6y ago
        >I do not believe that "Coreutils" is trademarked by GNU. That's exactly why GNU sort of coreutils should be called GNU/Coreutils. The comment said that GNU/Coreutils are equivalent to Coreutils, I said they aren't.
throwaway894345 6y ago
> you'd have to abandon compatibility with a great deal of scripts. Having tried to reproducibly package various software, the C layer (and especially the coreutils layer) are the absolute worst, and I wouldn't shed a tear if we started afresh with something more holistically designed.
- asguy 6y ago
  Absolute worst compared to what? Have you tried packaging software on Windows? If you have something better in mind, please implement it. It’d get used.
  - throwaway894345 6y ago
    "Worst" compared to packaging software in other languages. > If you have something better in mind, please implement it. It’d get used. Nonsense. It's not a dearth of better options that causes C folks to clutch their autotools and bash scripts and punting-altogether-on-dependency-management; we've had better options for decades. Change will happen, but it will take decades as individual packages become less popular in favor of newer alternatives with better features--build hygiene will improve because these newer projects are much more likely to be built in Rust or "disciplined C" by developers who are increasingly of a younger generation, less wed to the Old Ways of building software.
    - Spivak 6y ago
      Look I loathe autotools as a developer but it has the advantage of being extremely battle-tested and comes with lots of tooling to package it out of the box. In RPM spec files any autotools project can be packaged with more or less a single macro.
    - asguy 6y ago
      > It's not a dearth of better options that causes C folks to clutch their autotools and bash scripts and punting-altogether-on-dependency-management; we've had better options for decades. I build and package golang/Python/rust/C/C++ binaries using GNU make, bash, and Debian's tooling. I have dependency management, parallel builds, and things are reproducible. I do it that way, because I need a glue that will deploy to VMs, containers, bare metal, whatever. I don't use it because I'm scared of other tooling. I use it because I haven't seen a better environment for getting software I work on, into heterogeneous environments. I'm not attached to the old way; I'm attached to ways that I can be productive with.
vmchale 6y ago
fd is better IMO, plus it is more cross-platform. Same flags on Linux and Mac
- dralley 6y ago
  fd doesn't do the same thing at all. fd matches against filenames, rg matches against file contents.
alerighi 6y ago
Another thing to mention: there is not only the compatibility at the side of software (scripts), that to me is easy to maintain (just keep the old software with the new compatible one and you are done). The most important thing is compatibility with humans. The main reason because I tend to use a pretty standard setup is because I know that these tools are the standard, I need to ssh into a server to resolve some issue, well I have a system that I'm familiar with. If for example on my system I have fancy tools then I would stop using the classic UNIX tools, and then every time I would have to connect to another system (quite often) I would type wrong commands because they are different or totally not present or have to install the new fancy tools on the new system (and it's not always the case, for example if we talk about embedded devices with 8Mb of flash everything more than busybox is not possible). To me the GNU tools have the same problem, they got you used to non POSIX standard things (like putting flags after positional arguments) that hits you when you have to do some work on systems without them. And yes, there still exist system without the GNU tools, MacOS for example, or embedded devices, some old UNIX server that is still running, etc. Last thing, if we need to look at the future... better change everything. Let's be honest, if I could chose to switch entirely on a new shell and tools, and magically have it installed on every system that I use, it would probably be PowerShell. For the simple reason that we are no longer in the '80 where everything was a stream of text, and a shell that can manipulate complex objects is fantastic.
- pdimitar 6y ago
  > Last thing, if we need to look at the future... better change everything. Absolutely. We had decades to work with a fairly stable set of tools and they are not going anywhere. Whoever needs them, they are there and likely will be for several more decades. I am gradually migrating all my everyday DevOps workflows (I am a senior programmer and being good with tooling is practically mandatory for my work) to various Rust tools: ls => exa, find => fd, grep => rg, and more and more are joining each month. I am very happy about it! They are usually faster and work more predictably and (AFAICT) have no hidden surprises depending on the OS you use them in. > we are no longer in the '80 where everything was a stream of text, and a shell that can manipulate complex objects is fantastic. Absolutely (again). We need a modern cross-platform shell that does that. There are few interesting projects out there but so far it seems that the community is unwilling to adopt them. I am personally also guilty of this: I like zsh fine even though the scripting language is veeeeery far from what I'd want to use in a shell. Not sure how that particular innovation will explode but IMO something has to give at one point. Pretending everything is a text and wasting trillions of CPU hours in constantly parsing stuff is just irresponsible in so many ways (ecological included).

simias 6y ago

I think both approaches are interesting and valuable. Yes ripgrep is great for interactive usage, but in scripts I'll still be using grep for the foreseeable future. I think the situation with GNU coreutils on Solaris/BSDs is a bit different: a lot of the time the BSD/Solaris tools on one hand and the GNU tools on the other were incompatible in some ways. Some flags would be different (GNU ls vs. FreeBSD ls for instance have pretty significant flag differences) and then you have things like Make who have pretty profound syntax differences between flavours. As a result if you needed to write portable scripts you either had to go through a painful amount of testing and special casing for the various dialects, or you'd just mandate that people installed the GNU version of the tools on the system and use that. That's why you can pretty much assume that any BSD system in the wild these days has at the very least gmake and bash installed, just because it's used by third party packages. So IMO people used GNU on non-GNU systems mainly for portability or because they came from GNU-world and they were used to them. I know that there's a common idea that GNU tools were more fully featured or ergonomic than BSD equivalents, but I'm not entirely sold on that. I think it's mostly because people these days learn the GNU tools first, and they're likely to notice when a BSD equivalent doesn't support a particular feature while they're unlikely to notice the opposite because they simply won't use BSD-isms. For instance for a long time I was frustrated with GNU tar because it wouldn't automatically figure out when a compression algorithm was used (bz, gz or xz in general) and automatically Do The Right Thing. FreeBSD tar however did it just fine. Similarly I can get FreeBSD ls to do pretty much everything GNU ls can do, but you have to use different flags sometimes. If you don't take the time to learn the BSDisms you'll just think that the program is more limited or doesn't support all the features of the GNU version. An other example out of the top of my head is "fstat" which I find massively more usable than lsof, which is a lot more clunky with worse defaults IMO. It's mainly that lsof mixes disk and network resources by default, while fstat is dedicated to local resources and you use other programs like netstat to monitor network resources. Since I rarely want to monitor both at the same time when I'm debugging something, I find that it makes more sense to split those apart.

mprovost 6y ago
You're right, there were a lot of compatibility problems in the past because tools reused the same names but didn't behave the same. Sometimes they were completely different like ps - the BSD and System V versions are totally incompatible. There were attempts to unify some tools with something new, for example pax (which was supposed to replace the incompatible tar and cpio, but which basically nobody uses). These days most tools are getting new names (ripgrep, exa) where in the past you'd just call it the same thing. I think that decision has freed people up from having to maintain backwards compatibility with the GBU tools - which weren't always backwards compatible with the tools of the same name that they replaced, as you point out. Once you lose the requirement to be compatible you can actually rethink how the tool should be used and not worry about a decision that was made by someone in the 70s.