7 ms·

There seems to be a lot of confusion in this discussion about what the change is and how egrep and fgrep work. These are not symlinks like some have suggested b

• tomn 4y ago
  the overhead isn't even an extra fork, as it uses exec
  • hnlmorg 4y ago
    Good point. Though invoking $SHELL and parsing the script (as short as it is) can't be cheap either. Academically speaking of course; I'm not trying to justify the change on performance grounds.
  • txutxu 4y ago
    On most context involving "using grep", shell scripts, interactive shell, etc, it's nothing. 81 Vs 128 system calls on my laptop (just printing --version) $ strace -fc grep -E --version 2>&1 | tail -1 100.00 0.000000 0 81 3 total $ strace -fc egrep --version 2>&1 | tail -1 100.00 0.000000 0 128 6 total Measured with "time" 0m0.002s Vs 0m0.003s, always, testing 4 times each option. Most situations where it could be being used, probably are surrounded by much bigger optimizations to work on.
    • tomn 4y ago
      true. using hyperfine on my machine shows that grep is about 2x faster than egrep (and yeah, still nothing, 1.5 vs. 3ms)
• tannhaeuser 4y ago
  When it comes to avoiding forks, the question is why do egrep, fgrep have to be shell scripts rather than interpret the invocation name (argv[0]) for switching behavior. Also, I find the original POSIX spec to lump everything into a single grep binary questionable since egrep, fgrep developed as extension and restriction, resp. of grep under a classic regexp discourse and it isn't clear at all that automaton construction for powerful egrep support has to live in the same binary as fgrep which isn't using automaton construction at all.
  • hnlmorg 4y ago
    > When it comes to avoiding forks, the question is why do egrep, fgrep have to be shell scripts rather than interpret the invocation name (argv[0]) for switching behavior. I don't know. I'd have taken the argv[0] approach personally. And it's the approach some other grep implementations have taken too (eg https://github.com/freebsd/freebsd-src/blob/main/usr.bin/grep/grep.c#L327 https://github.com/freebsd/freebsd-src/blob/main/usr.bin/gre...). The argv[0] approach (to me) feels like a right kind of compromise.
    • cryptonector 4y ago
      Switching behavior according to argv[0] been used in Unix for decades. mv(1)/ln(1)/cp(1) historically are the same program, hardlinked.
      • hnlmorg 4y ago
        Wasn’t rm also the same? I vaguely recall it being little more than ‘mv $1 /dev/null’ under the hood. But this might be something I’m misremembering.
        • cryptonector 4y ago
          Yes. I made a mistake, I think cp(1) was separate.
  • ratmice 4y ago
• jomar 4y ago
  I too was surprised to see these were shell scripts. I was expecting the grep/fgrep/egrep names to be hard links to the same executable that would check `argv[0]`, as the BSD implementation does. More interesting than the commit diff is the brief discussion on the bug report from when this all happened a year ago, as referenced in the commit message: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49996 https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49996 Sigh. Time to retrain the old fingers again...
  • hnlmorg 4y ago
    Thanks for sharing that. It was an interesting read. Particularly this comment: > The irony... that one of our own tests used fgrep! If ever there was an argument for fgrep becoming a pseudo-standard that should remain, that would have been it.
• gatonegro 4y ago
  > I'd be surprised if that warning made its way to the mainstream distros. This is the first time I'm hearing about the warning, but I thought I'd run `egrep` and `fgrep` on my Arch system. Sure enough, both result in warnings: `egrep: warning: egrep/fgrep is obsolescent; using grep -E/-F` So it's made its way into Arch, at least. Though, like I said, this is the first time I'm hearing about this.
  • thayne 4y ago
    I use arch, and I had an alias for grep to expand to `egrep --color=always` in my interactive shell. I noticed real fast. Fortunately it's an easy fix to `grep -E --color=always`. sidenote: I wish that grep would allow running `grep -E -F`, and have the `-F` flag override the `-E` flag, rather than giving an error about conflicting options, so that I could have an alias like this to make extended regex the default, but allow me to change it with a flag.
    • NegativeLatency 4y ago
      You could write a little function that does that
      • thebeardisred 4y ago
        For me, that's the biggest takeaway: Nothing is stopping users from maintaining the behavior they had previously. It's being made explicit that the burden of maintaining that behavior is changing.
        • TillE 4y ago
          Yeah, this is a good point. This is basically just GNU saying that those old aliases are not something they want within the scope of grep itself.
        • MobiusHorizons 4y ago
          What exactly do you think the burden of _not_ changing a shell script is?
• chaxor 4y ago
  > "if concerned with performance why are you running code as a shell script" Rewrite grep in python before running - got it. :P
  • hnlmorg 4y ago
    That's not what I suggested. I was saying if you're writing performance sensitive code with a hot loop calling `egrep` then a smarter approach might be to use a language better tuned for performance which supports regex libraries. Shell scripts have their place and can out perform most languages if you're writing simple logic that can run across large datasets in parallel, such as cat very-large-file | grep "do not want" | sed -r 's/foo/bar/' > very-large-file-refactored (please excuse the useless use of `cat`, it's there to illustrate the direction of data flow) But in those types of scenarios the cost of $SHELL interpretation and fork() is massively outweighed by the savings of stream processing. So my point was: if you're writing a function which $SHELL interpretation and/or fork() create enough of a performance impact where you're looking to optimize how you exec `grep -E`, then maybe it's time to investigate whether $SHELL is the right language to write your function in.
    • jraph 4y ago
      > please excuse the useless use of `cat`, it's there to illustrate the direction of data flow Note that you can write < very-large-file Instead of cat very-large-file | To avoid a useless use of cat and keep the direction. Not that I care very much though. Using cat is less surprising to most people.
      • hnlmorg 4y ago
        > Using cat is less surprising to most people. Exactly :) STDIN redirection is a neat trick though so definitely worth highlighting. But it wouldn’t have helped with readability in my particular example.
    • chaxor 4y ago
      Perhaps I should have added a "/s", in addition to my ":p", but I appreciate the explanation here nonetheless
• vlovich123 4y ago
  > though it has already been committed to git for more than a year now and this is the first I've heard people complain about the change. This reasoning is flawed. I only noticed it a few months back on Arch. So between the time it takes the project to make a new release and it hitting Linux distros (Debian is slower and Ubuntu even moreso), don’t expect a flood of complaints to be time correlated with the change.
  • hnlmorg 4y ago
    > This reasoning is flawed. I wasn't making any reasoning. I was just making the observation that this change has been staged for a while now and that I'm surprised there hasn't been more noise about it before now. > don’t expect a flood of complaints to be time correlated with the change. I expect complaints to be correlated with whenever a blog/tweet/whatever moaning about the change happens to trend. I don't think it has much to do with when distros pick up the change because, as I also said, I expect distros to backport the old behavior. So I think the timing of any backlash is entirely dependent on the mood of the internet hive mind.
    • paulmd 4y ago
      > I was just making the observation that this change has been staged for a while now “There’s no point in acting surprised about it. All the planning charts and demolition orders have been on display at your local planning department in Alpha Centauri for 50 of your Earth years, so you’ve had plenty of time to lodge any formal complaint and it’s far too late to start making a fuss about it now. … What do you mean you’ve never been to Alpha Centauri? Oh, for heaven’s sake, mankind, it’s only four light years away, you know. I’m sorry, but if you can’t be bothered to take an interest in local affairs, that’s your own lookout. Energize the demolition beams.” Again, you're completely missing the point of the comment you're replying to: average users don't pay attention to what's being staged upstream. You're only going to get the real flood of complaints when it actually gets pushed out and people's houses start getting demolished.
      • hnlmorg 4y ago
• brundolf 4y ago
  Why wouldn't they just convert them to aliases instead of adding the warnings?
  • hnlmorg 4y ago
    aliases aren't a global state. They're shell and profile specific (as there are plenty of instances where a profile isn't loaded). With `egrep` defined as a shell script, any process (even callers who are not shells) will have the same behavior. That doesn't mean the shell script is the best solution though. Personally I'd rather they be a symlink to `grep` and have grep check the name it's been invoked as, and if it's "egrep" (or "e" if you're just checking the first byte) then implicitly enable the -E flag. This is how FreeBSD works and it feels like a more elegant solution in my opinion.
    • brundolf 4y ago
      Sure, makes sense- so then I guess I'd amend my question to that: I can't see why they'd add the warning instead of just redirecting the command
• kps 4y ago
  > I remember around ~10 years ago being told "you should never use `egrep` because it is slower than `grep -E`." I remember around 30 years ago reading that egrep was faster (the UNIX versions were entirely separate implementations, egrep of course being newer) and have used it ever since.
  • lamontcg 4y ago
    exactly the same here. although i'm using ripgrep more now, proving that you can teach an old dog new tricks.
• fnordpiglet 4y ago
  I am not sure I agree with the logic. On an absolute instruction count scale grep will often dwarf the instructions necessary for the higher level logic captured in the shell script. Insert appeal to andahls law here. Writing in some more optimal language often requires considerably more work, and if the marginal benefit of reducing a few forks is minuscule compared to grep performance, why would you do this?
  • hnlmorg 4y ago
    It depends on where your hot path is. If you’re writing the kind of performance critical code where the different between ‘egrep’ and ‘grep -E’ matters, then you shouldn’t be writing it in $SHELL to begin with. Not just because of the expense fork()ing, but because Bash is interpreted, because each exec requires loading new code into your L1 cache, and so on. It’s just never going to be as quick as something written in most other languages with a mature regex library. I’m not saying this as a criticism against shell scripting (I’m an author of an alternative $SHELL and shell scripting language, so do have a love for shell scripting). I’m saying it as a criticism against people who try to optimise against the wrong things. I guess in a way we are “violently agreeing” (as some say) because your point about the engineering effort mattering more than micro-optimisations is an example of the right kind of optimisations. Which is what I’m also, albeit badly, trying to describe.
• mike_hock 4y ago
  What's the problem? Just make egrep and fgrep shell aliases in the default /etc/profile (if you're a distro maintainer). Also, where's the extra fork? #!/bin/sh exec grep -E "$@"
• cryptonector 4y ago
  The maintainers' fears about using argv[0] to select behavior are unfounded. Anyone exec()'ing grep with an alternate argv[0] can just use (or not use) the -E/-F options to get the behavior they want. It's very simple. There is just no good reason for egrep or fgrep to be shell scripts or wrapper programs of any kind. Even if the were a good reason for having a wrapper program, the wrapper program could be a very tiny C (or even assembly) program that just exec()s grep with -E or -F added, thus avoiding any additional fork() calls (though there would still be one additional exec() call). But, really, argv[0] dispatching is plenty good enough. This warning, however, is much too annoying. Plus, argv[0] dispatching has worked for every other Unix and Unix-like OS of note for 4+ decades! What's so special about GNU in this regard? The answer is clearly: nothing.
• m463 4y ago
  wow. Didn't realize they are shell scripts on linux. on (an older) macos system, they are identical executables