6 ms·

Early on in my career, like most novice programmers, I thought that custom written C programs could be much faster than unix tools if written well and for a spe

Early on in my career, like most novice programmers, I thought that custom written C programs could be much faster than unix tools if written well and for a specific purpose. However, I could not beat the speed of unix tools like grep, cut or cat even once. That is when I realized just how well written these tools are and just how much optimization work has been done.

byte1918 8y ago
https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/?st=j3v3iw3c&sh=5651ea3c https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...
_bxg1 8y ago
It's amazing to me how programs like these have seemed to avoid the universal phenomenon of technical debt. They've crystallized into an ideal version of themselves, and haven't continued to decay past that point. Maybe it's because of the Unix philosophy of single-purpose programs; no feature-creep tends to mean no technical debt.
- Twirrim 8y ago
  By keeping a narrow focus and avoiding feature creep, the technical debt has been pretty minimal, and they've been able to pay it down easily. There's at least something to learn from the approach. It does also help that there is a certain type of developer that sees such things as a personal challenge and will go through hell and high water to come up with more and more efficient ways to do things.
  - deleted 8y ago
    [deleted]
- asveikau 8y ago
  There is truth in what you say, however, if you think GNU tools are free of technical debt or feature creep, look into how ./configure works, as an example. Or, there's the old joke about GNU echo: https://www.gnu.org/fun/jokes/echo-msg.en.html https://www.gnu.org/fun/jokes/echo-msg.en.html
  - wahern 8y ago
    Autoconf code is actually quite clean--you just need to know M4 and have an appreciation of the problems autoconf was built to overcome. What changed over time is that most proprietary unix systems disappeared[1], and POSIX compliance and general consistency have improved dramatically, BSDs and GNU included. Also, best practices have shifted to a continuous development model which keeps everybody on the upgrade treadmill. There's less concern with maintaining backwards compatibility and catering to those not running the latest environments. So if you make use of some newer Linux kernel API there's only a short window where people will put in the effort to maintain a compatibility mode, assuming they bother at all. Lastly, containers mean people often develop and ship static environments that can be maintained independently, sometimes never upgraded at all. What I find interesting is how people have begun to ditch autoconf in favor of even more complex (but newer and therefore cooler) build systems when ironically there's less need than ever for these things. Autoconf doesn't need replacing; such layers can often be left out entirely. That said, when feature detection and backwards compatibility truly matters there's no good alternative. CMake, for example, effectively requires the end user to install the latest version of CMake, and if you already expect someone to install the latest version of something then why the contrivance at all? I always sigh aloud whenever I download a project that relies on CMake because I know that I now have two problems on my hand, not just one. (But better CMake than the other alternatives--I just won't even bother.) [1] All that's left are AIX and Solaris. HP-UX/PA-RISC will be officially dead next year, and EOL for HP-UX/Itanium is 2025. From a commercial perspective Solaris seems to be deader than AIX, however Solaris still seems to see more development--especially improved POSIX and Linux compatibility. It's much easier to port to Solaris than AIX. It's a real shame Solaris is disappearing because on big machines with heavy workloads the OOM killer is a fscking nightmare on Linux. Solaris and Windows (and maybe AIX?) are the only operating systems that do proper and thorough memory accounting, permitting you to write reliable software.[2] The cloud services principle that says individual processes are expendable doesn't work when your job takes hours or days to run. (Or even just minutes, because workloads accumulate when the OOM killer starts shooting things down, and even in the cloud you run into hard limits on resource usage--i.e. cap on numbers of nodes. Memory overcommit is just like network buffer bloat--intended to improve things at the small scale but which results in catastrophic degradation at the macro level.) [2] You can disable overcommit on Linux but the model of overcommit is baked too deeply into the kernel's design. A machine can still end up with the OOM killer shooting down processes if, for example, the rate of dirty page generation outpaces the patience of the allocator trying to reclaim and access memory that is technically otherwise available.
- scarcely 8y ago
  If it only does one thing it is a tool. If it does many things it becomes a subordinate with an intelligence of its own, something you need to communicate with or talk to, as opposed to something you can simply use
- blihp 8y ago
  That's romanticizing it a bit... there's been a ton of feature creep[1] and a fair amount of technical debt. It's just more manageable because they've contained the scope of most of the core utilities and packages better than most 'modern' software. Also, don't underestimate the value of having a relatively stable set of maintainers over long periods of time. [1] If you think about it, things like colorization belong in grep about as much as SVG generation belongs in systemd.
  - BeeOnRope 8y ago
    How would you do colorization of matches outside of grep, other than reimplementing grep?
    - blihp 8y ago
      By optionally emitting generic structural markup which you could then use to format via colorization (or whatever other means you wanted) in a terminal, a GUI, a web page, etc. That would have been more consistent with the 'one tool, one job' philosophy of Unix. So then you could do something along the lines of 'grep --emit-markup ... | colorize-markup' (completely made up switch and command names) and use aliases/environment variables/etc. so that to a user could type something like 'cgrep' to get colorized output.
      - markrages 8y ago
        but the structure generator and decoder would be so closely coupled they might as well be the same program. Alternatively, you could have grep take a list of colors from a config file or environment variable and blindly interpolate those strings to make colored output[1]. This also allows color to show up automatically for interactive use. [1] this is in fact how grep works.
        hvis 8y ago
        That's not all one might want to do the structured out. Think of converting to HTML, as one example.
- pixelbeat__ 8y ago
  It takes a lot of focused work to keep them cohesive and performant https://www.gnu.org/software/coreutils/rejected_requests.html https://www.gnu.org/software/coreutils/rejected_requests.htm...
- jonstewart 8y ago
  GNU Grep’s source code is nigh-unreadable, and edge-case bugs are not unknown. I would say it has a fair amount of technical debt.
markrages 8y ago
https://ridiculousfish.com/blog/posts/old-age-and-treachery.html https://ridiculousfish.com/blog/posts/old-age-and-treachery....
vmchale 8y ago
It's possible to beat the speed of cat, actually. See e.g. https://hub.darcs.net/vmchale/fastcat https://hub.darcs.net/vmchale/fastcat. One trick is to not free the memory used by the buffer :p
- omn1 8y ago
  If you're on Linux, you can go way faster by using the splice syscall. :) See https://matthias-endler.de/2018/fastcat https://matthias-endler.de/2018/fastcat

byte1918 8y ago

https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/?st=j3v3iw3c&sh=5651ea3c https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

_bxg1 8y ago

It's amazing to me how programs like these have seemed to avoid the universal phenomenon of technical debt. They've crystallized into an ideal version of themselves, and haven't continued to decay past that point. Maybe it's because of the Unix philosophy of single-purpose programs; no feature-creep tends to mean no technical debt.

Twirrim 8y ago
By keeping a narrow focus and avoiding feature creep, the technical debt has been pretty minimal, and they've been able to pay it down easily. There's at least something to learn from the approach. It does also help that there is a certain type of developer that sees such things as a personal challenge and will go through hell and high water to come up with more and more efficient ways to do things.
- deleted 8y ago
  [deleted]
asveikau 8y ago
There is truth in what you say, however, if you think GNU tools are free of technical debt or feature creep, look into how ./configure works, as an example. Or, there's the old joke about GNU echo: https://www.gnu.org/fun/jokes/echo-msg.en.html https://www.gnu.org/fun/jokes/echo-msg.en.html
- wahern 8y ago
  Autoconf code is actually quite clean--you just need to know M4 and have an appreciation of the problems autoconf was built to overcome. What changed over time is that most proprietary unix systems disappeared[1], and POSIX compliance and general consistency have improved dramatically, BSDs and GNU included. Also, best practices have shifted to a continuous development model which keeps everybody on the upgrade treadmill. There's less concern with maintaining backwards compatibility and catering to those not running the latest environments. So if you make use of some newer Linux kernel API there's only a short window where people will put in the effort to maintain a compatibility mode, assuming they bother at all. Lastly, containers mean people often develop and ship static environments that can be maintained independently, sometimes never upgraded at all. What I find interesting is how people have begun to ditch autoconf in favor of even more complex (but newer and therefore cooler) build systems when ironically there's less need than ever for these things. Autoconf doesn't need replacing; such layers can often be left out entirely. That said, when feature detection and backwards compatibility truly matters there's no good alternative. CMake, for example, effectively requires the end user to install the latest version of CMake, and if you already expect someone to install the latest version of something then why the contrivance at all? I always sigh aloud whenever I download a project that relies on CMake because I know that I now have two problems on my hand, not just one. (But better CMake than the other alternatives--I just won't even bother.) [1] All that's left are AIX and Solaris. HP-UX/PA-RISC will be officially dead next year, and EOL for HP-UX/Itanium is 2025. From a commercial perspective Solaris seems to be deader than AIX, however Solaris still seems to see more development--especially improved POSIX and Linux compatibility. It's much easier to port to Solaris than AIX. It's a real shame Solaris is disappearing because on big machines with heavy workloads the OOM killer is a fscking nightmare on Linux. Solaris and Windows (and maybe AIX?) are the only operating systems that do proper and thorough memory accounting, permitting you to write reliable software.[2] The cloud services principle that says individual processes are expendable doesn't work when your job takes hours or days to run. (Or even just minutes, because workloads accumulate when the OOM killer starts shooting things down, and even in the cloud you run into hard limits on resource usage--i.e. cap on numbers of nodes. Memory overcommit is just like network buffer bloat--intended to improve things at the small scale but which results in catastrophic degradation at the macro level.) [2] You can disable overcommit on Linux but the model of overcommit is baked too deeply into the kernel's design. A machine can still end up with the OOM killer shooting down processes if, for example, the rate of dirty page generation outpaces the patience of the allocator trying to reclaim and access memory that is technically otherwise available.
scarcely 8y ago
If it only does one thing it is a tool. If it does many things it becomes a subordinate with an intelligence of its own, something you need to communicate with or talk to, as opposed to something you can simply use
blihp 8y ago
That's romanticizing it a bit... there's been a ton of feature creep[1] and a fair amount of technical debt. It's just more manageable because they've contained the scope of most of the core utilities and packages better than most 'modern' software. Also, don't underestimate the value of having a relatively stable set of maintainers over long periods of time. [1] If you think about it, things like colorization belong in grep about as much as SVG generation belongs in systemd.
- BeeOnRope 8y ago
  How would you do colorization of matches outside of grep, other than reimplementing grep?
  - blihp 8y ago
    By optionally emitting generic structural markup which you could then use to format via colorization (or whatever other means you wanted) in a terminal, a GUI, a web page, etc. That would have been more consistent with the 'one tool, one job' philosophy of Unix. So then you could do something along the lines of 'grep --emit-markup ... | colorize-markup' (completely made up switch and command names) and use aliases/environment variables/etc. so that to a user could type something like 'cgrep' to get colorized output.
    - markrages 8y ago
      but the structure generator and decoder would be so closely coupled they might as well be the same program. Alternatively, you could have grep take a list of colors from a config file or environment variable and blindly interpolate those strings to make colored output[1]. This also allows color to show up automatically for interactive use. [1] this is in fact how grep works.
      - hvis 8y ago
        That's not all one might want to do the structured out. Think of converting to HTML, as one example.
pixelbeat__ 8y ago
It takes a lot of focused work to keep them cohesive and performant https://www.gnu.org/software/coreutils/rejected_requests.html https://www.gnu.org/software/coreutils/rejected_requests.htm...
jonstewart 8y ago
GNU Grep’s source code is nigh-unreadable, and edge-case bugs are not unknown. I would say it has a fair amount of technical debt.

markrages 8y ago

https://ridiculousfish.com/blog/posts/old-age-and-treachery.html https://ridiculousfish.com/blog/posts/old-age-and-treachery....

vmchale 8y ago

It's possible to beat the speed of cat, actually. See e.g. https://hub.darcs.net/vmchale/fastcat https://hub.darcs.net/vmchale/fastcat. One trick is to not free the memory used by the buffer :p

omn1 8y ago
If you're on Linux, you can go way faster by using the splice syscall. :) See https://matthias-endler.de/2018/fastcat https://matthias-endler.de/2018/fastcat