11 ms·

Speed vs readability: https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github

by exxo_ 9y ago

Speed vs readability: https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c

ceronman 9y ago
Normally I think readability is more important than speed. But in this particular case, I think GNU is doing the right thing optimizing the code to the limit. This is the beautiful part of Unix: small tools that do only one thing well. Programs following this philosophy are very good abstractions. They do one very well defined thing so you can use them without having to understand how they work. I have used Unix for years and I've never felt the need to read the source code for `yes`. And because they do a very small thing, even if you need to read them, the overhead of optimization is not that much, for example, the optimized GNU yes is just under 100 LOC if you remove comments and help boilerplate. Yes, it's longer than the BSD version, but it's just a matter of minutes to understands what it does.
- IshKebab 9y ago
  I totally disagree. Nobody will ever want to use `yes` at 10 GB/s. They will want it to be reliable, and this sort of over-optimisation increases the risk of bugs.
  - avar 9y ago
    I'm having a hard time imagining a case where even a grossly complex 100,000 line implementation of yes(1) couldn't be trivially proven to be correct.
    - fredmorcos 9y ago
      Absolutely, I once trivially proved a 1000,000 line implementation of return 0 to be correct. I don't know why all comments are bothered by how much overkill this yes implementation is. Maybe they don't hold 100,000 PhDs like we do, am I right?
    - acdha 9y ago
      I agree with the general point that you could prove such a simple program correct relatively easily but that does still have a cost, which is always a concern in an open-source project. You still need someone to step up and do that work and continue to verify it periodically in the future – if that code is doing complicated things with buffering, that opens up possible odd bugs due to stdlib, gcc, maybe even kernel behaviour changes which might not affect simpler programs. Not a huge bit of work to be sure but for a non-commercial project you might have trouble finding a volunteer who cares about that tedium.
    - mikeash 9y ago
      Easy! Write a program that does a brute force check of Goldbach's conjecture on all integers. For each integer that passes the check, print a line. If you can prove this correct (or incorrect) you'll probably win a Fields medal.
  - yk 9y ago
    The ability to get 10 GB/s dummy date into a pipe from the command line could come in handy at some point, for stress testing or something. I am not sure if it is over-optimized. (And even with the optimizations the risk of bugs should be very small.)
    - masklinn 9y ago
      Of course you could just pipe from /dev/zero which will easily do 10GB/s on every machine.
      - yk 9y ago
        Just tried it, cat /dev/zero|pv >/dev/null gives me roughly 2/3 the speed of yes. (5 GB/s vs 7 GB/s) Plus yes gives you an arbitrary string, instead of only zeroes.
        tobik 9y ago
        That's because you're using cat and a pipe. Try this instead: pv > /dev/null < /dev/zero
        mikeash 9y ago
        For anyone who is like me and finds it uncomfortable that things are now out of order, note that you can still put the input redirection in front: </dev/zero pv >/dev/null
        vram22 9y ago
        Or even this: pv </dev/zero >/dev/null which is a common way of doing it (for any command with any inputs and outputs, not just the above ones), i.e.: command < input_source > output_dest All three pv command invocation variants, the one here and the two above, work. And it becomes more clear why they are the same, when you know that the redirections are done by the shell + kernel, and the command (such as pv) does not even know about it. So in all three cases, it does not see the redirections, because they are already done by the time the command starts running (with its stdin and stdout redirected to / from those respective sources). And that is precisely why the command works the same whether it is reading from the keyboard or a file or pipe, and whether it is writing to the screen or a file or pipe.
  - joosters 9y ago
    I've used 'yes' many times to generate huge amounts of data quickly. Back then, it never had the small string optimisation, but you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/urandom or even /dev/zero I'm glad it runs fast, and I hope that all OS utilities are optimised (and tested, of course!) instead of making their source code pretty. The fact is, most people want to use programs, not read them.
    - masklinn 9y ago
      > you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/zero That really doesn't make any sense, /dev/zero should be at least as fast as yes.
      - joosters 9y ago
        /dev/zero should be at least as fast as yes I agree, all I remember is that when I tried it, /dev/zero sometimes sucked performance-wise. I can't recall the exact circumstances as it was some time ago, and could have been on any of Linux/FreeBSD/SunOS/HP-UX/IRIX - perhaps it was the fastest common way at the time? On a recent x64 Linux, /dev/zero seems plenty fast enough now: $ dd bs=8k count=819200 if=/dev/zero of=/dev/null 819200+0 records in 819200+0 records out 6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.331137 s, 20.3 GB/s $ yes | dd bs=8k count=819200 of=/dev/null 819200+0 records in 819200+0 records out 6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.959551 s, 7.0 GB/s
        Tepix 9y ago
        No need for "dd", let "pv" get the data from /dev/zero directly: $ pv < /dev/zero > /dev/null [ 16GiB/s] But the version of yes using vmsplice() is even faster than that on my machine.
        pbhjpbhj 9y ago
        What's the line to test `yes` with `pv`? pv < /usr/bin/yes > /dev/null doesn't seem to work properly. FWIW I get 330MiB/s vs 8.4GiB/s for /dev/zero. [Incidentally first I've heard of pv but I've known about dd for a decade or two].
  - belorn 9y ago
    If you want to maximize readability and simplicity then writing `yes` in C is a bad choice. It is much easier, cleaner and shorter to write it in python or just use the shell which would normally be using the `yes` in the first place. Since `yes` is used in shells and builtins can be considered very reliable, here is a implementation as a shell function: yes(){ while :; do echo "${1:-y}"; done; } Python: import sys while True: if len(sys.argv) > 1: print(sys.argv[1]) else: print("y") And if you don't need the features of `yes` and only need a loop that prints 'y', then there really is hard to beat the simplicity of: while:; do echo "y"; done
    - raptorfactor 9y ago
      Is it really "easier, cleaner and shorter to write it in python"? Did you look at the OpenBSD implementation? https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c It's essentially line for line identical to your python code...
      - fredmorcos 9y ago
        But it's C, and they're on HN, so C sucks, C should die, C is not modern, C is scary, C is bad, C makes them think, C forces them to be careful.
        sctb 9y ago
        You've been on a tear of uncivil and unsubstantive commenting, and it has to stop. Often a good strategy for this is to slow down. High-quality reflective posts are what we're after instead of dismissive, snarky reflexive ones, and the former come more slowly.
      - belorn 9y ago
        The C example has three includes, two conditionals, two loops, and one function definition. The python example has a single include, conditional, and loop. For readability purpose it is easier to go through each lines of the python program than the OpenBSD C code. Its not massively different, but its distinguishable enough that I would choose the python version if I wanted to maximize readability, minimize syntax requirement and did not want to use shell script. The Shell function is in my view the superior choice if the audience is a programmer than know the shell script syntax. It is just a single loop and is written in the environment that the program is intended to be used in. The only drawback there is the speed.
  - th4dv 9y ago
    > Nobody will ever want to use `yes` at 10 GB/s Just because you say so?
    - bastawhiz 9y ago
      If the speed of yes is bounded by memory speed, doing anything useful will almost certainly consume that data at a far lower rate. Putting it on a disk, pushing it over a network, etc. will almost always be slower than yes is able to generate data.
      - cestith 9y ago
        The typical use case is piping it into another running program. Maybe someone wants to do that really quickly rather than putting it on disk or pushing it over a network.
  - MichaelBurge 9y ago
    'yes' to a file is how I sometimes benchmark disk speed. It should have fewer system calls than reading from /dev/zero and then writing. I actually checked just now, and it looks like you'd be making twice as many system calls with /dev/zero compared to generating the data locally: strace dd count=1 bs=512 if=/dev/zero of=test open("/dev/zero", O_RDONLY) = 3 dup2(3, 0) = 0 close(3) = 0 lseek(0, 0, SEEK_CUR) = 0 open("test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 dup2(3, 1) = 1 close(3) = 0 read(0,"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512 close(0) = 0 close(1) While 'strace yes > test2' is just a constant stream of write() calls. The difference matters if you're benchmarking e.g. some new SSD compared to a tmpfs on a machine with 100+ GB of RAM. It's always better if the tools have less overhead, because the comparison is more meaningful. Also consider that it can be faster to write to a local network than to disk. I've never done it, but I imagine that the kernel's not going to want to deal with your /dev/zero calls if it's spending all of its time writing to a 10GB switch. I can imagine some very specialized storage servers that could spend most of their time writing from memory buffers to a network switch, or if you're troubleshooting a slowdown in the networking itself.
    - wolfgang42 9y ago
      When I started this comment, I didn't think you were measuring what you thought you were measuring with those straces. 'strace yes > test2' only watches 'yes', not '> test2' (which is handled by the shell). Here's what your command outputs: $ strace yes > /tmp/test2 [various initialization steps] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [...] To measure everything, I started up a whole new shell, expecting to see a 'write(1, ...)', a 'read(1)', and a 'write(f, ...)': $ strace -f sh -c 'yes > /tmp/test2' [various initialization steps] [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [...] How does this possibly work? File descriptor 1 is supposed to be the terminal, not a file! Of course, the magic of file redirection: open("/tmp/test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 # Open the file, get FD 3 fcntl(1, F_DUPFD, 10) = 10 # Copy FD 1 (the terminal, STDOUT) to FD 10 temporarily close(1) = 0 # Close original FD 1 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 # Close FD 10 (the terminal, copy of STDOUT) when exec() is called dup2(3, 1) = 1 # Copy FD 3 (the file) to FD 1 (STDOUT) close(3) = 0 # Close FD 3 (the file's original descrptor)
  - nobodyorother 9y ago
    Hey, 640K is enough for anyone!
  - peterwwillis 9y ago
    Unix userland tools have evolved over decades to be as efficient as possible because they have historically underpinned everything the operating system does. The faster each tool works, the faster the other tools that depend on them work. If increased efficiency results in a bug, that bug can then be fixed, making it a net gain for system stability.
  - tomohawk 9y ago
    It's not about running 'yes' at 10GB/s, it's about less overhead to do a simple job. If this version of yes is 100x faster, that implies it using 1% of a cpu to do the same work that would otherwise occupy 100% of a cpu. This leaves more of the machine to do what is likely to be the intended task.
- YSFEJ4SWJUVU6 9y ago
  It's not really optimized to the limit – or perhaps it is, but then the limit is fairly easy to reach. When I saw this item here I reached for the terminal and wrote a simple (and non-compliant) alternative that simply initialises 8150 bytes of "y\n" and then loops a write forever. I understand that it is not a fully standard-compliant yes, and that maybe GNU yes is indeed fast, but that awfully simple program that takes all of 10 lines (everything included) and took me all of a minute to write performs just as well as far as pv is concerned. (I eventually completed a feature complete yes but I still think that simply not using `puts` is hardly optimising to the limit.)
avar 9y ago
One thing to keep in mind when looking at GNU programs is that they're often intentionally written in an odd style to remove all questions of Unix copyright infringement at the time that they were written. The long-standing advice when writing GNU utilities used to be that if the program you were replacing was optimized for minimizing CPU use, write yours to minimize memory use, or vice-versa. Or in this case, if the program was optimized for simplicity, optimize for throughput. It would have been very easy for the nascent GNU project to unintentionally produce a line-by-line equivalent of BSD yes.c, which would have potentially landed them in the 80/90s equivalent of the Google v.s. Oracle case.
- exxo_ 9y ago
  This is the first revision: https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f4637468a8136f672e60ee0e531/src/yes.c https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f46...
  - pulse7 9y ago
    (joke) They copied the API of the main() function... :-)
  - h2hn 9y ago
    And here is the madness: :) https://github.com/coreutils/coreutils/commit/35217221c211f3116f374f305654462195aa634a https://github.com/coreutils/coreutils/commit/35217221c211f3...
    - alkonaut 9y ago
      And the motivation: it's used as test inputs. Not sure I agree with completely destroying the readability of a perfectly readable file - but ok.
      - acqq 9y ago
        The motivation, as far as I see, was only that it "may be used." But see the post of belorn where he argues that the error handling seems to be good.
emmelaich 9y ago
This is why Linux and GNU has won. It's just a trade off. For utilities whose behaviour doesn't change we're happy to improve the speed.
- kw71 9y ago
  IOW, Can't leave anything alone.
  - _jal 9y ago
    You're always welcome to port old, slow utilities forward. That's the beauty of open source!
    - kw71 9y ago
      This work certainly has value but it is frustrating to keep up with everything being changed. And if changes affect me do the fashions and attitudes of those making the changes have synchronicity with the way that I use computers? In the old days I think we would have left yes in c because the compiler will build it on any platform. My first experience with Linux is porting land.c to one of the commercial Unix. Now I first met TCP/IP on a 3B2 and later met systems sold by SMI and DEC and SCO and we all mostly constructed packets the same way and a guy called Stevens had written some nice books about this that everyone had. I think I recall the commercial and free BSD also did things the usual way. But whoever figured out this interesting phenomenon land.c demonstrated happened to be a Linux user and this platform had some different ideas about it. I saw rewriting the relevant parts as an annoying, menial task.
- hultner 9y ago
  And it's also a source of many bugs (in the past and likely in the future as well). In most use cases today I'd rather have a slightly slower userland which is easily read (and audited) then one which compromises quality for speed in edge cases.
masklinn 9y ago
According to the article/experiment you don't have to bork up the code much, just copy your stuff into a big buffer before you start printing it (and print using write(2) instead of puts)
- microtherion 9y ago
  This works easily for the default case, which prints "y\n" (two bytes), which is likely to divide BUFSIZ. To handle the general case with the same efficiency, you have to have a buffer size that is a multiple of both BUFSIZ and the length of the string to be printed. It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been).
  - masklinn 9y ago
    > It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been). Why would it be slower to do a single, say, 8190 bytes write instead of 2730 3-byte writes?
    - deleted 9y ago
      [deleted]
    - AstralStorm 9y ago
      Generally writes not aligned to cache line are slightly slower on most common architectures, vastly slower on others. (Such as many MIPS) Small write calls themselves incur a considerable syscall overhead.
zakk 9y ago
I am wondering why the GNU version uses atexit (close_stdout); Aren't all streams closed at exit? And why closing stdout, anyway?
- tyingq 9y ago
  It's to make sure the buffer is flushed. Streams are closed, but not explicitly flushed at exit. Stdout only because it's the only filehandle with a buffer in yes.
  - LukeShu 9y ago
    In normal use, GNU `yes` does unbuffered IO on stdout. However, it does use buffered IO for --help and --version messages; it sets atexit(close_stdout) to cover both of those cases at once, rather than handling them both separately.
scandox 9y ago
Yeah I got a surprise when I wanted to see how strlen was implemented: https://github.com/lattera/glibc/blob/master/string/strlen.c https://github.com/lattera/glibc/blob/master/string/strlen.c
- _wmd 9y ago
  If you're on amd64 that's the wrong file: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/strlen.S;h=b5ab117c79e1e26c6d43535d74e63515c258deb0;hb=HEAD https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86... The pcmpeqb instruction is from SSE 4, it compares 16 bytes per op
agumonkey 9y ago
I wonder how much of BSD is written in this canonical style. I think unix v6 was mostly as clear. Linux suffered the real world penalty. But maybe BSD managed to keep it's source poetic.
- hultner 9y ago
  In general I love reading the code base of BSD-systems, NetBSD in particular is really beautiful and easy to follow.
- realusername 9y ago
  I generally look at some BSD sources when I want to know how some unix tools are working, they are always much much more readable than the GNU equivalent.
gkya 9y ago
Mah, how many times do you read a program and how many times do you execute it?
belorn 9y ago
A nitpick, but I notice that the BSD implementation do not catch errors when printing. In theory it could get EINTR and only write partial amounts of argv[1], especially if the argument string is really long and the program runs for a extended amount of time. The GNU version do catch EINTR. Naturally it could be that the C function puts used by most people in OpenBSD is implemented with built in EINTR catching loop, or that OpenBSD do not interrupts writes.
- joosters 9y ago
  You'd have to check the various POSIX standards to be sure - and have to then verify that the OS/libc actually follows them, but you can pretty much rely on every libc's I/O wrapper functions to handle interrupted system calls or incomplete writes. I've never seen any code check the return value of a printf() to verify that all the characters were printed. As you say, the GNU versions definitely handle EINTR - the linux man page for puts() just says it returns a non-negative value on success, it's not even specified whether it returns the number of bytes written or not.
- LukeShu 9y ago
  `puts` is an "stdio" function, not a system call. It won't EINTR, it correctly resumes if the underlying write() EINTRs. If it does get an error, it will return EOF, then you'll have to call `ferror()` to find out which error. But the point stands; there's no error handling in the OpenBSD version. But that could be considered a design decision; the OpenBSD version never gives up on writing data until you kill it; the GNU version bails at the first error.
asveikau 9y ago
I think the GNU version is surprisingly readable for what it does. I have seen code that is a lot less readable than this and doesn't have the performance benefits described in the article.
mallaidh 9y ago
And for a third example, Busybox's implementation: https://git.busybox.net/busybox/tree/coreutils/yes.c https://git.busybox.net/busybox/tree/coreutils/yes.c

by exxo_ 9y ago

ceronman 9y ago

Normally I think readability is more important than speed. But in this particular case, I think GNU is doing the right thing optimizing the code to the limit. This is the beautiful part of Unix: small tools that do only one thing well. Programs following this philosophy are very good abstractions. They do one very well defined thing so you can use them without having to understand how they work. I have used Unix for years and I've never felt the need to read the source code for `yes`. And because they do a very small thing, even if you need to read them, the overhead of optimization is not that much, for example, the optimized GNU yes is just under 100 LOC if you remove comments and help boilerplate. Yes, it's longer than the BSD version, but it's just a matter of minutes to understands what it does.

IshKebab 9y ago
I totally disagree. Nobody will ever want to use `yes` at 10 GB/s. They will want it to be reliable, and this sort of over-optimisation increases the risk of bugs.
- avar 9y ago
  I'm having a hard time imagining a case where even a grossly complex 100,000 line implementation of yes(1) couldn't be trivially proven to be correct.
  - fredmorcos 9y ago
    Absolutely, I once trivially proved a 1000,000 line implementation of return 0 to be correct. I don't know why all comments are bothered by how much overkill this yes implementation is. Maybe they don't hold 100,000 PhDs like we do, am I right?
  - acdha 9y ago
    I agree with the general point that you could prove such a simple program correct relatively easily but that does still have a cost, which is always a concern in an open-source project. You still need someone to step up and do that work and continue to verify it periodically in the future – if that code is doing complicated things with buffering, that opens up possible odd bugs due to stdlib, gcc, maybe even kernel behaviour changes which might not affect simpler programs. Not a huge bit of work to be sure but for a non-commercial project you might have trouble finding a volunteer who cares about that tedium.
  - mikeash 9y ago
    Easy! Write a program that does a brute force check of Goldbach's conjecture on all integers. For each integer that passes the check, print a line. If you can prove this correct (or incorrect) you'll probably win a Fields medal.
- yk 9y ago
  The ability to get 10 GB/s dummy date into a pipe from the command line could come in handy at some point, for stress testing or something. I am not sure if it is over-optimized. (And even with the optimizations the risk of bugs should be very small.)
  - masklinn 9y ago
    Of course you could just pipe from /dev/zero which will easily do 10GB/s on every machine.
    - yk 9y ago
      Just tried it, cat /dev/zero|pv >/dev/null gives me roughly 2/3 the speed of yes. (5 GB/s vs 7 GB/s) Plus yes gives you an arbitrary string, instead of only zeroes.
      - tobik 9y ago
        That's because you're using cat and a pipe. Try this instead: pv > /dev/null < /dev/zero
        mikeash 9y ago
        For anyone who is like me and finds it uncomfortable that things are now out of order, note that you can still put the input redirection in front: </dev/zero pv >/dev/null
        vram22 9y ago
        Or even this: pv </dev/zero >/dev/null which is a common way of doing it (for any command with any inputs and outputs, not just the above ones), i.e.: command < input_source > output_dest All three pv command invocation variants, the one here and the two above, work. And it becomes more clear why they are the same, when you know that the redirections are done by the shell + kernel, and the command (such as pv) does not even know about it. So in all three cases, it does not see the redirections, because they are already done by the time the command starts running (with its stdin and stdout redirected to / from those respective sources). And that is precisely why the command works the same whether it is reading from the keyboard or a file or pipe, and whether it is writing to the screen or a file or pipe.
- joosters 9y ago
  I've used 'yes' many times to generate huge amounts of data quickly. Back then, it never had the small string optimisation, but you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/urandom or even /dev/zero I'm glad it runs fast, and I hope that all OS utilities are optimised (and tested, of course!) instead of making their source code pretty. The fact is, most people want to use programs, not read them.
  - masklinn 9y ago
    > you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/zero That really doesn't make any sense, /dev/zero should be at least as fast as yes.
    - joosters 9y ago
      /dev/zero should be at least as fast as yes I agree, all I remember is that when I tried it, /dev/zero sometimes sucked performance-wise. I can't recall the exact circumstances as it was some time ago, and could have been on any of Linux/FreeBSD/SunOS/HP-UX/IRIX - perhaps it was the fastest common way at the time? On a recent x64 Linux, /dev/zero seems plenty fast enough now: $ dd bs=8k count=819200 if=/dev/zero of=/dev/null 819200+0 records in 819200+0 records out 6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.331137 s, 20.3 GB/s $ yes | dd bs=8k count=819200 of=/dev/null 819200+0 records in 819200+0 records out 6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.959551 s, 7.0 GB/s
      - Tepix 9y ago
        No need for "dd", let "pv" get the data from /dev/zero directly: $ pv < /dev/zero > /dev/null [ 16GiB/s] But the version of yes using vmsplice() is even faster than that on my machine.
        pbhjpbhj 9y ago
        What's the line to test `yes` with `pv`? pv < /usr/bin/yes > /dev/null doesn't seem to work properly. FWIW I get 330MiB/s vs 8.4GiB/s for /dev/zero. [Incidentally first I've heard of pv but I've known about dd for a decade or two].
- belorn 9y ago
  If you want to maximize readability and simplicity then writing `yes` in C is a bad choice. It is much easier, cleaner and shorter to write it in python or just use the shell which would normally be using the `yes` in the first place. Since `yes` is used in shells and builtins can be considered very reliable, here is a implementation as a shell function: yes(){ while :; do echo "${1:-y}"; done; } Python: import sys while True: if len(sys.argv) > 1: print(sys.argv[1]) else: print("y") And if you don't need the features of `yes` and only need a loop that prints 'y', then there really is hard to beat the simplicity of: while:; do echo "y"; done
  - raptorfactor 9y ago
    Is it really "easier, cleaner and shorter to write it in python"? Did you look at the OpenBSD implementation? https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c It's essentially line for line identical to your python code...
    - fredmorcos 9y ago
      But it's C, and they're on HN, so C sucks, C should die, C is not modern, C is scary, C is bad, C makes them think, C forces them to be careful.
      - sctb 9y ago
        You've been on a tear of uncivil and unsubstantive commenting, and it has to stop. Often a good strategy for this is to slow down. High-quality reflective posts are what we're after instead of dismissive, snarky reflexive ones, and the former come more slowly.
    - belorn 9y ago
      The C example has three includes, two conditionals, two loops, and one function definition. The python example has a single include, conditional, and loop. For readability purpose it is easier to go through each lines of the python program than the OpenBSD C code. Its not massively different, but its distinguishable enough that I would choose the python version if I wanted to maximize readability, minimize syntax requirement and did not want to use shell script. The Shell function is in my view the superior choice if the audience is a programmer than know the shell script syntax. It is just a single loop and is written in the environment that the program is intended to be used in. The only drawback there is the speed.
- th4dv 9y ago
  > Nobody will ever want to use `yes` at 10 GB/s Just because you say so?
  - bastawhiz 9y ago
    If the speed of yes is bounded by memory speed, doing anything useful will almost certainly consume that data at a far lower rate. Putting it on a disk, pushing it over a network, etc. will almost always be slower than yes is able to generate data.
    - cestith 9y ago
      The typical use case is piping it into another running program. Maybe someone wants to do that really quickly rather than putting it on disk or pushing it over a network.
- MichaelBurge 9y ago
  'yes' to a file is how I sometimes benchmark disk speed. It should have fewer system calls than reading from /dev/zero and then writing. I actually checked just now, and it looks like you'd be making twice as many system calls with /dev/zero compared to generating the data locally: strace dd count=1 bs=512 if=/dev/zero of=test open("/dev/zero", O_RDONLY) = 3 dup2(3, 0) = 0 close(3) = 0 lseek(0, 0, SEEK_CUR) = 0 open("test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 dup2(3, 1) = 1 close(3) = 0 read(0,"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512 close(0) = 0 close(1) While 'strace yes > test2' is just a constant stream of write() calls. The difference matters if you're benchmarking e.g. some new SSD compared to a tmpfs on a machine with 100+ GB of RAM. It's always better if the tools have less overhead, because the comparison is more meaningful. Also consider that it can be faster to write to a local network than to disk. I've never done it, but I imagine that the kernel's not going to want to deal with your /dev/zero calls if it's spending all of its time writing to a 10GB switch. I can imagine some very specialized storage servers that could spend most of their time writing from memory buffers to a network switch, or if you're troubleshooting a slowdown in the networking itself.
  - wolfgang42 9y ago
    When I started this comment, I didn't think you were measuring what you thought you were measuring with those straces. 'strace yes > test2' only watches 'yes', not '> test2' (which is handled by the shell). Here's what your command outputs: $ strace yes > /tmp/test2 [various initialization steps] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [...] To measure everything, I started up a whole new shell, expecting to see a 'write(1, ...)', a 'read(1)', and a 'write(f, ...)': $ strace -f sh -c 'yes > /tmp/test2' [various initialization steps] [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192 [...] How does this possibly work? File descriptor 1 is supposed to be the terminal, not a file! Of course, the magic of file redirection: open("/tmp/test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 # Open the file, get FD 3 fcntl(1, F_DUPFD, 10) = 10 # Copy FD 1 (the terminal, STDOUT) to FD 10 temporarily close(1) = 0 # Close original FD 1 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 # Close FD 10 (the terminal, copy of STDOUT) when exec() is called dup2(3, 1) = 1 # Copy FD 3 (the file) to FD 1 (STDOUT) close(3) = 0 # Close FD 3 (the file's original descrptor)
- nobodyorother 9y ago
  Hey, 640K is enough for anyone!
- peterwwillis 9y ago
  Unix userland tools have evolved over decades to be as efficient as possible because they have historically underpinned everything the operating system does. The faster each tool works, the faster the other tools that depend on them work. If increased efficiency results in a bug, that bug can then be fixed, making it a net gain for system stability.
- tomohawk 9y ago
  It's not about running 'yes' at 10GB/s, it's about less overhead to do a simple job. If this version of yes is 100x faster, that implies it using 1% of a cpu to do the same work that would otherwise occupy 100% of a cpu. This leaves more of the machine to do what is likely to be the intended task.
YSFEJ4SWJUVU6 9y ago
It's not really optimized to the limit – or perhaps it is, but then the limit is fairly easy to reach. When I saw this item here I reached for the terminal and wrote a simple (and non-compliant) alternative that simply initialises 8150 bytes of "y\n" and then loops a write forever. I understand that it is not a fully standard-compliant yes, and that maybe GNU yes is indeed fast, but that awfully simple program that takes all of 10 lines (everything included) and took me all of a minute to write performs just as well as far as pv is concerned. (I eventually completed a feature complete yes but I still think that simply not using `puts` is hardly optimising to the limit.)

avar 9y ago

One thing to keep in mind when looking at GNU programs is that they're often intentionally written in an odd style to remove all questions of Unix copyright infringement at the time that they were written. The long-standing advice when writing GNU utilities used to be that if the program you were replacing was optimized for minimizing CPU use, write yours to minimize memory use, or vice-versa. Or in this case, if the program was optimized for simplicity, optimize for throughput. It would have been very easy for the nascent GNU project to unintentionally produce a line-by-line equivalent of BSD yes.c, which would have potentially landed them in the 80/90s equivalent of the Google v.s. Oracle case.

exxo_ 9y ago
This is the first revision: https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f4637468a8136f672e60ee0e531/src/yes.c https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f46...
- pulse7 9y ago
  (joke) They copied the API of the main() function... :-)
- h2hn 9y ago
  And here is the madness: :) https://github.com/coreutils/coreutils/commit/35217221c211f3116f374f305654462195aa634a https://github.com/coreutils/coreutils/commit/35217221c211f3...
  - alkonaut 9y ago
    And the motivation: it's used as test inputs. Not sure I agree with completely destroying the readability of a perfectly readable file - but ok.
    - acqq 9y ago
      The motivation, as far as I see, was only that it "may be used." But see the post of belorn where he argues that the error handling seems to be good.

emmelaich 9y ago

This is why Linux and GNU has won. It's just a trade off. For utilities whose behaviour doesn't change we're happy to improve the speed.

kw71 9y ago
IOW, Can't leave anything alone.
- _jal 9y ago
  You're always welcome to port old, slow utilities forward. That's the beauty of open source!
  - kw71 9y ago
    This work certainly has value but it is frustrating to keep up with everything being changed. And if changes affect me do the fashions and attitudes of those making the changes have synchronicity with the way that I use computers? In the old days I think we would have left yes in c because the compiler will build it on any platform. My first experience with Linux is porting land.c to one of the commercial Unix. Now I first met TCP/IP on a 3B2 and later met systems sold by SMI and DEC and SCO and we all mostly constructed packets the same way and a guy called Stevens had written some nice books about this that everyone had. I think I recall the commercial and free BSD also did things the usual way. But whoever figured out this interesting phenomenon land.c demonstrated happened to be a Linux user and this platform had some different ideas about it. I saw rewriting the relevant parts as an annoying, menial task.
hultner 9y ago
And it's also a source of many bugs (in the past and likely in the future as well). In most use cases today I'd rather have a slightly slower userland which is easily read (and audited) then one which compromises quality for speed in edge cases.

masklinn 9y ago

According to the article/experiment you don't have to bork up the code much, just copy your stuff into a big buffer before you start printing it (and print using write(2) instead of puts)

microtherion 9y ago
This works easily for the default case, which prints "y\n" (two bytes), which is likely to divide BUFSIZ. To handle the general case with the same efficiency, you have to have a buffer size that is a multiple of both BUFSIZ and the length of the string to be printed. It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been).
- masklinn 9y ago
  > It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been). Why would it be slower to do a single, say, 8190 bytes write instead of 2730 3-byte writes?
  - deleted 9y ago
    [deleted]
  - AstralStorm 9y ago
    Generally writes not aligned to cache line are slightly slower on most common architectures, vastly slower on others. (Such as many MIPS) Small write calls themselves incur a considerable syscall overhead.

zakk 9y ago

I am wondering why the GNU version uses atexit (close_stdout); Aren't all streams closed at exit? And why closing stdout, anyway?

tyingq 9y ago
It's to make sure the buffer is flushed. Streams are closed, but not explicitly flushed at exit. Stdout only because it's the only filehandle with a buffer in yes.
- LukeShu 9y ago
  In normal use, GNU `yes` does unbuffered IO on stdout. However, it does use buffered IO for --help and --version messages; it sets atexit(close_stdout) to cover both of those cases at once, rather than handling them both separately.

scandox 9y ago

Yeah I got a surprise when I wanted to see how strlen was implemented: https://github.com/lattera/glibc/blob/master/string/strlen.c https://github.com/lattera/glibc/blob/master/string/strlen.c

_wmd 9y ago
If you're on amd64 that's the wrong file: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/strlen.S;h=b5ab117c79e1e26c6d43535d74e63515c258deb0;hb=HEAD https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86... The pcmpeqb instruction is from SSE 4, it compares 16 bytes per op

agumonkey 9y ago

I wonder how much of BSD is written in this canonical style. I think unix v6 was mostly as clear. Linux suffered the real world penalty. But maybe BSD managed to keep it's source poetic.

hultner 9y ago
In general I love reading the code base of BSD-systems, NetBSD in particular is really beautiful and easy to follow.
realusername 9y ago
I generally look at some BSD sources when I want to know how some unix tools are working, they are always much much more readable than the GNU equivalent.

gkya 9y ago

Mah, how many times do you read a program and how many times do you execute it?

belorn 9y ago

A nitpick, but I notice that the BSD implementation do not catch errors when printing. In theory it could get EINTR and only write partial amounts of argv[1], especially if the argument string is really long and the program runs for a extended amount of time. The GNU version do catch EINTR. Naturally it could be that the C function puts used by most people in OpenBSD is implemented with built in EINTR catching loop, or that OpenBSD do not interrupts writes.

joosters 9y ago
You'd have to check the various POSIX standards to be sure - and have to then verify that the OS/libc actually follows them, but you can pretty much rely on every libc's I/O wrapper functions to handle interrupted system calls or incomplete writes. I've never seen any code check the return value of a printf() to verify that all the characters were printed. As you say, the GNU versions definitely handle EINTR - the linux man page for puts() just says it returns a non-negative value on success, it's not even specified whether it returns the number of bytes written or not.
LukeShu 9y ago
`puts` is an "stdio" function, not a system call. It won't EINTR, it correctly resumes if the underlying write() EINTRs. If it does get an error, it will return EOF, then you'll have to call `ferror()` to find out which error. But the point stands; there's no error handling in the OpenBSD version. But that could be considered a design decision; the OpenBSD version never gives up on writing data until you kill it; the GNU version bails at the first error.

asveikau 9y ago

I think the GNU version is surprisingly readable for what it does. I have seen code that is a lot less readable than this and doesn't have the performance benefits described in the article.

mallaidh 9y ago

And for a third example, Busybox's implementation: https://git.busybox.net/busybox/tree/coreutils/yes.c https://git.busybox.net/busybox/tree/coreutils/yes.c