9 ms·
A little story about the `yes` Unix command
- eru 4y agoCompare https://codegolf.stackexchange.com/questions/215216/high-throughput-fizz-buzz https://codegolf.stackexchange.com/questions/215216/high-thr... > *By far the best score so far is by @ais523 - generating FizzBuzz at a throughput that seems to average somewhere around 54-56GiB/s. The best 'yes' in the linked article gets about 3GiB/s. To be fair: the optimized fizzbuzz is insane.
- anderskaseorg 4y agoThere’s a fastest yes competition on CGCC as well: https://codegolf.stackexchange.com/questions/199528/fastest-yes-in-the-west https://codegolf.stackexchange.com/questions/199528/fastest-... My entry outputs 28 TB/s. (I…may have done some creative interpretation of the rules.)
- WelcomeShorty 4y agoA solution that requires the rules to be amended is obviously very creative! Impressive out of the box thinking.
- MattPalmer1086 4y agoAlways love reading about performance enhancements to code. This is a great example of how much you can do with such a simple thing. I can't see that any of this optimisation is actually useful in the real world. Does yes actually need such a high throughput? Would anything suffer if it didn't? Probably not. Still fun though.
- iforgotpassword 4y agoMaybe some day you want to automate installing a program that needs about three billion confirmations a second regarding various things...
- MattPalmer1086 4y agoLet's hope not, but good to know yes has me covered :)
- amelius 4y agoIt's cheaper than a lawyer, that's for sure.
- deaddodo 4y agoI think if you dig through GNU coreutils or *BSD world (probably FreeBSD for the “newest and fastest”), you’d be surprised at how complex almost any of the binaries are. That’s just the benefit of 40+ years of constant updates and optimizations on feature-complete software.
- pie_flavor 4y agoPSA: Do not use `yes` on MacOS CI. In those environments you only get one CPU, which `yes` will use at 100% and never release, and your entire workflow will hang. There is such a thing as too much performance, ironically.
- dmateos 4y agoOnce one of our devops engineers was testing a script with nohup yes > output. The /home directory was mounted to an autoexpanding EFS on AWS. 23.4tbs and 2 months later we noticed the bill :)
- pyuser583 4y agoDare I ask: how much?
- dmateos 4y agoNot too much, like $4k extra a month i think, noticeable but not the end of the world. I assume a smaller place would just beg AWS for forgiveness and probably get it.
- HellsMaddy 4y agoI wonder what the quickest way to rack up the highest bill on AWS is.
- maccard 4y agoLambda functions on a put event on S3 that also put an object _into_ an s3 bucket is so common it's called out on the aws documentation page [0]. [0] https://docs.aws.amazon.com/lambda/latest/operatorguide/recursive-runaway.html https://docs.aws.amazon.com/lambda/latest/operatorguide/recu...
- ninjin 4y agoI always enjoy reading these kinds of write ups digging into why something is as fast as it is and how it interacts with a wider system. However, I do think that one should not arrive at the belief that this kind of optimisation is warranted everywhere and that code simplicity can also be a goal. The classic argument is to compare OpenBSD’s yes [1] to GNU coreutils’ yes [2] and contemplate on under which circumstances those additional MB/s of “y” will be critical enough to warrant the maintenance of more than one hundred additional lines. [1]: https://cvsweb.openbsd.org/src/usr.bin/yes/yes.c?rev=1.9&content-type=text/x-cvsweb-markup https://cvsweb.openbsd.org/src/usr.bin/yes/yes.c?rev=1.9&con... [2]: https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/yes.c https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/yes... Similarly, there is the NSFW comparison between Plan 9’s cat and GNU coreutils’ cat [3]. [3]: http://9front.org/img/longcat.png http://9front.org/img/longcat.png Just to reiterate in the end though. This is not an argument against optimisation and learning how to make something blazingly fast. But, there is such a thing as optimising the wrong thing and using speed as the only justification for merging a patch is probably not the right thing to do for a bigger project.
- pixelbeat__ 4y agoI carefully considered this before I optimized GNU yes. The reason it is useful is because yes can output anything, and so is useful to produce any repeated data for test files etc. You can see the justification detailed in the original optimization commit: https://github.com/coreutils/coreutils/commit/35217221 https://github.com/coreutils/coreutils/commit/35217221
- dncornholio 4y agoSo you actually wrote a new program. Yes was made for those pesky installers, not for producing large amounts of data. This would be my approach. Keep yes simple and create a new program that does spilling data well.
- jraph 4y agoAnd then you have a possibly confusing situation where you have two programs that do essentially the same thing, but one is faster, and the other one is possibly not provided by default. As a user, in cases it matters, you'd have to know the issue and bother installing the new program. This is worse. As developers, I think it's our duty to make life of users simpler, even if it makes our lives a bit more complicated. I'd argue that's what we are here for. I guess there's no ideal solution. But I think the "new" program does what the first one did better, and does not do anything worse. We are talking about a program that still under 1000 lines of code and that's not getting new features every month, or at all anyway, so maintainability does not seem to be a big issue? I see the reasoning but I don't see any actual practical drawback to have improved the original program directly in this specific case. I don't see any advantages of keeping "yes" dead simple neither. The new version is still pretty much readable and the extra time it takes to read it and modifying without doing mistakes seems worth the advantages.
- einpoklum 4y agoyes!
- tombert 4y agoIt always amazed me how fast “yes” is. Whenever I need to generate a really big file for whatever reason (e.g. testing multipart file uploads), I will run something like “yes > myfile.txt” for a few seconds. Similarly, I tried writing a naive version in node.JS a few years ago and it was several orders of magnitude slower than coreutils…, which is usually the case whenever I try and clone one of the “simple” utilities from coreutils.
- giuscri 4y ago`cat /dev/urandom > /tmp/myfile` is faster on macos though
- tombert 4y agoI suspect that just doing a `truncate` command is probably even faster. I just like the `yes` version because it requires basically zero thought on my end.
- nacs 4y ago`yes` is far faster on my Linux (2.2GB/s) vs /dev/urandom (0.38GB/s) -- not surprising since the `man` page for urandom says: "[urandom] is designed for security, not speed, and is poorly suited to generating large amounts of random data" `cat /dev/zero` is pretty much as fast as `yes` on my system however (2.1GB/s)
- NoboruWataya 4y agoI just tried "yes > myfile.txt" for a few seconds (less than five) and then checked the size. 1.4 billion lines! I knew yes was fast but that's pretty mind-blowing.
- tombert 4y agoThe GNU coreutils never fail to impress me with how insanely fast they go. One of my favorite blog posts of all time [1] really opened my eyes to how insanely optimized Unix really is. [1] https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- birdyrooster 4y agoWell that was a bunch of nonsense, cute I guess.
- Bancakes 4y agoRelated: How is GNU `yes` so fast? https://news.ycombinator.com/item?id=14542938 https://news.ycombinator.com/item?id=14542938
- a_c 4y agoToo lazy to figure out, since python's print is already buffered, is there anything we can do to beef up the python print() throughput?
- Damogran6 4y agoIt’s also a great way to synthetically max out a core. Want to check thermals? Got 8 cores? Spin up 8 yes > /dev/null And let them rip.
- jeffbee 4y agoFor amusement launch it with `yes | head -n 8 | xargs -n 1 -P 0 yes` This is how I thermally stress CPUs but I usually do `yes speed | head -n 8 | xargs -n 1 -P 0 openssl` instead.
- joveian 4y agoConvenient for slight stress testing, but if you want maximum thermal generation there is no substitute for FIRESTARTER: https://github.com/tud-zih-energy/FIRESTARTER https://github.com/tud-zih-energy/FIRESTARTER With my system hooked up to a watts up (but including the monitor and a couple of other things) yes > /dev/null on each core gets a bit above 42w, openssl speed on each core occasionally gets above 45w and running FIRESTARTER for a bit gets above 57w (on an i5-6260U (NUC) with hyperthreading disabled, 15W TDP with some attempted power restraint in the BIOS, firefox on decent websites like HN tend to use about 31w and I think that is something like 10-15w (I think 12w but it has been a bit) measuring the computer only). FIRESTARTER has some evolutionary algorithms as well, but at least on my CPU after hours they were still doing much worse than the default. I was wondering why there wasn't much discussion of BIOS power settings that I could find and after some testing found out that they are not effective at restraining max power use (I forget if they had any effect on typical power use either but I don't think it was much if they did). Also, the integrated GPU can use more power than the CPU and can't be limited. For that GpuTest is handy (but not open source): https://www.geeks3d.com/gputest/ https://www.geeks3d.com/gputest/ For me on the internal GPU, Pixmark Piano and Furmark use the most power, either can get 60-61w and adding FIRESTARTER in the background only adds another w or two. Similarly, checking temps with turbostat (PkgTmp), 2x yes seems to max out about 70, testing one of the higher power openssl tests on each core reaches 75, FIRESTARTER alone quickly reaches 80 and slowly ramps up to 90, and adding gputest got up to 96. Interstingly, the max temperature is reached when the CPU dethrottles too quickly after the gputest is done. It takes a few gputests alone in a row to get into the 80s (I got bored after 2x each alternating Furmark and Piano that hit 83). Similarly, looking at power useage with turbostat the max PkgWatt with yes is 8.26, with openssl speed ecdsa 9.88, with FIRSTARTER alone 17.62, + gputest 19.97 (or gputest alone, seems unreliable). Anyway, this is a long diversion to say that even fancy yes or openssl speed tests are not that great as CPU stress tests :).
- secondcoming 4y agoI’m really surprised that rewriting in Rust didn’t result in optimal performance immediately.
- kibwen 4y agoAny use of `println` in Rust is going to have its performance dominated by repeatedly locking stdout and flushing the buffer every time. It's a bit of an obscure fact (since most programs aren't bottlenecked on printing), but `println` in Rust is kind of just supposed to be a convenience for hello world and print-debugging. Rust provides `writeln` for serious use, which is more verbose but allows you more fine-grained control over locking, buffering, and error-handling. See also https://nnethercote.github.io/perf-book/io.html https://nnethercote.github.io/perf-book/io.html
- xpmatteo 4y agoThe simplest Unix command is arguably "true"
- gspr 4y ago… which the article also points out just two sentences in!
- f0xJtpvHYTVQ88B 4y agoGNU coreutils `true` is 80 lines https://github.com/coreutils/coreutils/blob/master/src/true.c https://github.com/coreutils/coreutils/blob/master/src/true.... Busybox `true` is 38 lines https://github.com/mirror/busybox/blob/master/coreutils/true.c https://github.com/mirror/busybox/blob/master/coreutils/true... The simplest implementation is 0 bytes https://twitter.com/rob_pike/status/966896123548872705 https://twitter.com/rob_pike/status/966896123548872705
- 2OEH8eoCRo0 4y agoNice writeup but I have a question: Why does `yes` need to be fast?
- reportgunner 4y agoWhen all you have is a hammer, everything looks like a nail.
- ninkendo 4y agoWhy would you want buffer in the yes command though? The point is to only provide output if stdout is being read (ie. stdout is available for writing.) That way you only provide as many lines of output as is needed. Isn’t filling a buffer going to be a huge waste of CPU for the common case where the command only needs to provide one or two “yes”es? (Common case as in, its intended use of working around scripts that interactively prompt you to continue, etc…)
- dolmen 4y ago1. Performance of yes is not a concern for that "common case" (well, is that case still common? who writes only interactive program that require yes/no anwers from the terminal?). 2. Filling a buffer has much less CPU (power) cost as it doesn't involve context switching like I/O.
- mignato 4y ago[flagged]
- grymoire1 4y agoIf you ever had to repair a disk using fsck when it had hundreds of errors, you would understand why yes(1) was created.
- exabrial 4y agoIs there a was to remove this off Debian? Caused more problems than and I’ve never used it for what it was designed.
- barbs 4y agoI suppose you could add an alias that does nothing instead?
- WhyNotHugo 4y agoOn my Asahi-Linux system, the rust version runs at about 15GB/s, but the gnu version runs as "just" 10GB/s. However, the author had the opposite results. I wonder if the architecture makes a difference.
- tymscar 4y agoCould you try running it on macOS on the same machine? I wonder if that would be any different.
- jepler 4y agoThis 'yes' implementation in Python also gets to the GB/s range on Linux, approximately on par with gnu yes: https://gist.github.com/jepler/5e46f4e46542367d75cde9d77d5865fe https://gist.github.com/jepler/5e46f4e46542367d75cde9d77d586... It takes the useful insights from this article and packages them back into a Python program. But there's so little actual Python code running that the efficiency relative to a compiled language is swamped by all the data copying to the kernel. vmsplice()ing is probably the next step in speed, however, vmsplice apparently doesn't fit well into the semantics of python (or probably rust) programs. trying to reduce the number of python bytecodes by calling `os.writev(1, many_bufs)` actually harmed performance, not sure why.
- JustinGarrison 4y agoI was setting up centralized logging on a bare metal Kubernetes cluster (500 GB HDD IIRC). I ran a deployment that ran the yes command to generate logs at the end of the day. When I returned to work in the morning, 2 of the nodes were unhealthy because their hard drives were full. A couple years later I was load testing Datadog logging from a cluster and also used yes (no logs on disk this time). I used the daily log quota for the entire Datadog account in 30 minutes.
- asciii 4y agoGreat article. I tried the command for a brief second with output to a text file. I cannot believe that on my M1 after 1-2 seconds had 2684272500 lines amounting to 5GB. Yikes
- deleted 4y ago[deleted]
- sbf501 4y agoComparing the 1979 version with the current version deserves its own whitepaper. Writing portable, robust code is nontrivial. This is a great example. For those who didn't RTFM: 1979 version: main(argc, argv) char **argv; { for (;;) printf("%s\n", argc>1? argv[1]: "y"); } Current 128-line GNU version: https://github.com/coreutils/coreutils/blob/master/src/yes.c https://github.com/coreutils/coreutils/blob/master/src/yes.c
- KerrAvon 4y agoI would argue this is pointless optimization for the sake of looking good on an artificial benchmark. Nobody needs a 3GB/s `yes` command! Modernize the syntax to C89 and the 1979 version is perfectly fine for any non-contrived use case. What would be way more interesting is fixing the standard library/OS interfaces to make the slow version fast, because that would likely benefit other simple filter commands as well. Might require using non-POSIX interfaces to do well, of course. edit: POSIX note
- sbf501 4y agoPutting aside locale, and other important things universal code should have in the first 75% of the new version: What is your argument that it is silly? Have you done a survey of usage, or researched its history. Don't be so quick to criticize code, especially from a huge, old, project (that might even be older than you).
- operator-name 4y agohttps://www.gnu.org/prep/standards/standards.html#Reading-Non_002dFree-Code https://www.gnu.org/prep/standards/standards.html#Reading-No... > Don’t in any circumstances refer to Unix source code for or during your work on GNU! (Or to any other proprietary programs.) > If you have a vague recollection of the internals of a Unix program, this does not absolutely mean you can’t write an imitation of it, but do try to organize the imitation internally along different lines, because this is likely to make the details of the Unix version irrelevant and dissimilar to your results. > For example, Unix utilities were generally optimized to minimize memory use; if you go for speed instead, your program will be very different. You could keep the entire input file in memory and scan it there instead of using stdio. Use a smarter algorithm discovered more recently than the Unix program. Eliminate use of temporary files. Do it in one pass instead of two (we did this in the assembler). > Or, on the contrary, emphasize simplicity instead of speed. For some applications, the speed of today’s computers makes simpler algorithms adequate. > Or go for generality. For example, Unix programs often have static tables or fixed-size strings, which make for arbitrary limits; use dynamic allocation instead. Make sure your program handles NULs and other funny characters in the input files. Add a programming language for extensibility and write part of the program in that language. > Or turn some parts of the program into independently usable libraries. Or use a simple garbage collector instead of tracking precisely when to free memory, or use a new GNU facility such as obstacks.
- jedberg 4y agoBack in the 90s I worked in a shared linux office environment. Sometimes people would leave themselves logged in, so of course it was your duty to mess with them if you found such an egregious security violation. Once we found one logged in and decided to add 'yes' as the last line of his bash login. That day we learned, via an angry phone call, that you can't Ctrl^C out of 'yes' via a remote shell. Today after reading this I'm assuming it's because of the way it floods the buffer. Poor guy had to call the one dude with root access to fix it for him.
- js2 4y agoI recall in college we'd `xhost +` and then start running things remotely on their desktop, playing sounds, etc, but it's been a very long time. Switching the keyboard layout to Dvorak is another fun one.
- spurgu 4y agoRecent HN discussion (2 months ago): https://news.ycombinator.com/item?id=31619076 https://news.ycombinator.com/item?id=31619076
- mpweiher 4y ago6.4 GB/s #include <libc.h> #define BUFSIZE (512*1024) static char buf[BUFSIZE]; int main(void){ memset(buf,'y',BUFSIZE); while (1) { write(1,buf,BUFSIZE); } return 0; }