12 ms·
When setting an environment variable gives you a 40x speedup
- CaliforniaKarl 8y agoThis was written by my coworker, Kilian Cavalotti. We have two main file stores: $SCRATCH is a Lustre cluster, which is extremely performant but doesn't do well with lots of inodes. And $HOME is a multi-node Isilon, which can handle all the inodes, but is not as performant. And we have users who sometimes like to put many files in single directories. You can find more information about us at https://srcc.stanford.edu https://srcc.stanford.edu And you can find more information about Sherlock at https://www.sherlock.stanford.edu https://www.sherlock.stanford.edu
- PunchTornado 8y agowhat is an Isilon? thanks
- Tushon 8y agohttps://en.wikipedia.org/wiki/Dell_EMC_Isilon https://en.wikipedia.org/wiki/Dell_EMC_Isilon
- loeg 8y agoAnd more on the product in particular: https://en.wikipedia.org/wiki/OneFS_distributed_file_system https://en.wikipedia.org/wiki/OneFS_distributed_file_system Disclaimer: I work for Dell.
- anardil 8y agoHyperscale Network Attached Storage! Disclaimer: I work on Isilon
- wahern 8y agoWhy was iSCSI support deprecated? I recently developed an interest an iSCSI, which makes me curious; not an Isilon user, though.
- CaliforniaKarl 8y agoMy guess would be that Isilon/OneFS is designed for file storage, instead of block storage, so having a block storage layer on top (like iSCSI) seems a little ungainly. Instead I expect Dell/EMC would prefer you investigate the VNX platform! Edit: By "file storage", I'm talking about storage mounted using protocols like SMB and NFS.
- JdeBP 8y agoThat's actually a disclosure, not a disclaimer.
- xenospn 8y agoClustered storage.
- arendtio 8y agoThanks for the additional information but sometimes I wonder that 15k+ files count as 'many' nowadays. I mean, an off-the-shelf Canon DSLR produces 20k files per directory (10k JPG + 10k raw). Nevertheless, I did some test your approach on my PC: On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance. Without modification, it takes 0.5 seconds to list a directory (on a slow consumer grade HDD) with 14448 files and after setting export LS_COLORS='ex=00:su=00:sg=00:ca=00:' it takes 4.4 seconds. However, listing a local (SSD) directory with 15k files takes just about 0.12 seconds and gets faster after settings the variable to 0.06 seconds. For all tests I drop the caches before testing, e.g.: echo 3 > /proc/sys/vm/drop_caches; time ls --color=always /mnt/nfs/many | wc -l
- mrb 8y ago"On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance" Run "strace -o logfile ls --color=always" and diff the two logfiles.
- arendtio 8y agoWell, this looks rather strange. As expected, settings the variable causes large changes as it removes a lot of lstat calls and also removes the color codes from the writes. So there seems to be no obvious explanation, why it becomes slower when doing less... # du -sh logfile_nfs* 1.7M logfile_nfs 668K logfile_nfs_exported However, the strace -c output looks kinda different: Without modifying the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEbtark/TMSDm9gT+jT0OK5ME8fE= https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb... With setting the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEbtark/TMSDm9gT+jT0OK5ME8fE= https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb...
- mrb 8y agoYou pasted the same link for both cases. Can't tell what's going on.
- 8y ago
- netsharc 8y agoTell your colleague I think emojis make the article look like it was written by a 12 year old. IMO. I guess I'm just an old fart.
- Faaak 8y agoI think they add a bit of emotion to the text written. It's not because it has colours that it's not a serious article.
- yani 8y agoIt is how most millenials write nowadays
- falsedan 8y agoIt’s how people trying to write like millennials write Postfix emoji, eurgh
- scrollaway 8y agoEven if it were, maybe you should have more respect for 12 year olds. IMO. I guess I'm not an ageist.
- gnufx 8y agoI don't understand the "lots of inodes" comment, but any filesystem limit on inodes isn't relevant to this, which is just due to RPCs to the metadata server(s). (ls -l is actually worse than coloured plain ls because the size data are on the OSS, not the MDS.) The canonical advice for general performance is to keep directories reasonably small on Lustre due to possible lock contention, but I don't know the circumstances for which that's actually relevant. [Metadata operations, such as large builds and tars, are typically slower on our Isilon than on the Lustre filesystem, which has no serious tuning as far as I know.]
- jabl 8y ago> The canonical advice for general performance is to keep directories reasonably small on Lustre due to possible lock contention, but I don't know the circumstances for which that's actually relevant. I helped a researched debug a Lustre performance issue a while ago. Each job was nothing special, read a few files (maybe a few GB total), do some (serial, no MPI or such) calculations taking maybe 10 min or so, produce output files, again a few GB. No problem, except when the person ran a several hundred of them in parallel as an array job the throughput per job dropped to a small fraction of normal. Turned out that all the jobs were using the same working directory. Slightly tweaking the workflow to have per-job directories fixed it.
- jf 8y agoThe tl;dr here is that "ls" can be much faster if you disable colorizing files based on the their file capabilities, setuid/setgid bits, or executable flag. LS_COLORS='ex=00:su=00:sg=00:ca=00:'
- neilv 8y agoAnd if you don't want angry fruit salad colors in your cool hipster retro-Matrix semitransparent tiling-window desktop theme: unset LS_COLORS alias l='/bin/ls -a' alias ll='/bin/ls -alF'
- Symbiote 8y ago> unset LS_COLORS Quoting from the article, "It turns out that when the LS_COLORS environment variable is not defined, or when just one of its <type>=color: elements is not there, it defaults to its embedded database and uses colors anyway."
- neilv 8y agoFrom looking at a version of the likely C source (`coreutils` source, as packaged by Debian), it looks like they're aggressively trying to use color. In addition to `LS_COLORS`, they also look at `COLORTERM`. There's also C variables to chase through the source; `print_with_color` seems most likely one to focus on. In addition to that code, the shell init scripts (both system-wide, and user-specific default ones that are copied in) might do things like checking whether a variable is set, and setting it, if not (or simply overriding, regardless).
- addicted 8y agoI’d argue that for the vast majority of manual usages of ls, the maybe second or so saved listing directory contents by removing colors will be dwarfed by the additional time it takes the human to parse its contents, without the additional context provided by those colors.
- neilv 8y ago
- codezero 8y agoI started reading this thinking it would talk about LC_ALL/locale related, but not too surprised to see another env variable throwing a wrench in the works.
- nisa 8y agofor context you refer to the fact that LC_ALL=C grep string hugefile is often magnitudes faster (also works for other shell utilities) - this is due to unicode handling that is either complex due to the nature of unicode or just performances badly on glibc - probably it's the complexity.
- jschwartzi 8y agoSeems analogous to xml parsing in Perl, where I realized a 10x speedup in parsing by switching from a general-purpose XML parser to a set of regular expressions.
- sk5t 8y agoUsing regex on serialized XML has the potential to go off the rails thanks to the wiiiide variety of ways to express character entities, import docs, use namespaces, escape or not, etc.; consider an event-y/SAX parser for performance if you might run afoul of such issues.
- jschwartzi 8y agoUnless you know exactly the format of the data you're getting back. So in my use case it's not arbitrary XML but rather it's constrained to a particular message with specific fields that are named in a specific way. So it's really a text response that looks like XML rather than being some arbitrary XML message that differs after each response. We're literally getting the exact same XML message back, but with different text in each field. The framework we were using insisted on parsing each response separately even though the responses were basically the same, but with different values in the exact same XML tags.
- _bxg1 8y ago"he mentioned his laptop was 1,000x faster than Sherlock to list this directory’s contents" "that’s a 40x speedup right there" So... what about... the rest?
- Bukhmanizer 8y agoI assume they were speaking colloquially.
- kilianc 8y agoGood point! But sometimes a large-scale parallel filesystem can't beat a local SSD on certain patterns. Also, we didn't see any timings for that 1,000x claim, so let's say it's an estimate. :)
- emmelaich 8y agoIt's a great article, thanks. But .. do people actually use `ls` to list so many files? I mean, it'd scroll off the terminal anyway.
- chris_wot 8y agoI admit to being guilty of the odd for i in $(ls) do Something dumb done
- myrandomcomment 8y agoIt is like you read my script.sh over my shoulder as I was writing it. :)
- zaphirplane 8y agoOut it interest why not xarg or find’s exec
- freedomben 8y agoNot GP, but I often find myself writing making statements (which is awkward with find), or using syntax that find struggles with.
- chapium 8y agotl;dr version LS_COLORS='ex=00:su=00:sg=00:ca=00:'
- franciscop 8y agoI thought this was about setting NODE_ENV=production, and reading other comments others know different ways. Seems there are many ways to get a system an order of magnitude faster with little, but deep work/knowledge.
- drb91 8y agoWhy are people so lazy at naming? https://en.m.wikipedia.org/wiki/Sherlock_(software) https://en.m.wikipedia.org/wiki/Sherlock_(software) What ever happened to meaningful names that described the particular, not the meme?
- brokensegue 8y agolike GNU?
- drb91 8y agoI refuse to defend any gnu or unix naming conventions.
- ngcc_hk 8y agoWow. Miss this for decades. Life waste so much on colour.
- rawrmaan 8y agoI really enjoyed the writing style. This was fun to read. Thanks!
- chapium 8y agoI feel the opposite, I'd rather they get to the point in the first paragraph then follow it up with casual examples of how they arrived at the solution.
- kilianc 8y agoTrue, that could have used a TL;DR, agreed.
- CaliforniaKarl 8y agoBut then people would miss the rainbows!
- whalesalad 8y agoThe irony here is that instead of just unsetting the alias that is proxying `ls` to `ls --color=auto` the solution was to leave the coloring in place but try and hack the override env var for defeating that behavior. ie, `var foo = 1 + 1 - 1` instead of `var foo = 1` Wouldn't it have been more straightforward to open one's .bashrc/.bash_profile and remove the alias?
- marvy 8y agoSure, but they WANTED colors. The just wanted them to be fast.
- kilianc 8y agoPrecisely. We want colors. We're refined people.
- freedomben 8y agoThe article mentioned that they knew this, but wanted to continue supporting as much color as possible.
- ehsankia 8y agoThe ideal solution would be to have something that disables coloring if there's more than X files. Someone with more linux-fu can probably come up with such a macro.
- laughinghan 8y agoHow is that more ideal than having coloring even on more than X files but still being fast?
- vagab0nd 8y ago> Having thousands of files in a single directory is usually not very file system-friendly, and definitely not recommended. I wonder, is this specific to the situation when I use "ls", or is it in general? I recently worked on a project where I need to store many small files on ext4, although these files are not read/written by human. I came across suggestions to group files into subdirectories and not put them all in one directory. Is there evidence that it's actually worth it on a modern filesystem? EDIT: by "many" I mean 10+ million files.
- tyingq 8y agoAt the very least, it would be more polite to anyone that might need to support, troubleshoot, remove the files with shell tools after removing your software, etc. I suspect it's also still a performance issue with many popular filesystems. See https://lwn.net/Articles/606995/ https://lwn.net/Articles/606995/ for more info.
- mcbits 8y agoFor future reference, next time you need to store millions of related small files, it might be worth checking if SQLite is an option instead of using the filesystem: https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html Performance aside, I wish NodeJS had a way to pack all its little turds into one SQLite file, out of sight. (Admittedly I hadn't thought to look for such a thing until now... Preliminary results are negatory.)
- civility 8y agoYou should benchmark SQLite before recommending it like this (for future reference). I think it's a very elegant library, and I admire the development philosophy that went into it. However, every time I've tried to use it for something that needs to be fast it ended up being a painful mistake.
- striking 8y agoCould you show us your methodology for your benchmark? We don't know under which circumstances you experienced degraded performance. I've seen a lot of people just missing an index or something, and their DB then ran fine, so that's why I ask.
- mrb 8y agoStory time. This took place around 2009. Back then I was working for Rapid7, on their network vulnerability scanner Nexpose. The codebase was mostly Java and was relatively large at 1M+ lines of code. We had many unit tests. Running the entire test suite took up to 40 minutes on Windows, and 20 minutes on Linux (Windows was always about twice slower on everything: building, product's startup time, etc.) The company had grown quickly to at least 30-50 software engineers. The problem was that every time one of them ran a build on his or her local machine (which happened multiple times a day) it would have to run the test suite and waste up to 40 minutes of this person's time. 40 minutes × dozens of engineers = lots of inefficiencies in the company. I loved solving performance issues so one day I remember arriving at the office and making it my mission to investigate if there was an easy way to speed up the test suite. Our build system was based on Ant, and our tests used the JUnit framework. After a little time profiling Ant with basic tools (top, strace, ltrace), and taking a few Java stack traces, I realized that most of the wasted time was not actually running the individual tests, but many instances of the JVM kept being started and destroyed between each test. Our Ant build file was running the JUnit test with fork=yes, which was required for a reason I don't recall at the moment. This forks the JVM for running the tests. Then a little googling lead me to this: https://ant.apache.org/manual/Tasks/junit.html https://ant.apache.org/manual/Tasks/junit.html While reading this documentation, I stumbled upon an unknown parameter to me: forkmode. What does it do? "forkmode: Controls how many JVMs get created if you want to fork some tests. Possible values are perTest (the default), perBatch and once. once creates only a single JVM for all tests while perTest creates a new JVM for each TestCase class. perBatch creates a JVM for each nested <batchtest> and one collecting all nested <test>s." Our Ant build file did not set forkmode, so it meant we were forking a new JVM for every test! I immediately tried forkmode=perBatch and... the test suite ran 10× faster! 40 minutes down to 4 minutes on Windows. And Linux ran it in 2 minutes instead of 20 minutes. I told my boss right away but he was unbelieving. He asked that I check with our most-experienced Java developer. I showed him my 1-line patch speeding the test suite 10× and he said "I guess you are right, we can commit that." By lunch time the fix was committed and everyone loved me :)
- quickthrower2 8y agoThat’s the time to ask for a raise!
- natmaka 8y agoAn often-neglected related trick is the "dir_index" feature of ext3/ext4
- tyingq 8y agoI suspect something like "ls | cat" would speed it up as well. Probably tests if stdout is a tty before bothering with stat() and color logic.
- kilianc 8y agoTrue, but then you would loose all the coloring. By targeting only the attributes that generate additional syscalls, you can keep the vast majority of the colors and get a nice speed bump. Win win.
- cespare 8y agoThe conclusion seems unsatisfactory. Why is lstat so much slower on their system than on the laptop?
- mbuna33 8y agoCompute clusters generally use distributed file systems, over the network...
- CaliforniaKarl 8y agoExactly that. We have 2000+ users, and 1000+ compute nodes. Each compute node does have a certain amount of local SSD (`$L_SCRATCH`) for job-local storage, but for everything else we have to use some sort of network-accessed file system. For us, that is NFSv4 for longer-lasting data (`$HOME` and `$GROUP_HOME`) to the Isilon over Ethernet, and short-term data (`$SCRATCH` and `$GROUP_SCRATCH`) to Lustre over Infiniband. (Yes, Infiniband! Hello, Mellanox!)
- markmm 8y agoRan it on my Fedora laptop and it took 0.02s for 10k files. No changes were necessary.
- fsniper 8y agoI was expecting some new flag about getdents bigger buffer size. http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html http://be-n.com/spw/you-can-list-a-million-files-in-a-direct...
- obhmt 8y agoAren't you a bit old to use that many emojis?
- HocusLocus 8y agoFave speedup story, in the 80s a good friend had written a custom accounting system in CBASIC and he came to me in regards to one client, a boat charter service management business that maintained a set of 'virtual double-entry books' for every one of its 200+ member yachts. The monthly process was taking 4 hours to complete. Could I improve on that? Well in a couple hours I discovered that his routines assembling the main batch file did repeated lookups in other files without caching any operation results, and because it was MS-DOS the OS didn't cache many sectors either. So the hundreds-of-thousands of preparatory operations were waiting for the hard disk platter to come around again, for each. Yes, even hard disk platter speed was a significant factor in those days. So I added a 15 element array that cached lookup operations in memory. From 4 hours to 15 minutes, Thank You Very Much.
- agumonkey 8y agoMay I contribute with an hybrid optimization. Some workers were tasked to remove duplicate in excel files by hand. They'd have to create filtered view for each entry, Excel being slow it took hours. Some woman had a 3000 lines spreadsheet and was about to lost it. I wrote a 8 lines VBA to count duplicates, just enough so she could filter anything that had count > 1 and then delete things on her own. Turning multi-day job into a two clicks operation. Probably my most happy lines of code in my entire life.
- Twirrim 8y agoMy favourite speed up never got used :( Had a vendor we got partnered with for a government contract. The company I worked for usually did all its own development of java based web applications for the state, but on this one it was decided to partner with someone else as they already had a solution in the space. We'd just run it for them. It was terrible. The UI/UX was fine. It did actually solve the problem for the state, and was pretty intuitive. On the software side, though it just flat out did not scale for a number of reasons (including the same one OP had, they loved to dump everything in a single directory). My particular favourite reason was the way they chose to produce reports. They'd take the business requirement, sit there with a visual database tool, and piece together a SQL statement to produce the report, no matter how complicated that query got. Then they'd drop that in their code and voila, effortless reporting! There were queries in their code base that would end up nesting some 40+ selects deep, e.g. SELECT foo FROM bar WHERE id = (SELECT id FROM monkey WHERE id = (SELECT ... and so on down the line. The MySQL query optimiser, at the time, didn't handle those queries very well, and it could end up taking 45 minutes or so to produce the final report, and it was showing signs of potentially exponential execution time growth. This was around the time MariaDB was starting to take off, and it turned out this issue was one of the first they'd fixed in the optimiser (MySQL followed suit with a completely overhauled query optimiser they'd been working on for a while), and so I switched over to MariaDB instead of MySQL. Free speed boost! Now it only took 10 minutes to produce the report! Everyone loved me... but I still wasn't happy. The final reports didn't look that complicated to me. I picked up the most egregious example, and rewrote it in perl (the only language I wrote in at the time). Along the way I identified that if they wanted to keep it in a single query they could easily cut it down to something significantly simpler, but also if I just made it four or five queries and wrote some _very_ simple logic around it in perl, I could get the report out within seconds, while drastically reducing load on the database server. I gave the developers the proof of concept and simplified SQL query, but neither got used. In the end the company that produced the software was facing bankruptcy, they couldn't support what they'd sold the state, at the price they'd sold it for. They weren't ready for that scale, didn't know how to code to that scale, and didn't really have a grasp of the costs of supporting at that scale. For various reasons, my employer ended up taking over the code base from them, with legal agreements that we'd only use it for this purpose, never sell it anywhere else etc. etc. The code base was _awful_. I thought the things I could see (queries) were bad. The code base was worse. Thankfully that was someone else's problem to work on :D
- deleted 8y ago[deleted]
- gnufx 8y agoIt may be worth mentioning that this sort of thing isn't just a problem for whoever is running it, because it may have a bad effect on other users if you hammer the metadata server(s). It's necessary to have monitoring which can identify the jobs responsible and take action. The Lustre system here was clobbered for several days because the admins had no means of doing that, despite all the facilities provided for and by Lustre.
- trollied 8y agoI was once flown out to look at a significant performance problem. The customer could not do a days worth of work in a day. We’d told them to buy a fully loaded Sun E15k, and it couldn’t keep up. I instrumented everything (recompiled everything so that the binaries spat out gprof data, iostat output, vmstat output, sybase monitoring) and it quickly became obvious that the DB driven config was configured in a way we’d not anticipated. The fix was easy (just use a btree in a couple of places instead of a linear array walk), but then we had a huge problem. With the fix, the CPU bottleneck became a 15% CPU utilisation. The system had been sized based upon our advice. Whoops. What happened next was political and I won’t go into it. Deliberately vague on the industry and customer because of the 7 figure cost involved!
- serpi 8y agoSo before diagnosing the problem you made the customer spend huge amounts of money based on a guess. Not the least bit surprised.
- yann63 8y agoThe author removed the coloring, because the default setup is for ls to list files with colors. He uses an environment variable to do that. Wouldn't it be simpler to use \ls instead of ls? This way you get the "basic version of the executable", not some alias. I always use \command instead of command in my shell scripts, because you never know which aliases a user set up.
- Pawamoy 8y agoBut aliases are not supposed to be available in subprocesses/scripts, unless you use shopt -s expand_aliases. Even if they were, you cannot ask all your users to rewrite their scripts to use \ls instead of ls.