5 ms·
This was written by my coworker, Kilian Cavalotti. We have two main file stores: $SCRATCH is a Lustre cluster, which is extremely performant but doesn't do wel
by CaliforniaKarl 8y ago
This was written by my coworker, Kilian Cavalotti. We have two main file stores: $SCRATCH is a Lustre cluster, which is extremely performant but doesn't do well with lots of inodes. And $HOME is a multi-node Isilon, which can handle all the inodes, but is not as performant. And we have users who sometimes like to put many files in single directories.
You can find more information about us at https://srcc.stanford.edu https://srcc.stanford.edu
And you can find more information about Sherlock at https://www.sherlock.stanford.edu https://www.sherlock.stanford.edu
- PunchTornado 8y agowhat is an Isilon? thanks
- Tushon 8y agohttps://en.wikipedia.org/wiki/Dell_EMC_Isilon https://en.wikipedia.org/wiki/Dell_EMC_Isilon
- loeg 8y agoAnd more on the product in particular: https://en.wikipedia.org/wiki/OneFS_distributed_file_system https://en.wikipedia.org/wiki/OneFS_distributed_file_system Disclaimer: I work for Dell.
- anardil 8y agoHyperscale Network Attached Storage! Disclaimer: I work on Isilon
- wahern 8y agoWhy was iSCSI support deprecated? I recently developed an interest an iSCSI, which makes me curious; not an Isilon user, though.
- CaliforniaKarl 8y agoMy guess would be that Isilon/OneFS is designed for file storage, instead of block storage, so having a block storage layer on top (like iSCSI) seems a little ungainly. Instead I expect Dell/EMC would prefer you investigate the VNX platform! Edit: By "file storage", I'm talking about storage mounted using protocols like SMB and NFS.
- JdeBP 8y agoThat's actually a disclosure, not a disclaimer.
- xenospn 8y agoClustered storage.
- arendtio 8y agoThanks for the additional information but sometimes I wonder that 15k+ files count as 'many' nowadays. I mean, an off-the-shelf Canon DSLR produces 20k files per directory (10k JPG + 10k raw). Nevertheless, I did some test your approach on my PC: On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance. Without modification, it takes 0.5 seconds to list a directory (on a slow consumer grade HDD) with 14448 files and after setting export LS_COLORS='ex=00:su=00:sg=00:ca=00:' it takes 4.4 seconds. However, listing a local (SSD) directory with 15k files takes just about 0.12 seconds and gets faster after settings the variable to 0.06 seconds. For all tests I drop the caches before testing, e.g.: echo 3 > /proc/sys/vm/drop_caches; time ls --color=always /mnt/nfs/many | wc -l
- mrb 8y ago"On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance" Run "strace -o logfile ls --color=always" and diff the two logfiles.
- arendtio 8y agoWell, this looks rather strange. As expected, settings the variable causes large changes as it removes a lot of lstat calls and also removes the color codes from the writes. So there seems to be no obvious explanation, why it becomes slower when doing less... # du -sh logfile_nfs* 1.7M logfile_nfs 668K logfile_nfs_exported However, the strace -c output looks kinda different: Without modifying the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEbtark/TMSDm9gT+jT0OK5ME8fE= https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb... With setting the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEbtark/TMSDm9gT+jT0OK5ME8fE= https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb...
- mrb 8y agoYou pasted the same link for both cases. Can't tell what's going on.
- 8y ago
- netsharc 8y agoTell your colleague I think emojis make the article look like it was written by a 12 year old. IMO. I guess I'm just an old fart.
- Faaak 8y agoI think they add a bit of emotion to the text written. It's not because it has colours that it's not a serious article.
- yani 8y agoIt is how most millenials write nowadays
- falsedan 8y agoIt’s how people trying to write like millennials write Postfix emoji, eurgh
- scrollaway 8y agoEven if it were, maybe you should have more respect for 12 year olds. IMO. I guess I'm not an ageist.
- gnufx 8y agoI don't understand the "lots of inodes" comment, but any filesystem limit on inodes isn't relevant to this, which is just due to RPCs to the metadata server(s). (ls -l is actually worse than coloured plain ls because the size data are on the OSS, not the MDS.) The canonical advice for general performance is to keep directories reasonably small on Lustre due to possible lock contention, but I don't know the circumstances for which that's actually relevant. [Metadata operations, such as large builds and tars, are typically slower on our Isilon than on the Lustre filesystem, which has no serious tuning as far as I know.]
- jabl 8y ago> The canonical advice for general performance is to keep directories reasonably small on Lustre due to possible lock contention, but I don't know the circumstances for which that's actually relevant. I helped a researched debug a Lustre performance issue a while ago. Each job was nothing special, read a few files (maybe a few GB total), do some (serial, no MPI or such) calculations taking maybe 10 min or so, produce output files, again a few GB. No problem, except when the person ran a several hundred of them in parallel as an array job the throughput per job dropped to a small fraction of normal. Turned out that all the jobs were using the same working directory. Slightly tweaking the workflow to have per-job directories fixed it.