cb321

61 ms·

631.
by cb321 5y ago
This is true and an ok point, but the writing of the discussed article even has a subtitle: "Git ls-files is 5 times faster than fd or find, but why?" My answer is "at least partly because `fd` & `find` are both slow - f
632.
by cb321 5y ago
So, when I do linux.git$ strace -fc fd -j1 --no-ignore --hidden --exclude .git --type file --type symlink>/dev/null strace: Process 24100 attached strace: Process 24101 attached % time seconds usecs/
633.
by cb321 5y ago
Sure..You could probably do X% of whatever setrlimit RLIMIT_NOFILE allows. The best API might be to take as a parameter a number (or fraction) of allowed open fd's and manage the resource sort of like C stdio FILE buffer sizes. OTOH,
634.
by cb321 5y ago
I cannot speak to why your "naive" C variant might have been slower than necessary. I might (wildly) guess that you did unnecessary string handling/allocation. You really just need one re-used buffer and a memcpy out of dir
635.
by cb321 5y ago
Running the tests like `-type l` or `-perm` or whatnot surely can be slow, but that is not in play in the article's use case. In my experience, fast vs. slow "reputations" are sadly very unreliable. Here is some system call c
636.
by cb321 5y ago
Agreed. Adding -j1 still leaves a very large (and wildly varying actually) number of futex calls shown by strace and slows down execution by another 1.5x. So, a little boost from threads but not much. Someone more interested (than I am) i
637.
by cb321 5y ago
plocate [1] is even faster (but of course with either you need to re-scan the whole FSes and rebuild indexes to have an up-to-date view). [1] https://www.linuxuprising.com/2021/09/plocate-is-much-faster...
638.
by cb321 5y ago
No question/disagreement that it's faster from having an index. It just need only be 1.8x faster -- not the much higher ratio reported in the article (on Linux, anyway). I was mostly just adding some performance color that the poi
639.
by cb321 5y ago
I got run times from the simplest single-threaded directory walk that are only 1.8x slower than git ls-files. (Min time of 10 runs with the git repo housed by /dev/shm on Linux 5.15.) The "simple" code is in https:&#x2
640.
by cb321 5y ago
While I agree the common pattern is to use void*/dynamic dispatch, this is not necessary. E.g., https://github.com/glouw/ctl/ or https://github.com/c-blake/bst show a couple ways to hav
641.
by cb321 5y ago
https://en.wikipedia.org/wiki/Tiny_C_Compiler probably compiles as fast or faster than the Pascal you remember. You can even use it as a backend for the Nim ( https://nim-lang.org/ ) compiler for subsec
642.
by cb321 5y ago
Also relevant is pixz [1] which can do parallel LZMA/XZ decompression as well as tar file indexing. [1] https://github.com/vasi/pixz
643.
by cb321 5y ago
Sadly, only a 2-button pointing device. :-)
644.
by cb321 5y ago
It isn't faster than the L1 memory system which is the most relevant thing to measure to compare hash algos, as already explained. (duplicated here for clarity/flow) Also, I perhaps could more fully motivate my time equation paren
645.
by cb321 5y ago
It isn't faster than the L1 memory system which is the most relevant thing to measure to compare hash algos, as already explained.
646.
by cb321 5y ago
>Finding sections of the file where data can be different is not only potentially hard but depends on the type of data. So can't be made generic I agree, and a cousin comment of mine suggests user-specifiable sampling which can alwa
647.
by cb321 5y ago
Sure. I agree various sampling strategies are a good design, as long as they don't add up to much IO. Making that a "user parameter" to be customizable seems ideal. What you suggest may make a fine default - if adapted to
648.
by cb321 5y ago
It really depends. The early bytes probably contain things like image dimensions which might discriminate a lot - or not at all. They might contain palette-style color information which might discriminate tons - or not at all. Most thing
649.
by cb321 5y ago
Feel free to measure it yourself instead of just speculating. (And as mentioned elsewhere, it is probably 1/128.) { EDIT: and I never measured meow - it could be faster than 1/16 cycle/byte but poorly assessed; just be sure
650.
by cb321 5y ago
This is roughly how https://github.com/c-blake/cligen/blob/master/examples/dups.... can work. There is little question that changing the scale of the problem very much lessons performance sensitivi
651.
by cb321 5y ago
I was only doing short < 1024 byte strings in L1 (like URI lengths-ish). I'm well aware no memory system on typical HW can deliver that (EDIT: "at large space scale"). The article itself mentions a similar calculation (b
652.
by cb321 5y ago
I have measured (on a Skylake core) the hash from Daniel Lemire, Owen Kaser, "Faster 64-bit universal hashing using carry-less multiplications, Journal of Cryptographic Engineering (to appear)" at t_cycles = (0.00741 +- 0.00098 cy
653.
by cb321 5y ago
I'm glad to hear you have some way to do your simple listings with standard tools, but I would say your comment is not really a fair rebut especially with RTFM snark. Your concept remains "quite limited", as I said. These l
654.
by cb321 5y ago
With `lc` you can have customized format strings so that you can see "size name", or "perms size name" or any subset/reordering of the report columns. And all of these are rolled up into a multi-column layout, thou
655.
by cb321 5y ago
Grouping/sorting also enhances human readability, and the options for this in GNU ls are quite limited.
656.
by cb321 5y ago
https://github.com/c-blake/lc can do that and more; Indeed you can group "dot directories" differently than "non-dot directories". When I sat down to write `lc` I went through all of exas issues an
657.
by cb321 5y ago
Author here; I have experience using it 100s of times per day since I wrote it. There are a couple of things that I don't use (like selinux) support that may be a little hinky. I have a few loyal users, but I don't hear much fee
658.
by cb321 5y ago
Actually, I did at first look into this kind of a modification of GNU ls, but its internal concepts of how to classify files is just not amenable to generalization along the lines of https://github.com/c-blake/lc . Ba
659.
by cb321 5y ago
I put some thought into that naming https://github.com/c-blake/lc which I think is basically uniformly more capable than exa.
660.
by cb321 5y ago
It depends how you count. There can be O(n log(n)^2) as you say, but I believe there are some comparison based algos where you only have to compare "fewer bits" to decide each comparison. So bit-wise cmps might still be O(n log

More ›