3 ms·

I would not joy until exa is tried over a fs with more than a million files. Huge file numbers are a thing now and it is becoming to be an engineering challenge

by fsniper 12y ago

I would not joy until exa is tried over a fs with more than a million files. Huge file numbers are a thing now and it is becoming to be an engineering challenge.

cytzol 12y ago
exa can certainly handle a million files. Try running "exa -lRT ~" and watch it produce a tree view of your entire home directory! However, by 'handle', I actually mean 'run without crashing'... if you run that command, it'll be several minutes before it produces any output. exa aligns its output into a tabular format with every column having the same length, which means it actually has to stat and examine every file in the list before it's able to output the first line. This is annoying, but (as far as I know) unavoidable.
- fsniper 12y ago
  I have not yet looked into the code but if it's using glibc and corresponding getdents code path with suboptimal buffers, it most probably will be slower then ls. more information about the getdents buffer issue: https://www.olark.com/developers-corner/you-can-list-a-directory-with-8-million-files-but-not-with-ls https://www.olark.com/developers-corner/you-can-list-a-direc... Again, beware before downvoting me to oblivion I have not yet looked into the code.
  - fsniper 12y ago
    With my limited knowledge of rust I've looked into it. Seemingly on unix types of OSes exa uses std::old_fs::readdir, it uses sys::unix::readdir and it uses libc::readdir_r Which would also have the suboptimal getdents buffer issue.
  - cytzol 12y ago
    I don't even have downvote privileges yet! I was unaware of the getdents issue - thanks. However, it is most definitely slower than ls, because I've only tested it on large directories, rather than using it with large directories daily, so I haven't found places to optimise it quite yet. I'm not sure if using Rust will let me avoid that buffer issue, as right now I'm just using the function provided in its standard library.
    - fsniper 12y ago
      Nope, If you are using libc readdir, gettdents issue is unavoidable.
    - geofft 12y ago
      You can call the raw system call using, for example, kmc's syscall! macro: https://github.com/kmcallister/syscall.rs https://github.com/kmcallister/syscall.rs I do wonder if it's worth it eventually for libstd to speak the syscall interface directly, instead of going through another language's standard library (and whatever bugs or design decisions that standard library might have).
- agumonkey 12y ago
  I also wish for perf measurements, cpu, memory usage etc. How shorter is it from GNU ls (even though exa is still new)? Also I wish you didn't attempt to add complex layout to a ls-like program. I don't know why all low-level utilities have formatting logic in them, in GNU ls, IIRC, it was 30% of the code !

by fsniper 12y ago

I would not joy until exa is tried over a fs with more than a million files. Huge file numbers are a thing now and it is becoming to be an engineering challenge.

cytzol 12y ago

exa can certainly handle a million files. Try running "exa -lRT ~" and watch it produce a tree view of your entire home directory! However, by 'handle', I actually mean 'run without crashing'... if you run that command, it'll be several minutes before it produces any output. exa aligns its output into a tabular format with every column having the same length, which means it actually has to stat and examine every file in the list before it's able to output the first line. This is annoying, but (as far as I know) unavoidable.

fsniper 12y ago
I have not yet looked into the code but if it's using glibc and corresponding getdents code path with suboptimal buffers, it most probably will be slower then ls. more information about the getdents buffer issue: https://www.olark.com/developers-corner/you-can-list-a-directory-with-8-million-files-but-not-with-ls https://www.olark.com/developers-corner/you-can-list-a-direc... Again, beware before downvoting me to oblivion I have not yet looked into the code.
- fsniper 12y ago
  With my limited knowledge of rust I've looked into it. Seemingly on unix types of OSes exa uses std::old_fs::readdir, it uses sys::unix::readdir and it uses libc::readdir_r Which would also have the suboptimal getdents buffer issue.
- cytzol 12y ago
  I don't even have downvote privileges yet! I was unaware of the getdents issue - thanks. However, it is most definitely slower than ls, because I've only tested it on large directories, rather than using it with large directories daily, so I haven't found places to optimise it quite yet. I'm not sure if using Rust will let me avoid that buffer issue, as right now I'm just using the function provided in its standard library.
  - fsniper 12y ago
    Nope, If you are using libc readdir, gettdents issue is unavoidable.
  - geofft 12y ago
    You can call the raw system call using, for example, kmc's syscall! macro: https://github.com/kmcallister/syscall.rs https://github.com/kmcallister/syscall.rs I do wonder if it's worth it eventually for libstd to speak the syscall interface directly, instead of going through another language's standard library (and whatever bugs or design decisions that standard library might have).
agumonkey 12y ago
I also wish for perf measurements, cpu, memory usage etc. How shorter is it from GNU ls (even though exa is still new)? Also I wish you didn't attempt to add complex layout to a ls-like program. I don't know why all low-level utilities have formatting logic in them, in GNU ls, IIRC, it was 30% of the code !