4 ms·

> but it is much too small to hold the kind of data you’re suggesting My toy has a 3 MB L1 cache, 8 MB L2. You'll notice from the same figures that the residen

by apjana 8y ago

> but it is much too small to hold the kind of data you’re suggesting My toy has a 3 MB L1 cache, 8 MB L2. You'll notice from the same figures that the resident memory usage of `nnn` was 3616 KB. And in the default start (not du) right now the figure is: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23034 vaio 20 0 14692 3256 2472 S 0.0 0.0 0:00.00 nnn -c 1 -i So it required around 350 KB for `du`. Oh yes, it can be cached at will. > check if a particular file is cached and if not I am a userland utility. I don't add deps on fs drivers (and maintain redundant code... I discourage code bloat as much as feature bloat) when standard APIs are available to read, write, stat. > However regardless of where that cache is, it’s volatile, it’s freeable. Yes!!! The worst case! And the worst case for you is where all dirs have changed. There you issue rescan, `nnn` issues rescan. `nnn` is accurate because it doesn't omit independent files for _personal_ optimization preferences. You do. You show _stale_, _wrong_ info in the worst case.

laumars 8y ago
3MB is not enough to store the kind of data you’re talking about. In your example you were coping files from an SSD to a removable device; what happens if that file is 4MB big? Or 20MB? Some program executables are multiple megabytes in size these days, your CPU cache is much more valuable storing that than it is storing a random 20MB XLSX file you happened to copy half an hour earlier. ;) > You do. You show wrong info in the worst case. Yes, I’d repeatedly said that myself. However I was making a case study rather than giving a lecture on how your tool should be written. Remember when I said “there is no right or wrong answer”? (You have ignored my point that you cannot guarantee the kernels / fs drivers cache isn’t also stale though. That does happen in some rare edge cases). My point is you shouldn’t be so certain that you’re method is the best because there are plenty of scenarios when accuracy and low memory usage is more costly (eg the fastest file copying tools are those that buffer data - costing more RAM during usage). Plus let’s not forget that syscalls are expensive too. But that’s the trade off you take and that is what is advantageous for the perpose you’re targeting nnn for. So there really isn’t a right or wrong answer here at all and memory usage is in fact only a very small part of the equation (which is what the other projects author what trying to say).
- apjana 8y ago
  > Some program executables are multiple megabytes in size these days I have shared the figures from my system right now in my earlier comment. The memory required for the meta info is extremely cache-able. And yes, let's call it a day now. In fact, I am happy we had a very informative discussion towards the end. This is what I expected from the other dev and you from the very beginning. I am always ready for a productive non-abusive technical discussion. And if I find someday `ncdu` takes comparable/lesser memory than `nnn` I will update that data verbatim as well. But I wouldn't take uncalled-for shitstorm in public forums from strangers lying low.
  - laumars 8y ago
    But that’s what you’re storing in the system memory, not CPU cache. Ok let’s say the kernel does cache recently accessed memeory in the L2 cache, you’re still competing with everything else - particularly in a multi-user or multi-threaded system. So that is not going to stay in L2 for long. You’re making so many assumptions about the host platform and they simply don’t stack up for the majority of cases - it’s just that most systems are fast enough to hide your incorrect assumptions. Also your understanding of how file system caching works is completely off. Granted that’s not something most normal engineers need to worry about, but given the claims you’re making about nnn, I would suggest you spend a little time researching the fundamentals here. Edit: > I am happy we had a very informative discussion towards the end. This is what I expected from the other dev That was exactly what the other dev was doing. You just weren’t ready to listen to him - which is why I started making comments about your “narky” replies. ;)
    - apjana 8y ago
      Well, I shared the numbers, I gave you ways to test it. We are done here.
      - laumars 8y ago
        but You’re not proving that the data you are expecting to be in L2 actually ends up in L2. Let alone persists long enough to be recalled from L2 when you do your rescans. Your test is crude and thus doesn’t prove anything. Which is what we have all been telling you right from the start!!!! You cannot just throw numbers up and say “look I’m right” if the test of flawed from the outset. But yes, it’s probably better we do call it a day.

by apjana 8y ago

laumars 8y ago

3MB is not enough to store the kind of data you’re talking about. In your example you were coping files from an SSD to a removable device; what happens if that file is 4MB big? Or 20MB? Some program executables are multiple megabytes in size these days, your CPU cache is much more valuable storing that than it is storing a random 20MB XLSX file you happened to copy half an hour earlier. ;) > You do. You show wrong info in the worst case. Yes, I’d repeatedly said that myself. However I was making a case study rather than giving a lecture on how your tool should be written. Remember when I said “there is no right or wrong answer”? (You have ignored my point that you cannot guarantee the kernels / fs drivers cache isn’t also stale though. That does happen in some rare edge cases). My point is you shouldn’t be so certain that you’re method is the best because there are plenty of scenarios when accuracy and low memory usage is more costly (eg the fastest file copying tools are those that buffer data - costing more RAM during usage). Plus let’s not forget that syscalls are expensive too. But that’s the trade off you take and that is what is advantageous for the perpose you’re targeting nnn for. So there really isn’t a right or wrong answer here at all and memory usage is in fact only a very small part of the equation (which is what the other projects author what trying to say).

apjana 8y ago
> Some program executables are multiple megabytes in size these days I have shared the figures from my system right now in my earlier comment. The memory required for the meta info is extremely cache-able. And yes, let's call it a day now. In fact, I am happy we had a very informative discussion towards the end. This is what I expected from the other dev and you from the very beginning. I am always ready for a productive non-abusive technical discussion. And if I find someday `ncdu` takes comparable/lesser memory than `nnn` I will update that data verbatim as well. But I wouldn't take uncalled-for shitstorm in public forums from strangers lying low.
- laumars 8y ago
  But that’s what you’re storing in the system memory, not CPU cache. Ok let’s say the kernel does cache recently accessed memeory in the L2 cache, you’re still competing with everything else - particularly in a multi-user or multi-threaded system. So that is not going to stay in L2 for long. You’re making so many assumptions about the host platform and they simply don’t stack up for the majority of cases - it’s just that most systems are fast enough to hide your incorrect assumptions. Also your understanding of how file system caching works is completely off. Granted that’s not something most normal engineers need to worry about, but given the claims you’re making about nnn, I would suggest you spend a little time researching the fundamentals here. Edit: > I am happy we had a very informative discussion towards the end. This is what I expected from the other dev That was exactly what the other dev was doing. You just weren’t ready to listen to him - which is why I started making comments about your “narky” replies. ;)
  - apjana 8y ago
    Well, I shared the numbers, I gave you ways to test it. We are done here.
    - laumars 8y ago
      but You’re not proving that the data you are expecting to be in L2 actually ends up in L2. Let alone persists long enough to be recalled from L2 when you do your rescans. Your test is crude and thus doesn’t prove anything. Which is what we have all been telling you right from the start!!!! You cannot just throw numbers up and say “look I’m right” if the test of flawed from the outset. But yes, it’s probably better we do call it a day.