4 ms·

It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same op

It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. As to slow operations... that is more likely because of synchronous implementation. The popular, naive implementation is, as above, to repeat same simple operation over and over again: read from source, write to destination, read from source, write to destination. A better implementation (what I would do) would be to pipeline operations. Basic pipeline would have three components, each streaming data to and/or from buffer 1: Read from source to pipeline buffer 2: Read from pipeline buffer to write to destination, write information about files to delete to another pipeline buffer 3: Read files to delete from pipeline buffer and execute deletions. Using *nix shell you could do something like that in single line 1: tar -c files to output 2: pipe output to tar -xv, write files to destination producing list of written files, pipe written files to output 3: read piped list of written files and remove them from input dir Now, this is not perfect because we are wasting performance on creating tar file when we immediately discard it, but you get the picture.

tetha 5y ago
> It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. Litte bit of a rant, but I see this SO MUCH in database layers in applications. Implement an operation for one row, slap a for loop around it, it works kinda quickly on the small test data set... and then prod has an intimate conversation with a brick wall. It has been 0 days at work since that happened. > As to slow operations... that is more likely because of synchronous implementation. Interestingly, I think the answer is a solid maybe and depends on the storage and how you issue your i-o operations. A flash storage will increase performance if you increase parallel operations, up to a point. However - and this code apparently was written 20 years ago - on spinning drives, parallel io-operations slow you down if the OS does not merge those operations. So it's entirely not obvious.
- lmilcin 5y ago
  On single drive you run a variation where you read X MB of data to a buffer, then write X MB of data out, then execute deletes, and so on. This lets avoid some of the problems with small files. Not all, because small files will unlikely to be consecutive, the head will have to jump a lot, and then you still need to do a lot of small writes to filesystem (to remove the files). There are obviously improvements you could do. For some filesystem you can just remove entire folders rather than remove the files individually just to remove parent folder.
TeMPOraL 5y ago
> The popular, naive implementation is, as above, to repeat same simple operation over and over again: read from source, write to destination, read from source, write to destination. Reminds me of the kind of patterns functional programming languages introduce, where you process data by describing operations on individual items and assembling them into "a stream". I'm always wary of those - without a good implementation and some heavy magic at the language level, they tend to become the kind of context-switching performance disaster you describe.
- lmilcin 5y ago
  Yes. Functional world is not impervious to leaky abstractions. I am personally of the opinion that, to be a good developer, you have to have mental model of what happens beneath. If you are programming in a high level language it is easy to try forget about the fact that your program runs on real hardware. I know, because I work mostly on Java projects and trying to talk to Java developers about real hardware is useless. I have an interview question where I ask "what prevents one process from dereferencing a pointer written out by another process on the same machine" and I get all sorts of funny answers and only 5-10% candidates even have beginning of understanding what is going on. Most don't know what virtual memory is or are surprised that two processes can resolve different values under same pointer.
  - MaxBarraclough 5y ago
    > Most don't know what virtual memory is or are surprised that two processes can resolve different values under same pointer. Yikes. How many of them have a degree in computer science?
    - yurishimo 5y ago
      Probably not many of them, because it doesn't usually take a graduate to write Java. Especially if your experience with Java is also limited to tools like Spring. I think the question is kind of pointless unless your specifically hiring people to work on an application that needs that sort of thru-put. Most apps don't need it.
deleted 5y ago
[deleted]

tetha 5y ago

> It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. Litte bit of a rant, but I see this SO MUCH in database layers in applications. Implement an operation for one row, slap a for loop around it, it works kinda quickly on the small test data set... and then prod has an intimate conversation with a brick wall. It has been 0 days at work since that happened. > As to slow operations... that is more likely because of synchronous implementation. Interestingly, I think the answer is a solid maybe and depends on the storage and how you issue your i-o operations. A flash storage will increase performance if you increase parallel operations, up to a point. However - and this code apparently was written 20 years ago - on spinning drives, parallel io-operations slow you down if the OS does not merge those operations. So it's entirely not obvious.

lmilcin 5y ago
On single drive you run a variation where you read X MB of data to a buffer, then write X MB of data out, then execute deletes, and so on. This lets avoid some of the problems with small files. Not all, because small files will unlikely to be consecutive, the head will have to jump a lot, and then you still need to do a lot of small writes to filesystem (to remove the files). There are obviously improvements you could do. For some filesystem you can just remove entire folders rather than remove the files individually just to remove parent folder.

TeMPOraL 5y ago

> The popular, naive implementation is, as above, to repeat same simple operation over and over again: read from source, write to destination, read from source, write to destination. Reminds me of the kind of patterns functional programming languages introduce, where you process data by describing operations on individual items and assembling them into "a stream". I'm always wary of those - without a good implementation and some heavy magic at the language level, they tend to become the kind of context-switching performance disaster you describe.

lmilcin 5y ago
Yes. Functional world is not impervious to leaky abstractions. I am personally of the opinion that, to be a good developer, you have to have mental model of what happens beneath. If you are programming in a high level language it is easy to try forget about the fact that your program runs on real hardware. I know, because I work mostly on Java projects and trying to talk to Java developers about real hardware is useless. I have an interview question where I ask "what prevents one process from dereferencing a pointer written out by another process on the same machine" and I get all sorts of funny answers and only 5-10% candidates even have beginning of understanding what is going on. Most don't know what virtual memory is or are surprised that two processes can resolve different values under same pointer.
- MaxBarraclough 5y ago
  > Most don't know what virtual memory is or are surprised that two processes can resolve different values under same pointer. Yikes. How many of them have a degree in computer science?
  - yurishimo 5y ago
    Probably not many of them, because it doesn't usually take a graduate to write Java. Especially if your experience with Java is also limited to tools like Spring. I think the question is kind of pointless unless your specifically hiring people to work on an application that needs that sort of thru-put. Most apps don't need it.

deleted 5y ago

[deleted]