3 ms·
> It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same
by tetha 5y ago
> It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on.
Litte bit of a rant, but I see this SO MUCH in database layers in applications. Implement an operation for one row, slap a for loop around it, it works kinda quickly on the small test data set... and then prod has an intimate conversation with a brick wall. It has been 0 days at work since that happened.
> As to slow operations... that is more likely because of synchronous implementation.
Interestingly, I think the answer is a solid maybe and depends on the storage and how you issue your i-o operations. A flash storage will increase performance if you increase parallel operations, up to a point. However - and this code apparently was written 20 years ago - on spinning drives, parallel io-operations slow you down if the OS does not merge those operations. So it's entirely not obvious.
- lmilcin 5y agoOn single drive you run a variation where you read X MB of data to a buffer, then write X MB of data out, then execute deletes, and so on. This lets avoid some of the problems with small files. Not all, because small files will unlikely to be consecutive, the head will have to jump a lot, and then you still need to do a lot of small writes to filesystem (to remove the files). There are obviously improvements you could do. For some filesystem you can just remove entire folders rather than remove the files individually just to remove parent folder.