6 ms·

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar ope

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

genpfault 5y ago
> doing similar operations to a connected android phone MTP is terrible[1]. [1]: https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Performance https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Perfor...
- jakub_g 5y ago
  Oh boy. I was thinking USB 2.0 is main reason for Android<>Windows copying of photos to suck so much, but the rabbit hole is much deeper. It's sad that with cloud being the solution for everything those days, this will probably never be improved within next decade.
  - userbinator 5y ago
    It's ridiculously wasteful and slow to have to upload to a cloud server who-knows-where, then download it again to the computer several feet away. Yet with Android having removed USB mass storage, that's often the easiest way. It's against the interests of those who profit off selling cloud storage to make local transfers easy.
    - steerablesafe 5y ago
      If I need to move a lot of data I just use adb and USB debugging to access the files. That is actually fast, but it's ridiculous that I have to do it.
      - mihi 5y ago
        Ha, I like that! I should try that as well.
  - BlueTemplar 5y ago
    I'm using Resilio's Sync (before known as BitTorrent) for this. I guess that technically, it's a kind of cloud, except I'm running it on my intranet ?
- initplus 5y ago
  MTP is an absolute dogs breakfast. I've had multiple experiences of issues on the device side causing the entire explorer.exe process to crash! Explorers handling of the MTP protocol is not resilient to badly behaved devices, would not at all be surprised if there are security implications where a badly behaved MTP device can get an RCE in explorer.
vidarh 5y ago
Probably because someone worked on a high enough abstraction not to spot it, then tested it on small enough files to not see enough of a performance issue to dig into what actually happened..
- TeMPOraL 5y ago
  Perhaps the common approach to performance testing is wrong (forget for a moment that most shops don't think about it at all; let's just consider quality developers/companies). Instead of just monitoring whether the product is fast enough for minimal/typical/peak expected usage, maybe it would be good to focus on determining a boundary. E.g. how much data does it take for the program to run 60 seconds? Or, in general, 10x longer than the maximum you'd consider acceptable? How much data does it take for it to run out of memory? These determinations can be made with few tests each. Start with some reasonable amount of data, keep doubling it until the test fails. Continue binary-searching between last success and last failure. The results may come out surprising. Performance does not scale linearly - just because your program takes 1 second for 1 unit of data, and 2 seconds for 2 units of data, doesn't mean it'll take 20 seconds for 20 units of data. It may as well take an hour. Picking a threshold far above what's acceptable, and continuously monitoring how much load is required to reach it, will quickly identify when something is working much slower than it should.
duxup 5y ago
I find it's kinda hell moving files from both my android AND iPhone ... it feels like a 20 year old process in terms of it working in fits and starts and random failures.
lmilcin 5y ago
It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. As to slow operations... that is more likely because of synchronous implementation. The popular, naive implementation is, as above, to repeat same simple operation over and over again: read from source, write to destination, read from source, write to destination. A better implementation (what I would do) would be to pipeline operations. Basic pipeline would have three components, each streaming data to and/or from buffer 1: Read from source to pipeline buffer 2: Read from pipeline buffer to write to destination, write information about files to delete to another pipeline buffer 3: Read files to delete from pipeline buffer and execute deletions. Using *nix shell you could do something like that in single line 1: tar -c files to output 2: pipe output to tar -xv, write files to destination producing list of written files, pipe written files to output 3: read piped list of written files and remove them from input dir Now, this is not perfect because we are wasting performance on creating tar file when we immediately discard it, but you get the picture.
- tetha 5y ago
  > It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. Litte bit of a rant, but I see this SO MUCH in database layers in applications. Implement an operation for one row, slap a for loop around it, it works kinda quickly on the small test data set... and then prod has an intimate conversation with a brick wall. It has been 0 days at work since that happened. > As to slow operations... that is more likely because of synchronous implementation. Interestingly, I think the answer is a solid maybe and depends on the storage and how you issue your i-o operations. A flash storage will increase performance if you increase parallel operations, up to a point. However - and this code apparently was written 20 years ago - on spinning drives, parallel io-operations slow you down if the OS does not merge those operations. So it's entirely not obvious.
  - lmilcin 5y ago
slver 5y ago
In theory all software is coded using a "many by default" approach. So every time batching matters, we take those batching opportunities automatically due to the way software is coded. In practice we only batch when it starts hurting. It doesn't hurt to delete files one by one on a normal file system. It's made for that. So the API wasn't "many by default" and that's how it works for zip files as well.
toast0 5y ago
copy + delete one at a time makes a lot of sense if you're working on a filesystem without a way to move without copying (I don't think you can actually move a file in fat32), because copy all could require more space than is available. The same could be true here where you're moving from a zip file to probably the same filesystem the zip files is in; if removing a file from the zip file is actually an in-place move data then truncate. The problem, of course, is that removing a file from the zip file is tremendously expensive. Reading the file with one syscall per byte doesn't help (especially post-Spectre workarounds that make syscalls more expensive).
- codetrotter 5y ago
  This is the real answer IMO. For example, if you have a 2TB drive with 50GB available space left and are trying to move 1TB of data, and the move is requiring copying, but all individual files on their own are less than 50GB in size, then I’d be pretty upset if my computer was unable to move the files just because it was wanting to not delete anything until the end. But ideally I’d want the system to delete at the end if possible, and to otherwise delete as needed, instead of either doing only all at end or only after every single file.
- m4lvin 5y ago
  > copy + delete [...] I don't think you can actually move a file in fat32 Wait, does that mean that a file which is larger than half of the partition in fat32 cannot be moved? Or not even be renamed?
  - toast0 5y ago
    Fat32 supports rename in the same directory as a simple operation. Moving a file though, I don't think so (and nobody corrected me, so I might be right).
    - toast0 5y ago
      Ok, well I just tried this on a full fat32 partition, I'm able to move a large file no problems reported, so I was wrong.
worewood 5y ago
It's so if the move fails midway you won't need to start from scratch.
formerly_proven 5y ago
IIRC explorer/shell namespaces are built on top of IStorage / IStream and those don't know bulk operations.
Juntu 5y ago
Config ctrl alt del svchost then delete all dll files don’t get consfused thats lifetime experience

genpfault 5y ago

> doing similar operations to a connected android phone MTP is terrible[1]. [1]: https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Performance https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Perfor...

jakub_g 5y ago
Oh boy. I was thinking USB 2.0 is main reason for Android<>Windows copying of photos to suck so much, but the rabbit hole is much deeper. It's sad that with cloud being the solution for everything those days, this will probably never be improved within next decade.
- userbinator 5y ago
  It's ridiculously wasteful and slow to have to upload to a cloud server who-knows-where, then download it again to the computer several feet away. Yet with Android having removed USB mass storage, that's often the easiest way. It's against the interests of those who profit off selling cloud storage to make local transfers easy.
  - steerablesafe 5y ago
    If I need to move a lot of data I just use adb and USB debugging to access the files. That is actually fast, but it's ridiculous that I have to do it.
    - mihi 5y ago
      Ha, I like that! I should try that as well.
- BlueTemplar 5y ago
  I'm using Resilio's Sync (before known as BitTorrent) for this. I guess that technically, it's a kind of cloud, except I'm running it on my intranet ?
initplus 5y ago
MTP is an absolute dogs breakfast. I've had multiple experiences of issues on the device side causing the entire explorer.exe process to crash! Explorers handling of the MTP protocol is not resilient to badly behaved devices, would not at all be surprised if there are security implications where a badly behaved MTP device can get an RCE in explorer.

vidarh 5y ago

Probably because someone worked on a high enough abstraction not to spot it, then tested it on small enough files to not see enough of a performance issue to dig into what actually happened..

TeMPOraL 5y ago
Perhaps the common approach to performance testing is wrong (forget for a moment that most shops don't think about it at all; let's just consider quality developers/companies). Instead of just monitoring whether the product is fast enough for minimal/typical/peak expected usage, maybe it would be good to focus on determining a boundary. E.g. how much data does it take for the program to run 60 seconds? Or, in general, 10x longer than the maximum you'd consider acceptable? How much data does it take for it to run out of memory? These determinations can be made with few tests each. Start with some reasonable amount of data, keep doubling it until the test fails. Continue binary-searching between last success and last failure. The results may come out surprising. Performance does not scale linearly - just because your program takes 1 second for 1 unit of data, and 2 seconds for 2 units of data, doesn't mean it'll take 20 seconds for 20 units of data. It may as well take an hour. Picking a threshold far above what's acceptable, and continuously monitoring how much load is required to reach it, will quickly identify when something is working much slower than it should.

duxup 5y ago

I find it's kinda hell moving files from both my android AND iPhone ... it feels like a 20 year old process in terms of it working in fits and starts and random failures.

lmilcin 5y ago

It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. As to slow operations... that is more likely because of synchronous implementation. The popular, naive implementation is, as above, to repeat same simple operation over and over again: read from source, write to destination, read from source, write to destination. A better implementation (what I would do) would be to pipeline operations. Basic pipeline would have three components, each streaming data to and/or from buffer 1: Read from source to pipeline buffer 2: Read from pipeline buffer to write to destination, write information about files to delete to another pipeline buffer 3: Read files to delete from pipeline buffer and execute deletions. Using *nix shell you could do something like that in single line 1: tar -c files to output 2: pipe output to tar -xv, write files to destination producing list of written files, pipe written files to output 3: read piped list of written files and remove them from input dir Now, this is not perfect because we are wasting performance on creating tar file when we immediately discard it, but you get the picture.

tetha 5y ago
> It is easier to abstract (at least nively). First, you abstract moving a single file, then you create abstraction for "all files" basically by repeating same operation for all files. You could do this for a subset of files and so on. Litte bit of a rant, but I see this SO MUCH in database layers in applications. Implement an operation for one row, slap a for loop around it, it works kinda quickly on the small test data set... and then prod has an intimate conversation with a brick wall. It has been 0 days at work since that happened. > As to slow operations... that is more likely because of synchronous implementation. Interestingly, I think the answer is a solid maybe and depends on the storage and how you issue your i-o operations. A flash storage will increase performance if you increase parallel operations, up to a point. However - and this code apparently was written 20 years ago - on spinning drives, parallel io-operations slow you down if the OS does not merge those operations. So it's entirely not obvious.
- lmilcin 5y ago

slver 5y ago

In theory all software is coded using a "many by default" approach. So every time batching matters, we take those batching opportunities automatically due to the way software is coded. In practice we only batch when it starts hurting. It doesn't hurt to delete files one by one on a normal file system. It's made for that. So the API wasn't "many by default" and that's how it works for zip files as well.

toast0 5y ago

copy + delete one at a time makes a lot of sense if you're working on a filesystem without a way to move without copying (I don't think you can actually move a file in fat32), because copy all could require more space than is available. The same could be true here where you're moving from a zip file to probably the same filesystem the zip files is in; if removing a file from the zip file is actually an in-place move data then truncate. The problem, of course, is that removing a file from the zip file is tremendously expensive. Reading the file with one syscall per byte doesn't help (especially post-Spectre workarounds that make syscalls more expensive).

codetrotter 5y ago
This is the real answer IMO. For example, if you have a 2TB drive with 50GB available space left and are trying to move 1TB of data, and the move is requiring copying, but all individual files on their own are less than 50GB in size, then I’d be pretty upset if my computer was unable to move the files just because it was wanting to not delete anything until the end. But ideally I’d want the system to delete at the end if possible, and to otherwise delete as needed, instead of either doing only all at end or only after every single file.
m4lvin 5y ago
> copy + delete [...] I don't think you can actually move a file in fat32 Wait, does that mean that a file which is larger than half of the partition in fat32 cannot be moved? Or not even be renamed?
- toast0 5y ago
  Fat32 supports rename in the same directory as a simple operation. Moving a file though, I don't think so (and nobody corrected me, so I might be right).
  - toast0 5y ago
    Ok, well I just tried this on a full fat32 partition, I'm able to move a large file no problems reported, so I was wrong.

worewood 5y ago

It's so if the move fails midway you won't need to start from scratch.

formerly_proven 5y ago

IIRC explorer/shell namespaces are built on top of IStorage / IStream and those don't know bulk operations.

Juntu 5y ago

Config ctrl alt del svchost then delete all dll files don’t get consfused thats lifetime experience