4 ms·

Everybody seems to be failing at reading comprehension today... His two tests look like this: 1. Network -> USB > I'm copying files from a my networked hard

by nknight 15y ago

Everybody seems to be failing at reading comprehension today... His two tests look like this: 1. Network -> USB > I'm copying files from a my networked hard drive to a USB stick. > It tells me it's going to take about 7 hours. For 3 Gig of data. 2. Network -> Local HDD -> USB > I copy it from the networked drive to my C drive. This takes about 4 minutes. > I then start copying it to the USB stick. This is varying between 8 minutes and 45 minutes, depending on whether it's copying a big file (fast) or lots of little files (slow). Test #1, network directly to USB, is showing a 7-hour time estimate. Test #2, network to HDD to USB, is showing an 8-45 minute time estimate (truth will be somewhere in between). This is not a generic complaint that "Windows I/O is slow", it's a complaint that the obvious method (straight from network to desired USB device) is many times slower than the less-obvious method of hopping through the local HDD for no good reason.

MatthewPhillips 15y ago
I agree with his conclusions and have experienced it myself while working with clients. Trying to copy a code project (with a lot of assets) of about 300mb says it will take 3 hours to go from my laptop over wifi on their LAN to an external USB drive hooked up to their server. When I have to copy something large I always request to either unplug their external hard drive and plug it into my laptop, or copy it from my laptop to one of their desktops an then copy it to the external drive. I do think there's something fishy about going over a network to a USB on Windows.
AndrewDucker 15y ago
Thank you :->
acqq 15y ago
I've actually tested this, so I have the exact answers! The author just saw the estimate of 1; he hasn't measured it, it seems. The estimation by Windows Explorer was wrong in my test but the actual difference wasn't so significant. In my tests, the time of "a lot of small files" is dominated by the USB stick slowness (800 files totaling 5 M took 80 sec from the HDD and 90 sec from the network share), and the time of "copying a big file" gets two times slower when the sizes of write blocks to USB stick are not "convenient" and that happens in the "network to USB" case (130 MB for 7 sec from HDD, 14 sec from the network share). Note that Windows by default doesn't cache writes to the USB stick; specifically, the scenario "plug out the stick without unmounting" is supported, whereas the last time I've checked I had to unmount USB sticks on Linux to be sure that everything is properly written. So once you don't cache writes you're at mercy of the USB stick, which is reasonably fast only for one specific block size (specific multiple, actually). When you don't get the best block size over the network, you'll get slower writes to the USB stick. What I see in my tests (on Windows XP) is that the copying is done by CopyFileEx API, which uses blocks of 65536 bytes (64KB) from the local HDD but uses blocks of 61440 bytes (that's exactly one 4K block less than 64KB) over the network, which is then slower on the USB stick. Conclusions: a) Don't copy small files to the USB stick on Windows if you need speed. It's hundreds of times faster (when the bytes are counted) to copy an archive of the files vs. a file by file. b) If you want efficient copying of bigger files from the network share to the USB stick, avoid CopyFileEx API function, at least on Windows XP, use the software which writes in 32K multiples.
- aidenn0 15y ago
  It would be neat to hack up a simple copy program that uses more USB friendly buffer sizes and see what happens. I don't have an XP machine handy so I can't easily test this.

by nknight 15y ago

MatthewPhillips 15y ago

I agree with his conclusions and have experienced it myself while working with clients. Trying to copy a code project (with a lot of assets) of about 300mb says it will take 3 hours to go from my laptop over wifi on their LAN to an external USB drive hooked up to their server. When I have to copy something large I always request to either unplug their external hard drive and plug it into my laptop, or copy it from my laptop to one of their desktops an then copy it to the external drive. I do think there's something fishy about going over a network to a USB on Windows.

AndrewDucker 15y ago

Thank you :->

acqq 15y ago

I've actually tested this, so I have the exact answers! The author just saw the estimate of 1; he hasn't measured it, it seems. The estimation by Windows Explorer was wrong in my test but the actual difference wasn't so significant. In my tests, the time of "a lot of small files" is dominated by the USB stick slowness (800 files totaling 5 M took 80 sec from the HDD and 90 sec from the network share), and the time of "copying a big file" gets two times slower when the sizes of write blocks to USB stick are not "convenient" and that happens in the "network to USB" case (130 MB for 7 sec from HDD, 14 sec from the network share). Note that Windows by default doesn't cache writes to the USB stick; specifically, the scenario "plug out the stick without unmounting" is supported, whereas the last time I've checked I had to unmount USB sticks on Linux to be sure that everything is properly written. So once you don't cache writes you're at mercy of the USB stick, which is reasonably fast only for one specific block size (specific multiple, actually). When you don't get the best block size over the network, you'll get slower writes to the USB stick. What I see in my tests (on Windows XP) is that the copying is done by CopyFileEx API, which uses blocks of 65536 bytes (64KB) from the local HDD but uses blocks of 61440 bytes (that's exactly one 4K block less than 64KB) over the network, which is then slower on the USB stick. Conclusions: a) Don't copy small files to the USB stick on Windows if you need speed. It's hundreds of times faster (when the bytes are counted) to copy an archive of the files vs. a file by file. b) If you want efficient copying of bigger files from the network share to the USB stick, avoid CopyFileEx API function, at least on Windows XP, use the software which writes in 32K multiples.

aidenn0 15y ago
It would be neat to hack up a simple copy program that uses more USB friendly buffer sizes and see what happens. I don't have an XP machine handy so I can't easily test this.