5 ms·

Can Apple Predict How Long a Transfer Takes?

13 points by kldavis4 12y ago

a2tech 12y ago
I believe the prediction is made based on the estimated size of the copy and it doesn't take into account the number of items to be copied. As any seasoned computer user will tell you, a large number of small files takes longer to copy then a similar sized large file. When the migration wizard is copying a huge number of small files the transfer goes far slower than when it gets into your documents and other large items.
- msandford 12y ago
  It doesn't seem like it would be TOO hard to get a list of all files to be transferred and sample say 1% of them randomly and from that judge the filesize / filecount statistics and from that make a better estimate. The only reason I could think NOT to do that is that it will take some time. But considering that most of the transfers take an hour or two anyhow, spending 3 minutes up-front to get a reasonable estimate of how much time you have to kill isn't bad. It's more helpful than useless, random guesses.
- bradleyland 12y ago
  All of that might be true, but it doesn't answer the question: Why isn't the Migration Assistant time prediction algorithm more accurate? Part of this might be the fact that the author is (or at least appears to be) using a traditional magnetic HDD, rather than an SSD. I say this because I've transferred many a MacBook Air over Thunderbolt in 40 minutes for smallish amounts of data. The longest time I've seen MA take was 4 hours for a MacBook Air with a 256 GB SSD (don't recall the utilization). A 10 hour transfer time smells of magnetic platters. The reason HDD vs SSD would impact prediction is that the throughput of an HDD is impacted more severely by a greater number of variables when transferring at the filesystem level than a SSD. Man, that was a mouthful. Stated simply, HDD performance is less predictable than SSD performance. Given the direction Apple is headed with primary storage, I could see them not investing much in tailoring their estimation algorithm toward a technology that is on the wane.
matthewmcg 12y ago
A bit frustrating, perhaps, but how often is migration assistant used by an individual user? I would think it would be more important to get the estimate right for routine batch file copies. Obligatory reference to XKCD #612: http://xkcd.com/612/ http://xkcd.com/612/
eps 12y ago
When copying files there are two costs - fixed per-file cost (of opening/closing a file) and variable cost of bulk copying. If you track both, you can pass the measurements through basic linear regression and get a simple formula - ETA = A * number-of-file + B * number-of-bytes This can be further refined to accomodate the cost of file extension for larger file, the overhead of directory creation, etc, but in practice basic two-term model works really well. There was an earlier post on HN on this, let me see if I can find it ... edit ... Nope, not HN. I was thinking of this - http://www.bvckup.com/support/forums/topic_show.pl?tid=101 http://www.bvckup.com/support/forums/topic_show.pl?tid=101
dudus 12y ago
http://xkcd.com/612/ http://xkcd.com/612/
shalmanese 12y ago
There are a couple of things Apple seems bizarrely bad at and a lot of them seem to involve networked file manipulation (Migration Assistant, Time Machine etc.) There seem to be network operations that have latencies orders of magnitude slower than they should be which brings down the performance of the entire application.

13 points by kldavis4 12y ago

I believe the prediction is made based on the estimated size of the copy and it doesn't take into account the number of items to be copied. As any seasoned computer user will tell you, a large number of small files takes longer to copy then a similar sized large file. When the migration wizard is copying a huge number of small files the transfer goes far slower than when it gets into your documents and other large items.

msandford 12y ago
It doesn't seem like it would be TOO hard to get a list of all files to be transferred and sample say 1% of them randomly and from that judge the filesize / filecount statistics and from that make a better estimate. The only reason I could think NOT to do that is that it will take some time. But considering that most of the transfers take an hour or two anyhow, spending 3 minutes up-front to get a reasonable estimate of how much time you have to kill isn't bad. It's more helpful than useless, random guesses.
bradleyland 12y ago
All of that might be true, but it doesn't answer the question: Why isn't the Migration Assistant time prediction algorithm more accurate? Part of this might be the fact that the author is (or at least appears to be) using a traditional magnetic HDD, rather than an SSD. I say this because I've transferred many a MacBook Air over Thunderbolt in 40 minutes for smallish amounts of data. The longest time I've seen MA take was 4 hours for a MacBook Air with a 256 GB SSD (don't recall the utilization). A 10 hour transfer time smells of magnetic platters. The reason HDD vs SSD would impact prediction is that the throughput of an HDD is impacted more severely by a greater number of variables when transferring at the filesystem level than a SSD. Man, that was a mouthful. Stated simply, HDD performance is less predictable than SSD performance. Given the direction Apple is headed with primary storage, I could see them not investing much in tailoring their estimation algorithm toward a technology that is on the wane.

matthewmcg 12y ago

A bit frustrating, perhaps, but how often is migration assistant used by an individual user? I would think it would be more important to get the estimate right for routine batch file copies. Obligatory reference to XKCD #612: http://xkcd.com/612/ http://xkcd.com/612/

eps 12y ago

When copying files there are two costs - fixed per-file cost (of opening/closing a file) and variable cost of bulk copying. If you track both, you can pass the measurements through basic linear regression and get a simple formula - ETA = A * number-of-file + B * number-of-bytes This can be further refined to accomodate the cost of file extension for larger file, the overhead of directory creation, etc, but in practice basic two-term model works really well. There was an earlier post on HN on this, let me see if I can find it ... edit ... Nope, not HN. I was thinking of this - http://www.bvckup.com/support/forums/topic_show.pl?tid=101 http://www.bvckup.com/support/forums/topic_show.pl?tid=101

dudus 12y ago

http://xkcd.com/612/ http://xkcd.com/612/

shalmanese 12y ago

There are a couple of things Apple seems bizarrely bad at and a lot of them seem to involve networked file manipulation (Migration Assistant, Time Machine etc.) There seem to be network operations that have latencies orders of magnitude slower than they should be which brings down the performance of the entire application.