4 ms·

Ok, can someone explain why this method performs so well? Take the highest serial number, add a tenth of it then subtract 1. Presto. What is the underlying lo

Ok, can someone explain why this method performs so well? Take the highest serial number, add a tenth of it then subtract 1. Presto. What is the underlying logic here?

sliverstorm 11y ago
You have a key error. Take the highest serial number, add that number divided by the sample size, subtract one. You can then intuitively observe that as sample size increases, the "m/n" component shrinks- our highest serial number is probably going to be a lot closer to the maximum. As sample size decreases all the way to 1, the "m/n" component grows- our largest sampled serial number is probably not very close to the maximum.
- alanh 11y ago
  To me, this is the best explanation. Very visual and intuitive. Cheers.
boxy310 11y ago
The "maximum number" is a statistic both of a population and of a sample drawn from said population. As the sample size goes up, the sample statistic will converge to the population statistic, due to the Law of Large Numbers. Because serial numbers are natural numbers (1, 2, 3, etc.) they are uniformly distributed from 1 through N-hat (the population maximum), and the convergence of the sample statistic to the population statistic will scale inversely to the sample size. Let's take the example in the order it was given in the article: 117, 232, 122, 173, 167, 12, 168, 204, 4, 229 First step we would project the maximum to be 117 + 117 / 1 - 1 = 233. Or, if we drew 1 sample from the population, we would expect the "maximum" to be the average case -- the population mean. We take a minus 1 because there's a degree of freedom lost in using the maximum twice in the same formula (handwaving a bit, but that's a straightforward enough answer for now). With the second case, we have 232 + 232 / 2 - 1, or 347. We would intuitively expect two cases to effectively be distributed evenly across the population, or at 1/3 and 2/3. (The statistics is a bit different due to the MAX operator here, but I won't go into that). With the third case, we have 232 + 232 / 3 -1 = 309. Note that while this statistic is still higher than the actual population statistic, it is still converging due to the additional case that helps us counterbalance the weight of the previous unusually-high sample case we observed. Does that help?
- pinaceae 11y ago
  so let's say I am being attacked by the first wave of tanks the Germans have fielded. The wave consists of 10 tanks, but from the first batch of tanks produced. 1,5,6,9,12,15,18,20,21,25 how did the allies double check that their sample was not skewed as I described?
  - justinsaccount 11y ago
    You'd basically be estimating the size of the first batch at that point. Which is fine, until they capture a tank somewhere else that has a number of 230.
  - AcerbicZero 11y ago
    It should also be noted that there was not one "Tank" SN used, as each component had its own set of serial numbers to be analyzed. In this specific example they used gearbox SN's along with chassis and engine SN to provide the estimate. These statistical estimations were cross-checked with a broad range of other sources, in an attempt to gather data on everything from the number of tanks produced, to the number of factories in use and a variety of supply chain factors.
- mihaifm 11y ago
  does this work for consecutive integers that don't start from 1?
  - Vraxx 11y ago
    It seems that it would trivially work if you knew the number that you start from, you'd just be shifting the uniform distribution (and subtracting whatever number it started from to get the overall estimate). It does seem to be a harder problem if are given that they do not start at 1, but that they are consecutive integers. I imagine you might do a similar sort of statistical estimate on the beginning of the range as well as the end of the range. This introduces more uncertainty, but I see no reason why this wouldn't work
    - 11y ago
justinsaccount 11y ago
It's not "add a tenth": > For example, let’s say 10 tanks were captured It's "add 1/n" where n is the number you collected. Think of this related problem. Numbers are randomly chosen from 1 of 2 ranges. One range is from 0-10, the other is 0-100. The numbers are: 4 3 1 8 5 2 7 Which range were the numbers chosen from? It could be either, but statistically the answer would be from 0-10, since if they were chosen from 0-100 the chances of all of the numbers being lower than 10 is very low. Now, say you didn't know the ranges ahead of time, and the numbers were: 117, 232, 122, 173, 167, 12, 168, 204, 4, 229 Now, the upper bound can't be 100 or 200, since there is a 232. So it is at LEAST 232. Which is more likely, that the upper bound is 256, 500 or 1000 ? Intuitively you can assume that if it was 500+, you would expect at least one number in the 300-500 range. The highest number should be 232 + some amount that is inline with all the other numbers. It could be exactly 232, but the chances that you happened to pick the highest number itself is also low. Basically you are solving this problem: If you were going to select 10 numbers at random from a range of 0 to N, how high should N be so that there is a high probability that the highest number you select is only 232. Since a uniform distribution is assumed, you are basically just adding one more uniformly distributed data point.
- SeriousM 11y ago
  I had the same question and your answer was pretty clear to me. Thank you!
- JeremyHoward 11y ago
  It's actually "add m/n" where n is the number you collected and m is the highest sampled serial number.

Ok, can someone explain why this method performs so well? Take the highest serial number, add a tenth of it then subtract 1. Presto. What is the underlying logic here?

sliverstorm 11y ago

You have a key error. Take the highest serial number, add that number divided by the sample size, subtract one. You can then intuitively observe that as sample size increases, the "m/n" component shrinks- our highest serial number is probably going to be a lot closer to the maximum. As sample size decreases all the way to 1, the "m/n" component grows- our largest sampled serial number is probably not very close to the maximum.

alanh 11y ago
To me, this is the best explanation. Very visual and intuitive. Cheers.

boxy310 11y ago

The "maximum number" is a statistic both of a population and of a sample drawn from said population. As the sample size goes up, the sample statistic will converge to the population statistic, due to the Law of Large Numbers. Because serial numbers are natural numbers (1, 2, 3, etc.) they are uniformly distributed from 1 through N-hat (the population maximum), and the convergence of the sample statistic to the population statistic will scale inversely to the sample size. Let's take the example in the order it was given in the article: 117, 232, 122, 173, 167, 12, 168, 204, 4, 229 First step we would project the maximum to be 117 + 117 / 1 - 1 = 233. Or, if we drew 1 sample from the population, we would expect the "maximum" to be the average case -- the population mean. We take a minus 1 because there's a degree of freedom lost in using the maximum twice in the same formula (handwaving a bit, but that's a straightforward enough answer for now). With the second case, we have 232 + 232 / 2 - 1, or 347. We would intuitively expect two cases to effectively be distributed evenly across the population, or at 1/3 and 2/3. (The statistics is a bit different due to the MAX operator here, but I won't go into that). With the third case, we have 232 + 232 / 3 -1 = 309. Note that while this statistic is still higher than the actual population statistic, it is still converging due to the additional case that helps us counterbalance the weight of the previous unusually-high sample case we observed. Does that help?

pinaceae 11y ago
so let's say I am being attacked by the first wave of tanks the Germans have fielded. The wave consists of 10 tanks, but from the first batch of tanks produced. 1,5,6,9,12,15,18,20,21,25 how did the allies double check that their sample was not skewed as I described?
- justinsaccount 11y ago
  You'd basically be estimating the size of the first batch at that point. Which is fine, until they capture a tank somewhere else that has a number of 230.
- AcerbicZero 11y ago
  It should also be noted that there was not one "Tank" SN used, as each component had its own set of serial numbers to be analyzed. In this specific example they used gearbox SN's along with chassis and engine SN to provide the estimate. These statistical estimations were cross-checked with a broad range of other sources, in an attempt to gather data on everything from the number of tanks produced, to the number of factories in use and a variety of supply chain factors.
mihaifm 11y ago
does this work for consecutive integers that don't start from 1?
- Vraxx 11y ago
  It seems that it would trivially work if you knew the number that you start from, you'd just be shifting the uniform distribution (and subtracting whatever number it started from to get the overall estimate). It does seem to be a harder problem if are given that they do not start at 1, but that they are consecutive integers. I imagine you might do a similar sort of statistical estimate on the beginning of the range as well as the end of the range. This introduces more uncertainty, but I see no reason why this wouldn't work
  - 11y ago

justinsaccount 11y ago

It's not "add a tenth": > For example, let’s say 10 tanks were captured It's "add 1/n" where n is the number you collected. Think of this related problem. Numbers are randomly chosen from 1 of 2 ranges. One range is from 0-10, the other is 0-100. The numbers are: 4 3 1 8 5 2 7 Which range were the numbers chosen from? It could be either, but statistically the answer would be from 0-10, since if they were chosen from 0-100 the chances of all of the numbers being lower than 10 is very low. Now, say you didn't know the ranges ahead of time, and the numbers were: 117, 232, 122, 173, 167, 12, 168, 204, 4, 229 Now, the upper bound can't be 100 or 200, since there is a 232. So it is at LEAST 232. Which is more likely, that the upper bound is 256, 500 or 1000 ? Intuitively you can assume that if it was 500+, you would expect at least one number in the 300-500 range. The highest number should be 232 + some amount that is inline with all the other numbers. It could be exactly 232, but the chances that you happened to pick the highest number itself is also low. Basically you are solving this problem: If you were going to select 10 numbers at random from a range of 0 to N, how high should N be so that there is a high probability that the highest number you select is only 232. Since a uniform distribution is assumed, you are basically just adding one more uniformly distributed data point.

SeriousM 11y ago
I had the same question and your answer was pretty clear to me. Thank you!
JeremyHoward 11y ago
It's actually "add m/n" where n is the number you collected and m is the highest sampled serial number.