3 ms·

While you’re right in absolute terms, I think you’re obscuring a valid point. The ONS having tests doesn’t change the fact that for the vast majority of the po

While you’re right in absolute terms, I think you’re obscuring a valid point. The ONS having tests doesn’t change the fact that for the vast majority of the population, being an essential worker was the only way to get a test. The ONS tests are not the only input to the announced R number are they? Maybe I’m misunderstanding horribly, but it seems like you’re saying there’s absolutely no correlation whatsoever between availability of tests to the general public and the accuracy of the announced R number for the general public. If I’m misunderstanding I do genuinely want to understand!

jlokier 6y ago
> it seems like you’re saying there’s absolutely no correlation whatsoever between availability of tests to the general public and the accuracy of the announced R number for the general public. Depends what you mean by accuracy, whether that's bias or uncertainty. If you're talking about bias, then I agree with the above statement. My estimate of the mean bias in published R estimates is zero. (Possibly on a logarithmic scale :-) But if you're talking about uncertainty and not bias, then in general more data is better provided its biases are known, but it's hard to say that focusing tests on a subset of the population reduces certainty. In a mathematical sense, to minimise uncertainty from sampling estimates if you have a fixed number of samples but a choice about which situations to assign them to, you want to focus more testing on the situations which provide the highest information content. That is not necessarily the same as spreading them evenly through the population in an unbiased manner. I think what you may be misunderstanding, and therefore misrepresenting, is the idea that a combination of statistically sampled tests (ONS) plus biased targeted tests (NHS, key workers, Test & Trace etc) results in "skewed" or more misleading R estimates than just the statistically sampled tests (ONS) by themselves. I think that's unlikely. I'm assuming the data is combined by competent professional statisticians. Assuming they are competent, the likelihood of any biases due to NHS sampling that you or I might think of not having already been evaluated by the statisticians is negligible. So, provided the bias can be estimated, combining data from multiple sources tends to reduce uncertainty rather than introducing bias in a particular direction. That's why I say my personal "estimate of mean bias" is zero. Published R may by higher or lower than true R, and we can take it for granted it will be off by some amount and constantly (and retroactively) revised with new data, which is fair enough for estimations, but I have no basis on which to assume corrected results are more likely to have bias in one direction or the other. A nice feature of having two or more kinds of sampling is that intentionally-randomly-sourced data (ONS) acts as an "anchor" on the interpretation of non-ONS data, allowing raw biases from various sources to be estimated and adjusted for, uncertainties to be estimated too, while at the same time trends (such as over time) remain trackable with the higher statistical power that comes from larger numbers of samples. In other words, a bit of the best of both worlds. And since r and R are trend parameters, that's quite helpful.
- DerDangDerDang 6y ago
  Thanks, that's enlightening. I am definitely not a statistician, as you've no doubt guessed! >I think what you may be misunderstanding, and therefore misrepresenting, is the idea that a combination of statistically sampled tests (ONS) plus biased targeted tests (NHS, key workers, Test & Trace etc) results in "skewed" or more misleading R estimates than just the statistically sampled tests (ONS) by themselves. Not quite, and I apologise if I've communicated poorly. Totally agree that multiple sampling methods lead to more convincing results. I avoid optimising until I've profiled with both a sampling AND an instrumenting profiler, for example. What I was trying (failing!) to get across is that I think that if the distribution of 'generally available' tests is sufficiently non-representative, then the statistically sampled test model potentially couldn't correct for it enough for the derived estimate to be reliable. Obviously the responsible subgroup at SAGE will have tried to weight accordingly, and the people running the individual studies will as you say, be much more skilled at accounting for potential bias than I am. I'm absolutely not saying that I've stumbled on some insight that they missed! Here's where I struggle - the pool of generally available tests at the time fall into two relatively narrow groups: patients exhibiting symptoms bad enough to require hospitalisation, and people at high risk of transmission. The random longitudinal studies don't have that problem of course and should be used to correct for the bias. I realise there are methods for quantifying bias and uncertainty, but isn't this more at risk of inaccuracy than simply having a higher N? N matters, surely? The higher confidence we have that any one set of data is representative, the higher confidence we can have in the final derived estimate. I just can't see any angle from which having tested more people we wouldn't have improved our ability to estimate R.

jlokier 6y ago

> it seems like you’re saying there’s absolutely no correlation whatsoever between availability of tests to the general public and the accuracy of the announced R number for the general public. Depends what you mean by accuracy, whether that's bias or uncertainty. If you're talking about bias, then I agree with the above statement. My estimate of the mean bias in published R estimates is zero. (Possibly on a logarithmic scale :-) But if you're talking about uncertainty and not bias, then in general more data is better provided its biases are known, but it's hard to say that focusing tests on a subset of the population reduces certainty. In a mathematical sense, to minimise uncertainty from sampling estimates if you have a fixed number of samples but a choice about which situations to assign them to, you want to focus more testing on the situations which provide the highest information content. That is not necessarily the same as spreading them evenly through the population in an unbiased manner. I think what you may be misunderstanding, and therefore misrepresenting, is the idea that a combination of statistically sampled tests (ONS) plus biased targeted tests (NHS, key workers, Test & Trace etc) results in "skewed" or more misleading R estimates than just the statistically sampled tests (ONS) by themselves. I think that's unlikely. I'm assuming the data is combined by competent professional statisticians. Assuming they are competent, the likelihood of any biases due to NHS sampling that you or I might think of not having already been evaluated by the statisticians is negligible. So, provided the bias can be estimated, combining data from multiple sources tends to reduce uncertainty rather than introducing bias in a particular direction. That's why I say my personal "estimate of mean bias" is zero. Published R may by higher or lower than true R, and we can take it for granted it will be off by some amount and constantly (and retroactively) revised with new data, which is fair enough for estimations, but I have no basis on which to assume corrected results are more likely to have bias in one direction or the other. A nice feature of having two or more kinds of sampling is that intentionally-randomly-sourced data (ONS) acts as an "anchor" on the interpretation of non-ONS data, allowing raw biases from various sources to be estimated and adjusted for, uncertainties to be estimated too, while at the same time trends (such as over time) remain trackable with the higher statistical power that comes from larger numbers of samples. In other words, a bit of the best of both worlds. And since r and R are trend parameters, that's quite helpful.

DerDangDerDang 6y ago
Thanks, that's enlightening. I am definitely not a statistician, as you've no doubt guessed! >I think what you may be misunderstanding, and therefore misrepresenting, is the idea that a combination of statistically sampled tests (ONS) plus biased targeted tests (NHS, key workers, Test & Trace etc) results in "skewed" or more misleading R estimates than just the statistically sampled tests (ONS) by themselves. Not quite, and I apologise if I've communicated poorly. Totally agree that multiple sampling methods lead to more convincing results. I avoid optimising until I've profiled with both a sampling AND an instrumenting profiler, for example. What I was trying (failing!) to get across is that I think that if the distribution of 'generally available' tests is sufficiently non-representative, then the statistically sampled test model potentially couldn't correct for it enough for the derived estimate to be reliable. Obviously the responsible subgroup at SAGE will have tried to weight accordingly, and the people running the individual studies will as you say, be much more skilled at accounting for potential bias than I am. I'm absolutely not saying that I've stumbled on some insight that they missed! Here's where I struggle - the pool of generally available tests at the time fall into two relatively narrow groups: patients exhibiting symptoms bad enough to require hospitalisation, and people at high risk of transmission. The random longitudinal studies don't have that problem of course and should be used to correct for the bias. I realise there are methods for quantifying bias and uncertainty, but isn't this more at risk of inaccuracy than simply having a higher N? N matters, surely? The higher confidence we have that any one set of data is representative, the higher confidence we can have in the final derived estimate. I just can't see any angle from which having tested more people we wouldn't have improved our ability to estimate R.