4 ms·

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them (https://link.springer.com/article/10.3758/s13423-015-0947-8 https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run variation (https://link.springer.com/article/10.1007/s10654-016-0149-3 https://link.springer.com/article/10.1007/s10654-016-0149-3 misconception 22).

jnwatson 1mo ago
Mind blown. The more I read about statistics, the less I know.
- exogenousdata 1mo ago
  “There are three kinds of lies: Lies, damned lies and statistics.” - Mark Twain (attributed but unsubstantiated to Benjamin Disraeli)
fr2029 1mo ago
the 2nd derivate of shannon covariance of noise begs to differ
- deleted 1mo ago
  [deleted]
stared 1mo ago
Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative". It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between. Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas. That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/ https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).
- ricardobeat 1mo ago
  The main problem here is that a model that wildly fluctuates with 60% - 100% - 80% results will have the same wilson score as one that repeatedly scores 80% - 80% - 80%. So the 'confidence interval' bar is meaningless. I'm not that well versed in statistics, but a standard box plot is probably the best alternative
  - stared 1mo ago
    A single result is binary. All we get from a run is which tasks were solved, which weren’t.
- tomrod 1mo ago
  I'm working on a practical review implementation on this! Great to hear others are thinking along the same way.
- breezybottom 1mo ago
  Both ways involve sophistry. If you don't like dirty tricks, statistics isn't for you.
maCDzP 1mo ago
Thank you for these, coz I learned a lot! Great that they are open access.
diseasedyak 1mo ago
Yah, prediction interval instead, right? (I'm still learning statistics) Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.
- spider-mario 1mo ago
  Yes. It’s maybe easier to reason about by imagining that we are trying to estimate the parameter of a Gaussian distribution. Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that. 95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”. 95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-intervals/#:~:text=Similarly%2C%20the%20shortest,for%20scale%20parameters%29%2E https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ). Posterior predictive distribution = taking into account the uncertainty on both μ and σ, distribution of samples that would be obtained by sampling from N(μ, σ) (which, because of that uncertainty, is a https://en.wikipedia.org/wiki/Compound_probability_distribution https://en.wikipedia.org/wiki/Compound_probability_distribut... but may have a convenient closed form https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conjugate_distributions https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conju... ), from which we can likewise extract a 95% interval.
mitxela 1mo ago
> The key confusion underlying the FCF is the confusion of what is known before observing the data — that the CI, whatever it will be, has a fixed chance of containing the true value — with what is known after observing the data. Frequentist CI theory says nothing at all about the probability that a particular, observed confidence interval contains the true value; it is either 0 (if the interval does not contain the parameter) or 1 (if the interval does contain the true value). They're really reaching here. There's a 95% chance it's 1 and 5% it's 0, and we don't know which, so it's 0.95. Like saying the probability your uncle died of a shark attack is either 0 or 1. But in reality I don't know anything about your uncle so it's the average probability that someone picked randomly died of a shark attack, until I know it about your uncle.
- spider-mario 1mo ago
  It’s not reaching. See my response here: https://news.ycombinator.com/item?id=49622516 https://news.ycombinator.com/item?id=49622516 > Like saying the probability your uncle died of a shark attack is either 0 or 1. For a frequentist, it is. If you want the Bayesian interpretation of probability, you must calculate the probability that the interval you ended up getting contains the true value, and you can easily end up with a value different from 95%. You can’t just stop at “it was drawn from a pool of intervals of which 95% contain the true value” since you have more information than that (the numerical bounds).
  - mitxela 1mo ago
    If I pick an interval uniformly from a pool of intervals of which 95% contain the true value, it has a 95% chance of containing the true value. Just like if I pick a ball from a bag of balls of which 50% are red, it has a 50% chance of being red.
    - spider-mario 1mo ago
      If you pick a ball from a bag of balls of which half are fuzzy and half are smooth, then before picking the ball, there is indeed a 50% chance that you will pick a fuzzy one. But once you pick it and have it in your hand and can feel the texture, are you still going to say “there’s a 50% chance that this specific ball I am holding is fuzzy because half of the balls in there were”?
soltanov 1mo ago
Task success is not enough; reliability and cost matter.

jnwatson 1mo ago

Mind blown. The more I read about statistics, the less I know.

exogenousdata 1mo ago
“There are three kinds of lies: Lies, damned lies and statistics.” - Mark Twain (attributed but unsubstantiated to Benjamin Disraeli)

fr2029 1mo ago

the 2nd derivate of shannon covariance of noise begs to differ

deleted 1mo ago
[deleted]

stared 1mo ago

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative". It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between. Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas. That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/ https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).

ricardobeat 1mo ago
The main problem here is that a model that wildly fluctuates with 60% - 100% - 80% results will have the same wilson score as one that repeatedly scores 80% - 80% - 80%. So the 'confidence interval' bar is meaningless. I'm not that well versed in statistics, but a standard box plot is probably the best alternative
- stared 1mo ago
  A single result is binary. All we get from a run is which tasks were solved, which weren’t.
tomrod 1mo ago
I'm working on a practical review implementation on this! Great to hear others are thinking along the same way.
breezybottom 1mo ago
Both ways involve sophistry. If you don't like dirty tricks, statistics isn't for you.

maCDzP 1mo ago

Thank you for these, coz I learned a lot! Great that they are open access.

diseasedyak 1mo ago

Yah, prediction interval instead, right? (I'm still learning statistics) Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.

spider-mario 1mo ago
Yes. It’s maybe easier to reason about by imagining that we are trying to estimate the parameter of a Gaussian distribution. Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that. 95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”. 95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-intervals/#:~:text=Similarly%2C%20the%20shortest,for%20scale%20parameters%29%2E https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ). Posterior predictive distribution = taking into account the uncertainty on both μ and σ, distribution of samples that would be obtained by sampling from N(μ, σ) (which, because of that uncertainty, is a https://en.wikipedia.org/wiki/Compound_probability_distribution https://en.wikipedia.org/wiki/Compound_probability_distribut... but may have a convenient closed form https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conjugate_distributions https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conju... ), from which we can likewise extract a 95% interval.

mitxela 1mo ago

> The key confusion underlying the FCF is the confusion of what is known before observing the data — that the CI, whatever it will be, has a fixed chance of containing the true value — with what is known after observing the data. Frequentist CI theory says nothing at all about the probability that a particular, observed confidence interval contains the true value; it is either 0 (if the interval does not contain the parameter) or 1 (if the interval does contain the true value). They're really reaching here. There's a 95% chance it's 1 and 5% it's 0, and we don't know which, so it's 0.95. Like saying the probability your uncle died of a shark attack is either 0 or 1. But in reality I don't know anything about your uncle so it's the average probability that someone picked randomly died of a shark attack, until I know it about your uncle.

spider-mario 1mo ago
It’s not reaching. See my response here: https://news.ycombinator.com/item?id=49622516 https://news.ycombinator.com/item?id=49622516 > Like saying the probability your uncle died of a shark attack is either 0 or 1. For a frequentist, it is. If you want the Bayesian interpretation of probability, you must calculate the probability that the interval you ended up getting contains the true value, and you can easily end up with a value different from 95%. You can’t just stop at “it was drawn from a pool of intervals of which 95% contain the true value” since you have more information than that (the numerical bounds).
- mitxela 1mo ago
  If I pick an interval uniformly from a pool of intervals of which 95% contain the true value, it has a 95% chance of containing the true value. Just like if I pick a ball from a bag of balls of which 50% are red, it has a 50% chance of being red.
  - spider-mario 1mo ago
    If you pick a ball from a bag of balls of which half are fuzzy and half are smooth, then before picking the ball, there is indeed a 50% chance that you will pick a fuzzy one. But once you pick it and have it in your hand and can feel the texture, are you still going to say “there’s a 50% chance that this specific ball I am holding is fuzzy because half of the balls in there were”?

soltanov 1mo ago

Task success is not enough; reliability and cost matter.