4 ms·

Complaining about "bad charting" and posting a chart with y-axis that doesn't start at 0 is kinda weird.

by vb-8448 1mo ago

andai 1mo ago
Yeah, this doesn't include older models, some of which were already saturating many common tasks a year ago. I'll often add them to the AA graph for reference.
paimapi 1mo ago
to be fair, you don't need to start the y-axis at zero [0] but for some of the graphs where the lowest value is close to 0 the best practice is to do so there's a fun Excel artifact where it auto-selects the 'relevant' range with no adjustment for how proportionally close to 0 the values are - a professional researcher publishing to a journal should know better (and should be ridiculed for not incorporating best practices) but for a personal blog by an SWE this really isn't the worst sin [0] https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have-to-start-at-zero-part-1-line-charts/ https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have...
- vb-8448 1mo ago
  IMO in this case is mandatory to start from 0 because it alters the visual perception. Just look at the first chart: the distance between Fable 5.1 and Sol is <5%, but it looks like 25 or 30%.
  - _aavaa_ 1mo ago
    I disagree. The y-axis is some arbitrary intelligence score that we use as a proxy for performance on whatever our specific task happens to be. So it doesn't matter if a model is a 0, 1, or 20 along this axis, they are all useless for the tasks I want. And as the complexity of your score increases, the cutoff goes up. We can quibble about where your personal cutoff is, but it aint 0.
    - vb-8448 1mo ago
      Y-axis is between 0 and 100. But even if it was between 0 and Inf+, it still gives you a wrong perspective, especially if you are not paying attention, on model capabilities.
datadrivenangel 1mo ago
Yeah this is really funny and annoying. They get better later on in the article, but it is questionable.
BeetleB 1mo ago
Not at all, as long as it's labeled as such. Coming from engineering/science, this is common. What is bad is starting at 0, showing an indicator of a gap, and suddenly starting at 30 or whatever after the gap.
- vb-8448 1mo ago
  It gives you a wrong perspective, especially if you are distracted, on model capabilities: Fable 5.1 is not 30% better than Sol, but is the very first impression you get when you look at the first graph. If I'm not wrong OAI tried a similar trick when GPT5 was announced ... they have been criticized a lot.
  - BeetleB 1mo ago
    > It gives you a wrong perspective, especially if you are distracted, on model capabilities Only if you aren't schooled in reading graphs. It's a given that you always have to look at the axes when interpreting a graph. How exactly would you zoom into a section of a graph and just show that section?
    - Sharlin 1mo ago
      Not starting at zero has long been used entirely intentionally in order to mislead people, particularly the general public.
    - vb-8448 1mo ago
      > Only if you aren't schooled in reading graphs ... So we can say the same about the authors "AA’s plot is misleading" claim, he is "not schooled in reading graphs"? > How exactly would you zoom into a section of a graph and just show that section? When building a chart is good practice to provide log scale switch and zoom&pan capabilities, so the reader can decide how to look at it.
      - BeetleB 1mo ago
        > So we can say the same about the authors "AA’s plot is misleading" claim, he is "not schooled in reading graphs"? Oh absolutely - as other commenters have pointed out. > When building a chart is good practice to provide log scale switch and zoom&pan capabilities, so the reader can decide how to look at it. For the majority of the time charts have existed, your "good practice" would have been impossible. Charts have historically been static images (e.g. published in a journal). So there have been conventions on how to depict them - and at times it is very appropriate to start from something other than 0. Here's an article from the UK's Office For National Statistics: https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have-to-start-at-zero-part-1-line-charts/ https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have...
spider-mario 1mo ago
Starting at 0 is really only useful if you’re plotting a ratio measure and not just an interval one (https://en.wikipedia.org/wiki/Level_of_measurement#Interval_scale https://en.wikipedia.org/wiki/Level_of_measurement#Interval_... ), which I’m not sure this “intelligence index” is.

Complaining about "bad charting" and posting a chart with y-axis that doesn't start at 0 is kinda weird.

by vb-8448 1mo ago

Complaining about "bad charting" and posting a chart with y-axis that doesn't start at 0 is kinda weird.

andai 1mo ago

Yeah, this doesn't include older models, some of which were already saturating many common tasks a year ago. I'll often add them to the AA graph for reference.

paimapi 1mo ago

to be fair, you don't need to start the y-axis at zero [0] but for some of the graphs where the lowest value is close to 0 the best practice is to do so there's a fun Excel artifact where it auto-selects the 'relevant' range with no adjustment for how proportionally close to 0 the values are - a professional researcher publishing to a journal should know better (and should be ridiculed for not incorporating best practices) but for a personal blog by an SWE this really isn't the worst sin [0] https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have-to-start-at-zero-part-1-line-charts/ https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have...

vb-8448 1mo ago
IMO in this case is mandatory to start from 0 because it alters the visual perception. Just look at the first chart: the distance between Fable 5.1 and Sol is <5%, but it looks like 25 or 30%.
- _aavaa_ 1mo ago
  I disagree. The y-axis is some arbitrary intelligence score that we use as a proxy for performance on whatever our specific task happens to be. So it doesn't matter if a model is a 0, 1, or 20 along this axis, they are all useless for the tasks I want. And as the complexity of your score increases, the cutoff goes up. We can quibble about where your personal cutoff is, but it aint 0.
  - vb-8448 1mo ago
    Y-axis is between 0 and 100. But even if it was between 0 and Inf+, it still gives you a wrong perspective, especially if you are not paying attention, on model capabilities.

datadrivenangel 1mo ago

Yeah this is really funny and annoying. They get better later on in the article, but it is questionable.

BeetleB 1mo ago

Not at all, as long as it's labeled as such. Coming from engineering/science, this is common. What is bad is starting at 0, showing an indicator of a gap, and suddenly starting at 30 or whatever after the gap.

vb-8448 1mo ago
It gives you a wrong perspective, especially if you are distracted, on model capabilities: Fable 5.1 is not 30% better than Sol, but is the very first impression you get when you look at the first graph. If I'm not wrong OAI tried a similar trick when GPT5 was announced ... they have been criticized a lot.
- BeetleB 1mo ago
  > It gives you a wrong perspective, especially if you are distracted, on model capabilities Only if you aren't schooled in reading graphs. It's a given that you always have to look at the axes when interpreting a graph. How exactly would you zoom into a section of a graph and just show that section?
  - Sharlin 1mo ago
    Not starting at zero has long been used entirely intentionally in order to mislead people, particularly the general public.
  - vb-8448 1mo ago
    > Only if you aren't schooled in reading graphs ... So we can say the same about the authors "AA’s plot is misleading" claim, he is "not schooled in reading graphs"? > How exactly would you zoom into a section of a graph and just show that section? When building a chart is good practice to provide log scale switch and zoom&pan capabilities, so the reader can decide how to look at it.
    - BeetleB 1mo ago
      > So we can say the same about the authors "AA’s plot is misleading" claim, he is "not schooled in reading graphs"? Oh absolutely - as other commenters have pointed out. > When building a chart is good practice to provide log scale switch and zoom&pan capabilities, so the reader can decide how to look at it. For the majority of the time charts have existed, your "good practice" would have been impossible. Charts have historically been static images (e.g. published in a journal). So there have been conventions on how to depict them - and at times it is very appropriate to start from something other than 0. Here's an article from the UK's Office For National Statistics: https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have-to-start-at-zero-part-1-line-charts/ https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have...

spider-mario 1mo ago

Starting at 0 is really only useful if you’re plotting a ratio measure and not just an interval one (https://en.wikipedia.org/wiki/Level_of_measurement#Interval_scale https://en.wikipedia.org/wiki/Level_of_measurement#Interval_... ), which I’m not sure this “intelligence index” is.