3 ms·

Stanford benchmarks and compares numerous Large Language Models

114 points by famouswaffles 4y ago

famouswaffles 4y ago
Interesting tidbits - Not all these models are instruct aligned - Cohere's suite is free for personal use - anthropic uses reinforcement learning from ai feedback for instruct alignment, removing humans from the process almost entirely(humans still make the rules the language model checks the responses against). - open ai's models are pretty poorly calibrated compared to its peers. Calibration is how well the model's confidence solving a problem tracks with how it actually performs that problem. It seems to be a consequence of the way they are performing the instruct tuning (or maybe it's a consequence of instruct tuning in general). The gpt-4 report actually shows this. Base gpt-4 was excellently calibrated but rlhf knocked it right out. The excellent calibration was also independently verified (in the medical domain) from the paper about gpt-4's medical prowess. https://arxiv.org/abs/2303.13375 https://arxiv.org/abs/2303.13375
- xiphias2 4y ago
  One of OpenAI's engineers / cofounders (Greg Brockman) specifically said it in an interview that GPT-4 itself is quite calibrated, it goes away with the fine tuning process (I don't remember exactly which part of the interview). Of course they are working on fixing it, I guess GPT-4.5 will contain the fix. https://www.youtube.com/watch?v=YtJEfTTD_Y4 https://www.youtube.com/watch?v=YtJEfTTD_Y4
Rapzid 4y ago
There should be a model trained only on what's publicly reachable through the great firewall. Then that should square off against a model trained on everything..
- xbmcuser 4y ago
  I feel that the one with the great firewall filter might actually be better for some domains.
keyle 4y ago
That's great, now I guess we need a page that ranks those benchmarks to tell us what we're looking at...
diziet 4y ago
It seems to be missing Llama and other Llama derived models, and GPT4.
- qwerty3344 4y ago
  > last updated 2023-03-19 Yep, anything that old is ancient in the AI world
  - nextworddev 4y ago
    Singularity when? Plot twist:..
    - xbmcuser 4y ago
      Singularity will happen when we have real quantum computing then we wont know if the Ai is alive or not or both.
      - kevviiinn 4y ago
        I don't think that's how that works
lee101 4y ago
[dead]

114 points by famouswaffles 4y ago

Interesting tidbits - Not all these models are instruct aligned - Cohere's suite is free for personal use - anthropic uses reinforcement learning from ai feedback for instruct alignment, removing humans from the process almost entirely(humans still make the rules the language model checks the responses against). - open ai's models are pretty poorly calibrated compared to its peers. Calibration is how well the model's confidence solving a problem tracks with how it actually performs that problem. It seems to be a consequence of the way they are performing the instruct tuning (or maybe it's a consequence of instruct tuning in general). The gpt-4 report actually shows this. Base gpt-4 was excellently calibrated but rlhf knocked it right out. The excellent calibration was also independently verified (in the medical domain) from the paper about gpt-4's medical prowess. https://arxiv.org/abs/2303.13375 https://arxiv.org/abs/2303.13375

xiphias2 4y ago
One of OpenAI's engineers / cofounders (Greg Brockman) specifically said it in an interview that GPT-4 itself is quite calibrated, it goes away with the fine tuning process (I don't remember exactly which part of the interview). Of course they are working on fixing it, I guess GPT-4.5 will contain the fix. https://www.youtube.com/watch?v=YtJEfTTD_Y4 https://www.youtube.com/watch?v=YtJEfTTD_Y4

Rapzid 4y ago

There should be a model trained only on what's publicly reachable through the great firewall. Then that should square off against a model trained on everything..

xbmcuser 4y ago
I feel that the one with the great firewall filter might actually be better for some domains.

keyle 4y ago

That's great, now I guess we need a page that ranks those benchmarks to tell us what we're looking at...

diziet 4y ago

It seems to be missing Llama and other Llama derived models, and GPT4.

qwerty3344 4y ago
> last updated 2023-03-19 Yep, anything that old is ancient in the AI world
- nextworddev 4y ago
  Singularity when? Plot twist:..
  - xbmcuser 4y ago
    Singularity will happen when we have real quantum computing then we wont know if the Ai is alive or not or both.
    - kevviiinn 4y ago
      I don't think that's how that works

lee101 4y ago

[dead]