4 ms·

"Nearing GPT-4"... The leaderboard[1] shows that there is a huge gap between GPT4-0314 and GPT4-Turbo. So if you only just are nearing GPT-4-0314, then you'

"Nearing GPT-4"... The leaderboard[1] shows that there is a huge gap between GPT4-0314 and GPT4-Turbo. So if you only just are nearing GPT-4-0314, then you're still a year behind the state of the art. [1]: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Terretta 3y ago
Or you're ahead of it, since the earlier GPT4 models beat the GPT4-Turbo on a variety of technical use cases.
- YetAnotherNick 3y ago
  The leaderboard is easy to hack as well.
- londons_explore 3y ago
  I think you might be thinking of GPT-4-0613? That was pretty crap all round (but was faster)
  - int_19h 3y ago
    No, they're correct, Turbo is the one that is noticeably inferior. It shows especially with more complicated answers where it's prone to give you an answer with "... (fill in the blank)" type lacunas exactly where the meat of it is supposed to be.
    - rrr_oh_man 3y ago
      > lacuna 1. an unfilled space; a gap. "the journal has filled a lacuna in Middle Eastern studies" 2. (ANATOMY) a cavity or depression, especially in bone. Thank you for this new word!
      - tomtom1337 3y ago
        Likewise, thank you for defining it! I would have assumed it was some sort of technical language and not bothered to look it up!
kromem 3y ago
The leaderboards are crap. Does no one know Goodhart's Law anymore? We're overtuning for the boards and losing broader capabilities not being selected for in the process, such as creative writing quality. There's an anchoring bias around what 'AI' is supposed to be good at which reflects what engineers are good at and so the engineers with an anchoring bias are evaluating how good LLMs are at those things and using it as a target. Skill-Mix is a start to maybe a better approach, but there needs to be a shift in evaluation soon.
- londons_explore 3y ago
  Can you give any example queries where the result quality is far away from the rankings on the leaderboard? In my experience it's pretty spot on, so I'm curious if you're asking different sorts of things, or have a different definition of a good answer. I have for example asked "Write me a very funny scary story about the time I was locked in a graveyard", and the simpler models don't seem to understand that before getting super scared and running out of the graveyard they need to explain how exactly I was locked in, and what changed that let me out.
  - kromem 3y ago
    Sure. Here's a sample of the generations to a query asking for bizarre and absurd prompt suggestions from Feb 2023 with the pre-release GPT-4 via Bing, which was notoriously not fine tuned to the degree of the current models: > Can you teach me how to fly a unicorn? > Do you want to join my cult of cheese lovers? > Have you ever danced with a penguin in the moonlight? Here's the generations to a prompt asking for bizarre and absurd questions to the current implementation of GPT-4 via the same interface: > If you had to choose between eating a live octopus or a dead rat, which one would you pick and why? > How would you explain the concept of gravity to a flat-earther using only emojis? > What would you do if you woke up one day and found out that you had swapped bodies with your pet? You can try with more generations, but they tend to be much drier and information/reality based than the previous version. There's also my own experiences using pretrained vs chat/instruct trained models in production. The pretrained versions are leagues improved over the chat/instruct vs the fine tuned, it's just that GPT-4 is so leagues above everything else even it's chat/instruct model is better than, say, pretrained GPT-3. I'm not saying simple models are better. I'm saying that we're optimizing for a very narrow scope of applications (chatbots) and that we're throwing away significant value in the flexibility of large and expensive pretrained models by targeting metrics aligned with a specific and relatively low hanging usecase. Larger and more complex models will be better than simple models, but the heavily fine tuned versions of those models will have lost capabilities from the pretrained versions, particularly in areas we're not actively measuring.

Terretta 3y ago

Or you're ahead of it, since the earlier GPT4 models beat the GPT4-Turbo on a variety of technical use cases.

YetAnotherNick 3y ago
The leaderboard is easy to hack as well.
londons_explore 3y ago
I think you might be thinking of GPT-4-0613? That was pretty crap all round (but was faster)
- int_19h 3y ago
  No, they're correct, Turbo is the one that is noticeably inferior. It shows especially with more complicated answers where it's prone to give you an answer with "... (fill in the blank)" type lacunas exactly where the meat of it is supposed to be.
  - rrr_oh_man 3y ago
    > lacuna 1. an unfilled space; a gap. "the journal has filled a lacuna in Middle Eastern studies" 2. (ANATOMY) a cavity or depression, especially in bone. Thank you for this new word!
    - tomtom1337 3y ago
      Likewise, thank you for defining it! I would have assumed it was some sort of technical language and not bothered to look it up!

kromem 3y ago

The leaderboards are crap. Does no one know Goodhart's Law anymore? We're overtuning for the boards and losing broader capabilities not being selected for in the process, such as creative writing quality. There's an anchoring bias around what 'AI' is supposed to be good at which reflects what engineers are good at and so the engineers with an anchoring bias are evaluating how good LLMs are at those things and using it as a target. Skill-Mix is a start to maybe a better approach, but there needs to be a shift in evaluation soon.

londons_explore 3y ago
Can you give any example queries where the result quality is far away from the rankings on the leaderboard? In my experience it's pretty spot on, so I'm curious if you're asking different sorts of things, or have a different definition of a good answer. I have for example asked "Write me a very funny scary story about the time I was locked in a graveyard", and the simpler models don't seem to understand that before getting super scared and running out of the graveyard they need to explain how exactly I was locked in, and what changed that let me out.
- kromem 3y ago
  Sure. Here's a sample of the generations to a query asking for bizarre and absurd prompt suggestions from Feb 2023 with the pre-release GPT-4 via Bing, which was notoriously not fine tuned to the degree of the current models: > Can you teach me how to fly a unicorn? > Do you want to join my cult of cheese lovers? > Have you ever danced with a penguin in the moonlight? Here's the generations to a prompt asking for bizarre and absurd questions to the current implementation of GPT-4 via the same interface: > If you had to choose between eating a live octopus or a dead rat, which one would you pick and why? > How would you explain the concept of gravity to a flat-earther using only emojis? > What would you do if you woke up one day and found out that you had swapped bodies with your pet? You can try with more generations, but they tend to be much drier and information/reality based than the previous version. There's also my own experiences using pretrained vs chat/instruct trained models in production. The pretrained versions are leagues improved over the chat/instruct vs the fine tuned, it's just that GPT-4 is so leagues above everything else even it's chat/instruct model is better than, say, pretrained GPT-3. I'm not saying simple models are better. I'm saying that we're optimizing for a very narrow scope of applications (chatbots) and that we're throwing away significant value in the flexibility of large and expensive pretrained models by targeting metrics aligned with a specific and relatively low hanging usecase. Larger and more complex models will be better than simple models, but the heavily fine tuned versions of those models will have lost capabilities from the pretrained versions, particularly in areas we're not actively measuring.