4 ms·

the excellent performance demonstrated by the models fully proves the crucial role of reinforcement learning in the optimization process What if this reinforce

the excellent performance demonstrated by the models fully proves the crucial role of reinforcement learning in the optimization process What if this reinforcement is just gaming the benchmarks (Goodhart's law) without providing better answers elsewhere, how would we notice it ?

m3kw9 2y ago
When actual people start using it
dartos 2y ago
I mean all optimization algorithms do is game a benchmark. That’s the whole point. The hard part is making the benchmark meaningful in the first place.
- TeMPOraL 2y ago
  Yeah, and if anything, RL has a rep of being too good at this job, because of all the cases where it gamed a benchmark by picking up on some environmental factor the supervisors hadn't thought of (numerical instabilities, rounding, bugs, etc.).
  - porridgeraisin 2y ago
    My favourite is this one: https://news.ycombinator.com/item?id=43113941 https://news.ycombinator.com/item?id=43113941
    - magicalhippo 2y ago
      The ML version of Professor Farnsworth[1]: It came to me in a dream, and I forgot it in another dream. [1]: https://www.imdb.com/title/tt0584424/quotes/?item=qt0439248 https://www.imdb.com/title/tt0584424/quotes/?item=qt0439248
- einpoklum 2y ago
  No, that is patently false. Many optimization algorithms which computer scientists, mathematicians or software developers devise do not involve benchmakrs at all, and apply to all possible inputs/instances of their respective computational problems.
  - CuriouslyC 2y ago
    Plot twist: the loss function for training is basically a benchmark
    - dartos 2y ago
      Is it a plot twist if it’s the whole plot?
  - brookst 2y ago
    Example?
    - hgomersall 2y ago
      Those times when people write code with loads of theoretical micro optimisations that they never actually test against a benchmark because otherwise they wouldn't do it.
      - brookst 2y ago
        Perhaps “example” means different things in different cultures.
        hgomersall 2y ago
        I was being somewhat facetious.
mentalgear 2y ago
The trick is that the benchmarks must have a wide enough distribution so that a well scoring model is potentially useful for the widest span of users. There also would need to be a guarantee (or checking of the model somehow) that model providers don't just train on the benchmarks. Solutions are dynamic components (random names, numbers, etc) or private parts of benchmarks.
- brookst 2y ago
  A common pattern is for benchmarks owners to hold back X% of their set so they can independently validate that models perform similarly on the holdback set. See: FrontierMath / OpenAI brouhaha.
CamperBob2 2y ago
You could ask the same question of a student who has just graduated after passing specific tests in school.
- brookst 2y ago
  Student, lawyer, doctor, etc.
porridgeraisin 2y ago
Typically you train it on one set and test it on another set. If you see that the differences between the two sets are significant enough and yet it has maintained good performance on the test set, you claim that it has done something useful [alongside gaming the benchmark that is the train set]. That "side effect" is always the useful part in any ML process. If the test set is extremely similar to the train set then yes, it's goodharts law all around. For modern LLMs, it's hard to make a test set that is different from what it has trained on, because of the sheer expanse of the training data used. Note that the two sets are different only if they are statistically different. It is not enough that they simply don't repeat verbatim.
Lerc 2y ago
A large amount of work in the last few years has gone into building benchmarks because models have been going though and beating them at a fairly astonishing rate. It's generally accepted as true that passing any one of them does not constitute fully general intelligence but the difficult part has been finding things that they cannot do. They are giving them more and more difficult tasks. The ARC prize in particular was designed to be focused on reasoning more than knowledge. The 87.5% score achieved in such a short time by throwing lots of resources at conventional methods was quite a surprise. You can at least have a degree of confidence that they will perform well in the areas covered by the benchmarks (as long as they weren't contaminated) and with enough benchmarks you get fairly broad coverage.
- aydyn 2y ago
  > does not constitute fully general intelligence but the difficult part has been finding things that they cannot do I am very surprised when people say things like this. For example, the best ChatGPT model continues to lie to me on a daily basis for even basic things. E.g. when I ask it to explain what code is contained on a certain line on github, it just makes up the code and the code it's "explaining" isn't found anywhere in the repo. From my experience, every model is untrustworthy and full of hallucinations. I have a big disconnect when people say things like this. Why?
  - daniel_iversen 2y ago
    I’m splitting hairs a little bit but I feel like there should be a difference in how we think about current “hard(er)” limitations of the models vs limits in general intelligence and reasoning, I.e I think the grandparent comment is talking about overall advancement in reasoning and logic and in that finding things AI “cannot do” whereas you’re referring to what is more classify as a “known issue”. Of course it’s an important issue that needs to get fixed and yes technically until we don’t have that kind of issue we can’t call it “general intelligence” but I do think the original comment is about something different than a few known limitations that probably a lot of models have (and that frankly you’d have thought wouldn’t be that difficult to solve!?)
kittikitti 2y ago
We've been able to pass the Turing Test on text, audio, and short form video (think AI's on video passing coding tests). I think there's an important distinction now with AI streamers where people notice they are AI's eventually. Now there might pop up AI streamers where you don't know they're an AI. However, there's a ceiling on how far digital interactions on the Turing Test can go. The next big hurdle towards AGI is physical interactions, like entering a room.

m3kw9 2y ago

When actual people start using it

dartos 2y ago

I mean all optimization algorithms do is game a benchmark. That’s the whole point. The hard part is making the benchmark meaningful in the first place.

TeMPOraL 2y ago
Yeah, and if anything, RL has a rep of being too good at this job, because of all the cases where it gamed a benchmark by picking up on some environmental factor the supervisors hadn't thought of (numerical instabilities, rounding, bugs, etc.).
- porridgeraisin 2y ago
  My favourite is this one: https://news.ycombinator.com/item?id=43113941 https://news.ycombinator.com/item?id=43113941
  - magicalhippo 2y ago
    The ML version of Professor Farnsworth[1]: It came to me in a dream, and I forgot it in another dream. [1]: https://www.imdb.com/title/tt0584424/quotes/?item=qt0439248 https://www.imdb.com/title/tt0584424/quotes/?item=qt0439248
einpoklum 2y ago
No, that is patently false. Many optimization algorithms which computer scientists, mathematicians or software developers devise do not involve benchmakrs at all, and apply to all possible inputs/instances of their respective computational problems.
- CuriouslyC 2y ago
  Plot twist: the loss function for training is basically a benchmark
  - dartos 2y ago
    Is it a plot twist if it’s the whole plot?
- brookst 2y ago
  Example?
  - hgomersall 2y ago
    Those times when people write code with loads of theoretical micro optimisations that they never actually test against a benchmark because otherwise they wouldn't do it.
    - brookst 2y ago
      Perhaps “example” means different things in different cultures.
      - hgomersall 2y ago
        I was being somewhat facetious.

mentalgear 2y ago

The trick is that the benchmarks must have a wide enough distribution so that a well scoring model is potentially useful for the widest span of users. There also would need to be a guarantee (or checking of the model somehow) that model providers don't just train on the benchmarks. Solutions are dynamic components (random names, numbers, etc) or private parts of benchmarks.

brookst 2y ago
A common pattern is for benchmarks owners to hold back X% of their set so they can independently validate that models perform similarly on the holdback set. See: FrontierMath / OpenAI brouhaha.

CamperBob2 2y ago

You could ask the same question of a student who has just graduated after passing specific tests in school.

brookst 2y ago
Student, lawyer, doctor, etc.

porridgeraisin 2y ago

Typically you train it on one set and test it on another set. If you see that the differences between the two sets are significant enough and yet it has maintained good performance on the test set, you claim that it has done something useful [alongside gaming the benchmark that is the train set]. That "side effect" is always the useful part in any ML process. If the test set is extremely similar to the train set then yes, it's goodharts law all around. For modern LLMs, it's hard to make a test set that is different from what it has trained on, because of the sheer expanse of the training data used. Note that the two sets are different only if they are statistically different. It is not enough that they simply don't repeat verbatim.

Lerc 2y ago

A large amount of work in the last few years has gone into building benchmarks because models have been going though and beating them at a fairly astonishing rate. It's generally accepted as true that passing any one of them does not constitute fully general intelligence but the difficult part has been finding things that they cannot do. They are giving them more and more difficult tasks. The ARC prize in particular was designed to be focused on reasoning more than knowledge. The 87.5% score achieved in such a short time by throwing lots of resources at conventional methods was quite a surprise. You can at least have a degree of confidence that they will perform well in the areas covered by the benchmarks (as long as they weren't contaminated) and with enough benchmarks you get fairly broad coverage.

aydyn 2y ago
> does not constitute fully general intelligence but the difficult part has been finding things that they cannot do I am very surprised when people say things like this. For example, the best ChatGPT model continues to lie to me on a daily basis for even basic things. E.g. when I ask it to explain what code is contained on a certain line on github, it just makes up the code and the code it's "explaining" isn't found anywhere in the repo. From my experience, every model is untrustworthy and full of hallucinations. I have a big disconnect when people say things like this. Why?
- daniel_iversen 2y ago
  I’m splitting hairs a little bit but I feel like there should be a difference in how we think about current “hard(er)” limitations of the models vs limits in general intelligence and reasoning, I.e I think the grandparent comment is talking about overall advancement in reasoning and logic and in that finding things AI “cannot do” whereas you’re referring to what is more classify as a “known issue”. Of course it’s an important issue that needs to get fixed and yes technically until we don’t have that kind of issue we can’t call it “general intelligence” but I do think the original comment is about something different than a few known limitations that probably a lot of models have (and that frankly you’d have thought wouldn’t be that difficult to solve!?)

kittikitti 2y ago

We've been able to pass the Turing Test on text, audio, and short form video (think AI's on video passing coding tests). I think there's an important distinction now with AI streamers where people notice they are AI's eventually. Now there might pop up AI streamers where you don't know they're an AI. However, there's a ceiling on how far digital interactions on the Turing Test can go. The next big hurdle towards AGI is physical interactions, like entering a room.