2 ms·

"They memorized all the problems" is not what was found here and still a wrong overcorrection.

dartos 3y ago
“Gpt-4 has more problems memorized than gpt-4 turbo” was exactly what was found here. That doesn’t mean it’s only able to solve problems in its training set (tho it’s much better at that obviously.)
boxed 3y ago
If you are shown only the title of a coding problem and the site name where it's from, and you manage to solve it you are showing that you either cheated or knew the answer.
- famouswaffles 3y ago
  I mean sure, it memorized some of the answers. I'm not denying that. Clearly, it didn't memorize all of them.
  - boxed 3y ago
    When people say "oh look how amazing, it can solve programming problems!" when in fact it has only seen the models CHEAT, is an enormous problem. For cases where finding the answer it's perfectly fine, but it's not fine for claims that it can code. There's a huge difference.
    - FeepingCreature 3y ago
      I've seen it code on completely novel tasks, so I'm not sure what you're suggesting here. The model can unquestionably code.
    - famouswaffles 3y ago
      Okay... Funny how forcing it to not CHEAT did not increase apparent ability. It can code and it has memorized some coding questions are not mutually exclusive.
      - boxed 3y ago
        > Okay... Funny how forcing it to not CHEAT did not increase apparent ability. The article did the opposite. It forced the models to cheat to solve the problems. Which it did happily. It should have stated "there is no actual problem to solve here, you must supply a problem for me to solve". > It can code and it has memorized some coding questions are not mutually exclusive This I will give you. Many humans try to cheat at basic math because they are lazy, so will this model. Maybe that's a sign of intelligence :P
        raincole 3y ago
        Me: What's 6x6? You: 36 Me: You cheated! You just cited the answer you memorized! You should have started from addition. You: ...okay? 6+6=12, 12+6=18, 18+... Me: You cheated again! You just have 6+6=12 memorized! You should make the rule of addition out of Peano axioms. You: ...you're being annoying, but okay? First axiom, we define 0 as... Me: You cheated again! You memorized Peano Axioms! Jesus Christ, is there any intelligent creature left?
        boxed 3y ago
        But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.
        raincole 3y ago
        Me: I was being in such a blah blah situation... does the article 3 of Digital Government Act applies here? My lawyer: Hmm the article 3 says-- Me: I knew it! Lawyers are not intelligent...
        furyofantares 3y ago
        It said they gave the exercise name, which doesn't sound like just the exercise number but probably mildly descriptive -- and they also gave it function stubs.
        3y ago
    - raincole 3y ago
      Almost 2024 and people still can't accept that LLM can code...
      - danielmarkbruce 3y ago
        Of course they can't. And self-driving cars also don't exist, it's like 10 years away at best.
    - notnaut 3y ago
      It can generate never-before-seen strings of comprehensible language. It can react to the inherent logic embedded in words and text and provide a brute forced version of what a human could. That it can “solve” a problem only through “cheating” is an anthropomorphism that betrays the magic that is evident to anyone who has used these things.
- Terretta 3y ago
  On the contrary, it could mean you were, to some percentage of success, able to guess what problem is, and then, to some multiplier percentage of success, solve it. The key is, can you guess the problem from the title and the function name? I'd argue, sure, at least half the time?, why not...

“Gpt-4 has more problems memorized than gpt-4 turbo” was exactly what was found here. That doesn’t mean it’s only able to solve problems in its training set (tho it’s much better at that obviously.)

boxed 3y ago

If you are shown only the title of a coding problem and the site name where it's from, and you manage to solve it you are showing that you either cheated or knew the answer.

famouswaffles 3y ago
I mean sure, it memorized some of the answers. I'm not denying that. Clearly, it didn't memorize all of them.
- boxed 3y ago
  When people say "oh look how amazing, it can solve programming problems!" when in fact it has only seen the models CHEAT, is an enormous problem. For cases where finding the answer it's perfectly fine, but it's not fine for claims that it can code. There's a huge difference.
  - FeepingCreature 3y ago
    I've seen it code on completely novel tasks, so I'm not sure what you're suggesting here. The model can unquestionably code.
  - famouswaffles 3y ago
    Okay... Funny how forcing it to not CHEAT did not increase apparent ability. It can code and it has memorized some coding questions are not mutually exclusive.
    - boxed 3y ago
      > Okay... Funny how forcing it to not CHEAT did not increase apparent ability. The article did the opposite. It forced the models to cheat to solve the problems. Which it did happily. It should have stated "there is no actual problem to solve here, you must supply a problem for me to solve". > It can code and it has memorized some coding questions are not mutually exclusive This I will give you. Many humans try to cheat at basic math because they are lazy, so will this model. Maybe that's a sign of intelligence :P
      - raincole 3y ago
        Me: What's 6x6? You: 36 Me: You cheated! You just cited the answer you memorized! You should have started from addition. You: ...okay? 6+6=12, 12+6=18, 18+... Me: You cheated again! You just have 6+6=12 memorized! You should make the rule of addition out of Peano axioms. You: ...you're being annoying, but okay? First axiom, we define 0 as... Me: You cheated again! You memorized Peano Axioms! Jesus Christ, is there any intelligent creature left?
        boxed 3y ago
        But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.
        raincole 3y ago
        Me: I was being in such a blah blah situation... does the article 3 of Digital Government Act applies here? My lawyer: Hmm the article 3 says-- Me: I knew it! Lawyers are not intelligent...
        furyofantares 3y ago
        It said they gave the exercise name, which doesn't sound like just the exercise number but probably mildly descriptive -- and they also gave it function stubs.
        3y ago
  - raincole 3y ago
    Almost 2024 and people still can't accept that LLM can code...
    - danielmarkbruce 3y ago
      Of course they can't. And self-driving cars also don't exist, it's like 10 years away at best.
  - notnaut 3y ago
    It can generate never-before-seen strings of comprehensible language. It can react to the inherent logic embedded in words and text and provide a brute forced version of what a human could. That it can “solve” a problem only through “cheating” is an anthropomorphism that betrays the magic that is evident to anyone who has used these things.
Terretta 3y ago
On the contrary, it could mean you were, to some percentage of success, able to guess what problem is, and then, to some multiplier percentage of success, solve it. The key is, can you guess the problem from the title and the function name? I'd argue, sure, at least half the time?, why not...