3 ms·
> Claude will readily notice when the game tells it that an attack from an electric-type Pokémon is “not very effective” against a rock-type opponent, for insta
by kibwen 2y ago
> Claude will readily notice when the game tells it that an attack from an electric-type Pokémon is “not very effective” against a rock-type opponent, for instance. Claude will then squirrel that factoid away in a massive written knowledge base for future reference later in the run.
But these models already know all this information??? Surely it's ingested Bulbapedia, along with a hundred zillion terabytes of every other Pokemon resource on the internet, so why does it need to squirrel this information away? What's the point of ruining the internet with all this damn parasitic crawling if the models can't even recall basic facts like "thunderbolt is an electric-type move", "geodude is a rock-type pokemon", "electric-type moves are ineffective against rock-type pokemon"?
- gbear605 2y agoClaude definitely does know it (I just checked). Plausibly they’re giving that as an example of other things that it actually needs to save, but yeah, it’s a bad example.
- stonemetal12 2y agoIt has trained on multiple guides and walkthroughs and whatnot for the game from the internet. Theoretically it knows the game inside and out, it can recall all of that if you ask about it. It lacks the ability to turn that into something other than text. To me that shines a light on the claim that there is a real conceptual space limited by the output being generated text.
- bob1029 2y ago> ruining the internet with all this damn parasitic crawling I feel like we may be conflating the performance implications of crawling with the intellectual property ones.
- devnullbrain 2y agoPC's complaint may be related to this: https://x.com/GergelyOrosz/status/1905000078839845303 https://x.com/GergelyOrosz/status/1905000078839845303
- ToValueFunfetti 2y agoClaude has no problem recalling type advantages, but it has no metacognition and fails terribly at knowing what it knows versus what it needs to write down.
- meroes 2y agoSame could be said for Sudoku. How many times has it seen natural numbers 1-9 and Sudoku tutorials and board states crawling the entire public internet. And yet they can’t solve a Sudoku or even a half size board on their own.
- devnullbrain 2y agoWell, it's not to play Pokemon. Pokemon is a contrived example to research and display reasoning. That would be a useful step for the real world use-cases that aren't index in fine detail, as a human would. Actually, getting this to work is important for avoiding the thing you're complaining about.
- timabdulla 2y agoThis is the most interesting aspect to me. I had Claude generate a guide to all the gyms in Pokemon Red and instructions for how to quickly execute a play through [0]. It obviously knows the game through and through. Yet even with encyclopedic knowledge of the game, it's still a struggle for it to play. Imagine giving it a gave of which it knows nothing at all. [0] https://claude.site/artifacts/d127c740-b0ab-43ba-af32-3402e694c0dd https://claude.site/artifacts/d127c740-b0ab-43ba-af32-3402e6...