3 ms·

I do not share your optimism. Right now we are seeing what AI can do with one hand tied behind its back. In my experience, when Github Copilot makes mistakes th

I do not share your optimism. Right now we are seeing what AI can do with one hand tied behind its back. In my experience, when Github Copilot makes mistakes they are frequently ones that it could’ve avoided if it ran static analysis on its suggestion and realized it was mentioning variable names that aren’t in my code. If the next version of Github Copilot integrates with basic VS Code tooling like Intellisense, it will be insanely powerful. Like, I see a lot of people saying things like “It’s just pattern-matching, it still needs an engineer to guide it”. That’s true of this version of Github Copilot. If it improves at the same clip as some of these other large models, I suspect it will become much much less true in a short time horizon.

dinkumthinkum 4y ago
If that’s true why is it so bad at solving logic puzzles?
- Uehreka 4y ago
  It’s bad at solving logic puzzles today. We’ve just spent a decade-ish watching AI leapfrog over hurdle after hurdle that people said were going to be “impossible” or “would require reasoning/creativity” or “are at best 10 years away”. It feels foolish to assume we won’t see another unexpected leap. I would be unsurprised if GPT-4 or 5 suddenly becomes really good at math and logic problems. And long before that I expect AI to be hard to distinguish from a human in terms of day-to-day executive function stuff.
  - djmips 4y ago
    Everyone essentially doesn't know. We are but extrapolating something we don't understand to either future failure or success depending on a gut feeling. Some like you are saying to expect the unexpected while others are saying to expect this to stall out. I've even heard the openAI CEO couch his predictions behind 'assuming we don't stall out'. In this article and elsewhere it's been pointed out that how core abilities of the large language models are not well understood. Without this understanding everyone is just guessing at the future.
  - dinkumthinkum 4y ago
    Why would a language model like this even be good at solving logic puzzles, even GPT-20? I’m not sure west specifically knowledgeable people said would be impossible ten years ago that has been proven wrong now. Right now it’s not even good at answering facts which it has been trained on, let alone actual reasoning. Why does Sky-Net seem more likely than this specific technology having a plateau?

dinkumthinkum 4y ago

If that’s true why is it so bad at solving logic puzzles?

Uehreka 4y ago
It’s bad at solving logic puzzles today. We’ve just spent a decade-ish watching AI leapfrog over hurdle after hurdle that people said were going to be “impossible” or “would require reasoning/creativity” or “are at best 10 years away”. It feels foolish to assume we won’t see another unexpected leap. I would be unsurprised if GPT-4 or 5 suddenly becomes really good at math and logic problems. And long before that I expect AI to be hard to distinguish from a human in terms of day-to-day executive function stuff.
- djmips 4y ago
  Everyone essentially doesn't know. We are but extrapolating something we don't understand to either future failure or success depending on a gut feeling. Some like you are saying to expect the unexpected while others are saying to expect this to stall out. I've even heard the openAI CEO couch his predictions behind 'assuming we don't stall out'. In this article and elsewhere it's been pointed out that how core abilities of the large language models are not well understood. Without this understanding everyone is just guessing at the future.
- dinkumthinkum 4y ago
  Why would a language model like this even be good at solving logic puzzles, even GPT-20? I’m not sure west specifically knowledgeable people said would be impossible ten years ago that has been proven wrong now. Right now it’s not even good at answering facts which it has been trained on, let alone actual reasoning. Why does Sky-Net seem more likely than this specific technology having a plateau?