4 ms·

> To me they seem to be pretty damn smart That's the sorcery mentioned in the GP, the issue comes when people believe it to be smart however in reality it is

by bilekas 6mo ago

> To me they seem to be pretty damn smart That's the sorcery mentioned in the GP, the issue comes when people believe it to be smart however in reality it is just a next word prediction. Gives the impression it's actually thinking, and this is by design. Personally I think it's dangerous in the sense it gives users a false sense of confidence in the LLM and so a LOT of people will blindly trust it. This isn't a good thing.

handoflixue 6mo ago
What's the difference between "smart" and "next word prediction", at this point? Back when they first came out, sure, but now they can write code and create art. What would it take for you to concede a future model was smart?
- bilekas 6mo ago
  My personal take would always be that it produces something that isn't in the training set, ie: Demonstrable Creativity, or innovation. For example, it's training set it purely engineering and code with general language data set, would be "aware" what art is, but has never seen an artistic image, aware what colours are and able to create something it never saw before. Like a child with a paintbrush, there is an intuitive behavior that happens.
  - handoflixue 6mo ago
    Can you name any examples of a human doing this? I learned about colors, color theory, and so forth in school. I've definitely seen artistic images before. They can already create something they've never seen - you can prompt ChatGPT to generate images, and there's a few dedicated models for it: https://chatgpt.com/images/ https://chatgpt.com/images/ Terence Tao feels like they've done innovative work on mathematics: https://www.scientificamerican.com/article/amateur-armed-with-chatgpt-vibe-maths-a-60-year-old-problem/ https://www.scientificamerican.com/article/amateur-armed-wit...
jeremyjh 6mo ago
I'm curious how you think "word predictor" meaningfully describes an instruct model that has developed novel mathematical proofs that have eluded mathematicians for decades? edit: You cannot predict all the actions or words of someone smarter than you. If I could always predict Magnus Carlsen's next chess move, I'd be at least as good at chess as Magnus - and that would have to involve a deep understanding of chess, even if I can't explain my understanding. I can't predict the next token in a novel mathematical proof unless I've already understood the solution.
- GigaDingus 6mo ago
  I think that's more of a limitation in how people think about word predictors If you can predict the words a bright person will say about X... Isn't that some truly astounding tool? That could be used in myriad useful ways if one is a little creative with it Since it's also "alien" it can also detect and explore paths that we simply haven't noticed since their biases aren't quite the same as ours
  - deleted 6mo ago
    [deleted]
- ThrowawayR2 6mo ago
  Terence Tao himself answers that question (https://www.nature.com/articles/d41586-026-01246-9 https://www.nature.com/articles/d41586-026-01246-9) : "In almost any other application, the biggest Achilles heel of AI is that it makes unverifiable mistakes. But in mathematics, almost uniquely, you can automatically check the output — at least if the output is supposed to be the proof of a theorem, although that is not the only thing mathematicians do. So, AI companies have recognized that their most unambiguous successes — if they’re going to have any — are going to come from mathematics. In my opinion, there are many use cases of AI that are risky and controversial. In mathematics, the downsides are much more limited" AI successes in mathematics don't generalize to successes in other fields as the AI promoters want to suggest.
  - jeremyjh 6mo ago
killerstorm 6mo ago
Why do you assume I'm naive? I knew how LLMs work since 2019 and I've been testing their capabilities. I believe they actually are smart in every meaningful way. "Next word prediction" just means that answer is generated through computation. I don't think computation can't be smart. If you believe that LLMs are probabilitic and humans aren't, how do you explain randomness in human behavior? E.g. people making random typos. Have you ever tried to analyze your own behavior, understand how you function? Or do you just inherently believe you're smarter than any computation?

by bilekas 6mo ago

handoflixue 6mo ago

What's the difference between "smart" and "next word prediction", at this point? Back when they first came out, sure, but now they can write code and create art. What would it take for you to concede a future model was smart?

bilekas 6mo ago
My personal take would always be that it produces something that isn't in the training set, ie: Demonstrable Creativity, or innovation. For example, it's training set it purely engineering and code with general language data set, would be "aware" what art is, but has never seen an artistic image, aware what colours are and able to create something it never saw before. Like a child with a paintbrush, there is an intuitive behavior that happens.
- handoflixue 6mo ago
  Can you name any examples of a human doing this? I learned about colors, color theory, and so forth in school. I've definitely seen artistic images before. They can already create something they've never seen - you can prompt ChatGPT to generate images, and there's a few dedicated models for it: https://chatgpt.com/images/ https://chatgpt.com/images/ Terence Tao feels like they've done innovative work on mathematics: https://www.scientificamerican.com/article/amateur-armed-with-chatgpt-vibe-maths-a-60-year-old-problem/ https://www.scientificamerican.com/article/amateur-armed-wit...

jeremyjh 6mo ago

I'm curious how you think "word predictor" meaningfully describes an instruct model that has developed novel mathematical proofs that have eluded mathematicians for decades? edit: You cannot predict all the actions or words of someone smarter than you. If I could always predict Magnus Carlsen's next chess move, I'd be at least as good at chess as Magnus - and that would have to involve a deep understanding of chess, even if I can't explain my understanding. I can't predict the next token in a novel mathematical proof unless I've already understood the solution.

GigaDingus 6mo ago
I think that's more of a limitation in how people think about word predictors If you can predict the words a bright person will say about X... Isn't that some truly astounding tool? That could be used in myriad useful ways if one is a little creative with it Since it's also "alien" it can also detect and explore paths that we simply haven't noticed since their biases aren't quite the same as ours
- deleted 6mo ago
  [deleted]
ThrowawayR2 6mo ago
Terence Tao himself answers that question (https://www.nature.com/articles/d41586-026-01246-9 https://www.nature.com/articles/d41586-026-01246-9) : "In almost any other application, the biggest Achilles heel of AI is that it makes unverifiable mistakes. But in mathematics, almost uniquely, you can automatically check the output — at least if the output is supposed to be the proof of a theorem, although that is not the only thing mathematicians do. So, AI companies have recognized that their most unambiguous successes — if they’re going to have any — are going to come from mathematics. In my opinion, there are many use cases of AI that are risky and controversial. In mathematics, the downsides are much more limited" AI successes in mathematics don't generalize to successes in other fields as the AI promoters want to suggest.
- jeremyjh 6mo ago

killerstorm 6mo ago

Why do you assume I'm naive? I knew how LLMs work since 2019 and I've been testing their capabilities. I believe they actually are smart in every meaningful way. "Next word prediction" just means that answer is generated through computation. I don't think computation can't be smart. If you believe that LLMs are probabilitic and humans aren't, how do you explain randomness in human behavior? E.g. people making random typos. Have you ever tried to analyze your own behavior, understand how you function? Or do you just inherently believe you're smarter than any computation?