3 ms·

> the real-world stupidity of these models such as his example of a coding agent toggling between generating bug #1 vs bug #2, which in fact largely explains wh

by riku_iki 11mo ago

> the real-world stupidity of these models such as his example of a coding agent toggling between generating bug #1 vs bug #2, which in fact largely explains why the current economic and visible impact is much less than if the "AI is PhD level" benchmark narrative was actually true. this could be true in the past, but in recent weeks I started more and more trust top AI models and less PhDs I work with. Quality jump is very real imo.

nateglims 11mo ago
Are you a mathematician? I’m not an expert on the math field but it seems like they are hitting the same issues everyone else has: current LLMs still more or less need to be supervised by an expert and struggle to do something actually novel or build out a complicated proof correctly.
- riku_iki 11mo ago
  I work in math heavy applied setting. Randomly hired PhDs are also need to be supervised, end results being monitored, code be reviewed or they will make lots of mistakes, and my view is if you throw some problem like: build optimization model for this kind of problem on this kind of data, LLMs may produce better results.
- HarHarVeryFunny 11mo ago
  There's a limit to how much novelty you're going to get from an LLM, especially in areas like programming and math where they've been heavily RL'd NOT to be novel, even to extent that the base model supports, and instead generate much narrower more proscribed outputs. The limit to the novelty you are going to get from an LLM is essentially the "deductive/generative closure" of the training data. To be truly novel and move past the limits of your own past experience requires things like curiosity, continual learning, and the autonomy/agency to explore and learn.
  - riku_iki 11mo ago
    but what is the share of PhD workforce who is doing novel and creative things compared to following some mechanical workflow?

nateglims 11mo ago

Are you a mathematician? I’m not an expert on the math field but it seems like they are hitting the same issues everyone else has: current LLMs still more or less need to be supervised by an expert and struggle to do something actually novel or build out a complicated proof correctly.

riku_iki 11mo ago
I work in math heavy applied setting. Randomly hired PhDs are also need to be supervised, end results being monitored, code be reviewed or they will make lots of mistakes, and my view is if you throw some problem like: build optimization model for this kind of problem on this kind of data, LLMs may produce better results.
HarHarVeryFunny 11mo ago
There's a limit to how much novelty you're going to get from an LLM, especially in areas like programming and math where they've been heavily RL'd NOT to be novel, even to extent that the base model supports, and instead generate much narrower more proscribed outputs. The limit to the novelty you are going to get from an LLM is essentially the "deductive/generative closure" of the training data. To be truly novel and move past the limits of your own past experience requires things like curiosity, continual learning, and the autonomy/agency to explore and learn.
- riku_iki 11mo ago
  but what is the share of PhD workforce who is doing novel and creative things compared to following some mechanical workflow?