4 ms·

What is your sense on how well current AI models can handle mathematics, vs human language? Is hallucinating or inventing info a problem with math too? Do you

What is your sense on how well current AI models can handle mathematics, vs human language? Is hallucinating or inventing info a problem with math too? Do you guys do anything to minimize, mitigate, or solve it? And do you think current AI models will be able to solve as yet unsolved problems in math and physics, perhaps with increasingly large datasets or augments like Trees of Thought or process supervision? Or do you think it will require a different fundamental model?

visarga 3y ago
Not even humans with PhD can do physics research without a lab. How could a model with just a few tokens of context window and no body do that?
- Zondartul 3y ago
  Most of researchers' work is with formulas and reading articles. Even if you had a lab, you'd still need to think long and hard about what exactly you are trying to measure, and then design an experiment on paper before it can be carried out. So it's possible, in theory, that an AI-in-a-box could do the hardest parts of science aside from experiments. Whether modern LLMs are any good at it is a different question.
- ElFitz 3y ago
  What would should have a lab have, today, in the context of research on topics like string theory? That a simple office with a desk would not have, I mean. Genuinely curious.
- SkyMarshal 3y ago
  Obviously I’m talking about mathematical/computational physics, not experimental physics. I don’t expect the AI to build itself a tokamak to run experiments on.
easygenes 3y ago
As someone who has been using GPT-4 and Claude working with Jupyter notebooks the last month, I’ll say that sometimes I am surprised by how well they can understand and segment out a problem. Other times it is obvious I’m asking too much of them, because no amount of clever reframing gets them to a right answer. A surprise win was as much luck I had talking through an issue and getting correct, functional one-shot time grid and 3D plots including extra complications for the calculation of the axes. A recent lose was trying to get it to generate a script that could optimize the gravity coefficient in the Reddit algorithm based on target timeframes and historic data. It offered salvageable output, if you pieced together several attempts and reframes, but never pulled it over the finish line.

visarga 3y ago

Not even humans with PhD can do physics research without a lab. How could a model with just a few tokens of context window and no body do that?

Zondartul 3y ago
Most of researchers' work is with formulas and reading articles. Even if you had a lab, you'd still need to think long and hard about what exactly you are trying to measure, and then design an experiment on paper before it can be carried out. So it's possible, in theory, that an AI-in-a-box could do the hardest parts of science aside from experiments. Whether modern LLMs are any good at it is a different question.
ElFitz 3y ago
What would should have a lab have, today, in the context of research on topics like string theory? That a simple office with a desk would not have, I mean. Genuinely curious.
SkyMarshal 3y ago
Obviously I’m talking about mathematical/computational physics, not experimental physics. I don’t expect the AI to build itself a tokamak to run experiments on.

easygenes 3y ago

As someone who has been using GPT-4 and Claude working with Jupyter notebooks the last month, I’ll say that sometimes I am surprised by how well they can understand and segment out a problem. Other times it is obvious I’m asking too much of them, because no amount of clever reframing gets them to a right answer. A surprise win was as much luck I had talking through an issue and getting correct, functional one-shot time grid and 3D plots including extra complications for the calculation of the axes. A recent lose was trying to get it to generate a script that could optimize the gravity coefficient in the Reddit algorithm based on target timeframes and historic data. It offered salvageable output, if you pieced together several attempts and reframes, but never pulled it over the finish line.