LLMs don't do math in that sense. They build a string of tokens out of a billion pre-weighted ones that gets a favorable probably distribution when taking your

LLMs don't do math in that sense. They build a string of tokens out of a billion pre-weighted ones that gets a favorable probably distribution when taking your prompt into account. Change your prompt, get a different printout. There is no semantic understanding (in the sense of does what is printed make sense) and it therefore cannot plausibility check its response. A LLM will just print gibberish if it gets the best probability distribution of tokens. I'm sure that's something that will be addressed over time, but we are not there yet. I'm not keen on marketing words like "superintelligence" but boiling it down that's what in my mind the OP said. These systems are limited in ways that we do not yet fully appreciate. They are not silver bullets for all or maybe even many problems. We need to figure out where they can be deployed for greater benefit.