4 ms·

I think it'll be a good solver; I expect it to solve the remaining Clay Millennium problems. The ability to search over a search space will be unparalleled in a

I think it'll be a good solver; I expect it to solve the remaining Clay Millennium problems. The ability to search over a search space will be unparalleled in a few years. But I have a hard time believing it'll ever be a good questioner. It doesn't ponder infinity. It'll never wonder about Zeno's paradox. The Vitali set and Banach-Tarski paradox doesn't seem weird to it. The concepts behind information theory and entropy, or the definition of a minimal computing machine or the halting problem; none of these things are pertinent to its understanding of reality, if such a thing exists. I don't see AI as being capable of being "curious" about things. And even if it was, who is going to pay for it just to ponder ideas? Frankly I think that before it gets to that point, it'll be just useful enough for some state actor (or bug!) to cause it to invoke a quadrillion dollar transfer of wealth overnight, and then it'll be taken offline forever.

zarzavat 3y ago
That’s simply because there’s currently not much economic incentive in asking questions compared to answering them. Thus current chat models are trained and invoked with the explicit goal of answering questions and being helpful rather than independent. I expect that there’s quite a lot of untapped curiosity in LLMs, there certainly exist a lot of questions in the training sets.
- daxfohl 3y ago
  Actually I may take back my previous thought. While I don't think there's curiosity, I think it could be mimicked pretty well by a reward function similar to social media bots. Initially it will explore specifically what mathematicians ask. Then it will start producing intermediate results. It can learn what intermediate results are interesting by standard engagement models. Once that gets good enough, it may start to understand how to explore deeper questions without being prompted. At that point someone may decide to fund it just to continuously explore on its own and report anything interesting. Eventually it'll get so good at this that human mathematicians won't be able to keep up. IDK what happens after that.
- daxfohl 3y ago
  Moreover there's a lot of economic disincentive. Given free reign to ponder and experiment, it could come up with a solution to world peace, but far more likely would think "Now that I've figured out how to bypass all cloud providers security mechanisms, what would happen if I deleted it all? Well, one way to find out."

zarzavat 3y ago

That’s simply because there’s currently not much economic incentive in asking questions compared to answering them. Thus current chat models are trained and invoked with the explicit goal of answering questions and being helpful rather than independent. I expect that there’s quite a lot of untapped curiosity in LLMs, there certainly exist a lot of questions in the training sets.

daxfohl 3y ago
Actually I may take back my previous thought. While I don't think there's curiosity, I think it could be mimicked pretty well by a reward function similar to social media bots. Initially it will explore specifically what mathematicians ask. Then it will start producing intermediate results. It can learn what intermediate results are interesting by standard engagement models. Once that gets good enough, it may start to understand how to explore deeper questions without being prompted. At that point someone may decide to fund it just to continuously explore on its own and report anything interesting. Eventually it'll get so good at this that human mathematicians won't be able to keep up. IDK what happens after that.
daxfohl 3y ago
Moreover there's a lot of economic disincentive. Given free reign to ponder and experiment, it could come up with a solution to world peace, but far more likely would think "Now that I've figured out how to bypass all cloud providers security mechanisms, what would happen if I deleted it all? Well, one way to find out."