5 ms·

Our current "AI" does only interpolation, not extrapolation. You feed it a large input corpus and it digests it in clever ways. Then when you probe for somethi

by sfink 4y ago

Our current "AI" does only interpolation, not extrapolation. You feed it a large input corpus and it digests it in clever ways. Then when you probe for something contained wholly within the space described by that corpus, it is amazingly good at fabricating something plausible to match the point in that space that you requested. Which covers a lot of stuff and is very useful, but does very little for problems that require extrapolation. It can't expand the edges, it can't come up with anything truly original. It can't solve problems that people haven't already solved and written down the solutions somewhere the AI could find them. Another way of saying it: AI today is much better at memory than thought. Career advice for young people today: specialize in pushing the edges, not in filling in details. There will be some areas where applying existing stuff will hold out and be useful for a long time, but you'll always be racing against the AI. Colonize the parts of problem space where AI doesn't have the imagination to go. (Of course, humans are notoriously bad at correctly recognizing what does or doesn't require originality. Hell, we think we're making decisions about what to do every minute of every day, when in fact we're just a bunch of dancing meat automata following ingrained patterns 99% of the time.)

Valakas_ 4y ago
This goes back to the root of what creativity really is. Some say creative people don't have anything special in their "brains". They are just able to mix match stuff they already know in different ways, that others didn't consider. In essence there is no such thing as extrapolation for the human mind. We only interpolate in new ways. We can only remix what we know, and with that generate something new. How do we actually get new, non-interpolated stuff? By going out into the world and doing things with it. Collecting new data. Experiments, experiencing things, seeing new stuff, reading about new stuff. That's not so different from the AI. Our brain is not something magical that works in mysterious un-replicable ways. Much of AI field has been in fact trying to replicate how we learn, and how our brain works.
visarga 4y ago
When verification is possible, the interpolation machine starts to extrapolate. There is a process that generates ideas/solutions, and a process that tests them. An artist and a critic, a scientist and a lab. Together they form the experimentation loop. Let's take the game of Go for example. Testing who won a game is trivial. AlphaGo managed to beat humans in a few days of self-training. In other words, those edges you speak about can be pushed with massive search and verification. There is no reason we can't do massive search + verification for math and code. This is a good way to create training data where it doesn't exist in sufficient quantity. Other things can be simulated with expensive computation, and then "distilled" into fast neural networks. Then we apply the neural net to fast-search solutions. In the end we need to verify some of them (thinking of weather simulations, new materials, new drugs, ...) Also reminded of the recent AlphaTensor who leveraged massive learning from verification to beat Strassen's algorithm who was state of the art for 50 years. The is no reason neural nets should remain purely interpolative if we can manufacture good training data by running computation or experiments. Ideas are cheap, verification matters. Generative outputs are worthless without verification.
tbalsam 4y ago
Er....this feels like every main criticism going back to even the expert systems pre-90's. Could you please provide some support for your argument? This was repeated a lot in the early days of the modern wave (2012-2016) but was pretty thoroughly debunked as we've explored generalization and how these models disentangle intrinsic concepts and compute with them. Heck, even modern transformers are restricted memory Turing complete and use that to their advantage. Also, I do not mean to be rude, but frankly saying "colonize the parts of the problem space where AI doesn't have the imagination to go" is frankly rather terrible as it leans on the imagination argument of AI. At this point, being adaptable to co-integrate will be good, otherwise I could see people following that stuck inside of some kind of Sisyphean pseuso-Luddite escapist nightmare. Source for opinions: have been involved in ML in some form for most of the modern wave, and am appropriately (quite) skeptical about the AI takeover/revolution/eventual singularity belief/etc.
MetaWhirledPeas 4y ago
It can write an original poem using unique prompts I give it. Are you suggesting a similar poem exists somewhere else? I think not. It cobbles together examples, sure, but that's exactly what we do too. Everything "original" we make is riddled with subconscious outside influences.
klabb3 4y ago
Like other responders, I think your take is oversimplifying. It’s hard to both classify what imagination is, how to judge whether AI has it, and we also have humans tuning these models, which may or may not contribute to its blandness. That said, I think it’s still fair to say that most interpolation type of work is looking very threatened by AI as it is today, so the converse (don’t go into those fields) is probably sound advice in light of current developments. As for the rest, I suspect we’ll be forced to chisel off piece by piece from our zeitgeist of “imagination”. Some pieces will fall off quickly, as AIs replace them, and some will take longer, perhaps a lot longer. I do find it interesting that computers ended up killing it in unpredictable domains such as style transfer and NLP, while being mediocre-at-best in eg humor. It may mean that we have over- and underestimated aspects of what traits are unique and sophisticated.
willbudd 4y ago
> It can't expand the edges, it can't come up with anything truly original. You're dead wrong about that. Remember Deepmind's original claim to fame? Alphago? It came up with tactics and strategies that top professional go players considered to be novel to the point of being revolutionary. If you look at the way games are played by human professionals in the years before and after Alphago came on to the stage, you'll find they're different to an extent greater than any other upheaval in the many centuries-long history of the game. AI is a much more diverse field than you seem to give it credit for. Paint with overly broad strokes like that, and you're bound to go astray.

by sfink 4y ago

Valakas_ 4y ago

This goes back to the root of what creativity really is. Some say creative people don't have anything special in their "brains". They are just able to mix match stuff they already know in different ways, that others didn't consider. In essence there is no such thing as extrapolation for the human mind. We only interpolate in new ways. We can only remix what we know, and with that generate something new. How do we actually get new, non-interpolated stuff? By going out into the world and doing things with it. Collecting new data. Experiments, experiencing things, seeing new stuff, reading about new stuff. That's not so different from the AI. Our brain is not something magical that works in mysterious un-replicable ways. Much of AI field has been in fact trying to replicate how we learn, and how our brain works.

visarga 4y ago

When verification is possible, the interpolation machine starts to extrapolate. There is a process that generates ideas/solutions, and a process that tests them. An artist and a critic, a scientist and a lab. Together they form the experimentation loop. Let's take the game of Go for example. Testing who won a game is trivial. AlphaGo managed to beat humans in a few days of self-training. In other words, those edges you speak about can be pushed with massive search and verification. There is no reason we can't do massive search + verification for math and code. This is a good way to create training data where it doesn't exist in sufficient quantity. Other things can be simulated with expensive computation, and then "distilled" into fast neural networks. Then we apply the neural net to fast-search solutions. In the end we need to verify some of them (thinking of weather simulations, new materials, new drugs, ...) Also reminded of the recent AlphaTensor who leveraged massive learning from verification to beat Strassen's algorithm who was state of the art for 50 years. The is no reason neural nets should remain purely interpolative if we can manufacture good training data by running computation or experiments. Ideas are cheap, verification matters. Generative outputs are worthless without verification.

tbalsam 4y ago

Er....this feels like every main criticism going back to even the expert systems pre-90's. Could you please provide some support for your argument? This was repeated a lot in the early days of the modern wave (2012-2016) but was pretty thoroughly debunked as we've explored generalization and how these models disentangle intrinsic concepts and compute with them. Heck, even modern transformers are restricted memory Turing complete and use that to their advantage. Also, I do not mean to be rude, but frankly saying "colonize the parts of the problem space where AI doesn't have the imagination to go" is frankly rather terrible as it leans on the imagination argument of AI. At this point, being adaptable to co-integrate will be good, otherwise I could see people following that stuck inside of some kind of Sisyphean pseuso-Luddite escapist nightmare. Source for opinions: have been involved in ML in some form for most of the modern wave, and am appropriately (quite) skeptical about the AI takeover/revolution/eventual singularity belief/etc.

MetaWhirledPeas 4y ago

It can write an original poem using unique prompts I give it. Are you suggesting a similar poem exists somewhere else? I think not. It cobbles together examples, sure, but that's exactly what we do too. Everything "original" we make is riddled with subconscious outside influences.

klabb3 4y ago

Like other responders, I think your take is oversimplifying. It’s hard to both classify what imagination is, how to judge whether AI has it, and we also have humans tuning these models, which may or may not contribute to its blandness. That said, I think it’s still fair to say that most interpolation type of work is looking very threatened by AI as it is today, so the converse (don’t go into those fields) is probably sound advice in light of current developments. As for the rest, I suspect we’ll be forced to chisel off piece by piece from our zeitgeist of “imagination”. Some pieces will fall off quickly, as AIs replace them, and some will take longer, perhaps a lot longer. I do find it interesting that computers ended up killing it in unpredictable domains such as style transfer and NLP, while being mediocre-at-best in eg humor. It may mean that we have over- and underestimated aspects of what traits are unique and sophisticated.

willbudd 4y ago

> It can't expand the edges, it can't come up with anything truly original. You're dead wrong about that. Remember Deepmind's original claim to fame? Alphago? It came up with tactics and strategies that top professional go players considered to be novel to the point of being revolutionary. If you look at the way games are played by human professionals in the years before and after Alphago came on to the stage, you'll find they're different to an extent greater than any other upheaval in the many centuries-long history of the game. AI is a much more diverse field than you seem to give it credit for. Paint with overly broad strokes like that, and you're bound to go astray.