4 ms·
This network shows how it is possible to represent meaning into a vector of 100-600 real numbers, first from images to vectors, then from vectors to text. Philo
by visarga 10y ago
This network shows how it is possible to represent meaning into a vector of 100-600 real numbers, first from images to vectors, then from vectors to text. Philosophers have always wondered about the nature of thought, but this representation model creates for the first time the ability to work with sensorially grounded abstract concepts in AI. It's not so mysterious after all. It is possible to merge multiple sensorial modalities in a common meaning space.
In order to get full AI we need to add behavior and embodiment to these meaning vectors. They need to be trained by reinforcement learning, to learn the behavior that maximizes rewards. Meaning vectors are just a small part of the final system, equivalent to our ability to see and speak. The most difficult part is that of learning behavior.
- gradys 10y agoWell put. I think it's really valuable to take inventory of human cognitive capabilities that we don't yet know how to implement in neural nets. We seem to have more or less solved perception. Given a high dimensional, "raw" input space, we know how to process it into a more usable, more abstract representation. Work on perception has also given us certain limited kinds of behavior. We can generate images from abstract representations by inverting our image recognition architectures. RNNs can do perception over sequences (e.g. of words), but can also generate language output or a sequence of control commands for robotics. One area in which I think human cognition is far ahead of neural network research is control flow. Maybe there's a better name for this. We seem to be able to encounter a new cognitive challenge and quickly design a mental program to solve it. We can attend to relevant sensory streams, various kinds of memory, and then design (sometimes novel) behavior to solve the problem. Work on attention and architectures with more sophisticated internal representations like stack-augmented RNNs are definitely moving in this direction, but it seems like we have much further to go on this front than in visual perception, for example.
- bbctol 10y agoI'd be much more hesitant to talk about "meaning" in this context. Are we working with abstract concepts? We're working with images and text, and creating connections between the two. It may be that this approach, ramped up in processing power and complexity, can completely mimic a human's response to images; we may also hit a wall where new techniques are needed to address what you'd call "meaning."
- visarga 10y agoYes, could be. Word vectors and other kinds of embeddings seem promising, a little too good to be true. There might be a glass ceiling we're not seeing yet.
- zardo 10y agoThinking about behavior as a sequence of actions and operators to combine actions... It may not be all that different than speech generation. Speech is a behavior after all.
- visarga 10y agoYes, and also other functions such as attention, memory reading and writing, comparing - they form an assembly language for the "neural turing machine". Doing all sorts of mental operations is behavior as well. So it can be learned by reinforcement.
- nl 10y agoSee the paper 'zero shot learning through cross modal transfer' which might be worth reading. It's one of the papers that kicked off this approach to image captioning, but is much more ambitious. Lots of things in it don't really quite work, but it shows where this work is going.