3 ms·
The encoder is the neural-net that converts the input to the embedding vector. The decoder is the neural-net that converts that vector into output. What that em
by saynay 5y ago
The encoder is the neural-net that converts the input to the embedding vector. The decoder is the neural-net that converts that vector into output. What that embedding vector "means" is whatever the entire algorithm has learned it means.
For more simplified look at embeddings, I would look at Word2Vec (although, it doesn't involve transformers). It encodes single words, instead of entire phrases, and does so by looking at their relative position to other words while being trained.
Embeddings are just vectors, and so you can do math or compare them to other embeddings. The famous example is E(king) - E(man) + E(woman) = E(queen)
- mrfusion 5y agoSo you’re saying the encoding could be a neural net OR something like word2vec?
- saynay 5y agoSorry, didn't see this sooner. Word2Vec was just an example of an easier-to-understand embedding (it is actually frequently used as an input to these language models, actually). The process of encoding the input data stream to an embedding is done by a NN. The part that tripped me up when I was first trying to understand it is that the encoding is not like encoding elsewhere in compsci. It isn't trying to convert the input to a well-known encoding format or anything. The "encoder" is just the name of the neural-net that is converting the big, multi-dimensional input into a single-dimension, fixed-length vector. The "decoder" is just the neural-net that converts it to output. That fixed-length vector is entirely meaningless to anything other than the two neural-nets trained to generate/consume it.
- mrfusion 5y agoThanks. That’s helping the clear it up. So what’s the point of the encoding and decoding steps? Then the transformer is just another neural net in between that does some funny business with attention?
- saynay 5y agoThe transformer is the neural net in the encode / decode steps. For something like a language model, it takes an input phrase, encodes it to something that has "meaning" to the net, then decodes it into an output phrase. The "attention" mechanism learns how much attention to give to each part of the input, anywhere in it. In languages, the previous common mechanisms (recurrent networks), would only look backwards, with decreasing weight for things further away.