4 ms·
> I've seen zero reference to any technology that would abstract and wield concepts. ChatGPT (and I assume other LLMs as well) uses an embeddings layer. This
by wcoenen 4y ago
> I've seen zero reference to any technology that would abstract and wield concepts.
ChatGPT (and I assume other LLMs as well) uses an embeddings layer.
This is how it can translate concepts between domains. It is working with abstractions in a high dimensional semantic space.
https://openai.com/blog/new-and-improved-embedding-model https://openai.com/blog/new-and-improved-embedding-model
- toss1 4y agoThanks, that's good news, and looks like a step in the right direction. they say: >>Embeddings are numerical representations of concepts converted to number sequences, which make it easy for computers to understand the relationships between those concepts. Beyond that claim, there is nearly no information on how this works, and the examples give hints of not any kind of concept, but an additional abstraction layer of clustering words with similar meanings. It is not the abstraction of an understanding, but a more far-reaching thesaurus lookup. (edit) It's an astonishingly good map of words, and this makes it better. The clusters may sort of map resembling concepts, but the kinds of errors it makes show that it's not actually wielding concepts to think. That said, a journey of a thousand miles begins with a single step, and this looks like a good step.