3 ms·
> Doesn't Language itself encode multimodal experiences Of course it does. We immediately encode pictures/words/everything into vectors anyway. In practice we
by danielmarkbruce 2y ago
> Doesn't Language itself encode multimodal experiences
Of course it does. We immediately encode pictures/words/everything into vectors anyway. In practice we don't have great text datasets to describe many things in enough detail, but there isn't any reason we couldn't.
- heyjamesknight 2y agoThere are absolutely reasons that we cannot capture the entirety—or even a proper image—of human cognition in semantic space. Cognition is not purely semantic. It is dynamic, embodied, socially distributed, culturally extended, and conscious. LLMs are great semantic heuristic machines. But they don't even have access to those other components.
- danielmarkbruce 2y agoThe LLM embeddings for a token cover much more than semantics. There is a reason a single token embedding dimension is so large. You are conflating the embedding layer in an LLM and an embedding model for semantic search.
- heyjamesknight 2y agoI don't think we're using the term semantic in the same way. I mean "relating to meaning in language."
- danielmarkbruce 2y agoThe embedding layer in an llm deals with much more than the meaning. It has to capture syntax, grammar, morphology, style and sentiment cues, phonetic and orthographic relationships and 500 other things that humans can't even reason about but exist in words combinations.
- heyjamesknight 2y agoI'll give you that. I was including those in "semantic space," but the distinction is fair. My original point still stands: the space you've described cannot capture a full image of human cognition.