4 ms·
> What they've proven is that you can build functioning systems that perform intelligent tasks purely at the level of syntax. But there is no (or very little) u
by TeMPOraL 3y ago
> What they've proven is that you can build functioning systems that perform intelligent tasks purely at the level of syntax. But there is no (or very little) understanding of semantics.
That's not how LLMs work though, and I'm increasingly convinced that "syntax" and "semantics" are turning into annoyingly useless ideas people forget are descriptive, in the same way grammar books and dictionaries are descriptive.
My model of LLMs is that, in training, they're positioning inputs in an absurdly high-dimensional[0] latent space. That space is big enough to encode anything you'd consider "syntax" and "semantics" on some subset of dimensions. As a result, the model sidesteps the issue of "source of meaning" - there is no meaning but that formed through associations (proximity). This is pretty much how we do it, too - when you think of "chair", there is no token for platonic ideal of a chair in your mind. The word "chair" has meanings and connotations defined via other words, which themselves are defined via other words, ad infinitum, with the only grounding being associations to sensory inputs.
--
[0] - On the order of 100 000 dimensions for GPT-4, perhaps more now for GPT-4V / GPT-4-Turbo.
- somewhereoutth 3y ago> big enough Not even slightly. See Cantor's diagonal argument. See also Plato's cave for syntax vs semantics.
- TeMPOraL 3y ago> Not even slightly. See Cantor's diagonal argument. Cantor's diagonal argument, if important, can just become one of the hundred thousand dimensions of the latent space. "Coordinate #12321 - degree of similarity to diagonal argument". > See also Plato's cave for syntax vs semantics. I don't see it. All there ever is are projections. There are just more dimensions in the mind space than in the physical space.