3 ms·
Very interesting! Thank you. I’ll try to explain something I’m thinking, it comes down to a type of agglutination. https://en.wikipedia.org/wiki/Agglutination
by alchemist1e9 3y ago
Very interesting! Thank you.
I’ll try to explain something I’m thinking, it comes down to a type of agglutination.
https://en.wikipedia.org/wiki/Agglutination https://en.wikipedia.org/wiki/Agglutination
The ASTs need to become sequences of tokens for LLMs to work well. Also the embedding space is related, this should all be specifically optimized for code, not based on general human language .
Of course the description of the code would be english. But my point is that a subspace of the encoding should be specifically designed for an agglutination based sequential expression of ASTs.
Not sure any of that makes sense to someone with more expertise in this space than me.
- kevinlu1248 3y agoThis is really interesting, will take a look. So is this basically English to code translation?
- alchemist1e9 3y agoYeah what I’m thinking about is that and how an LLM could use ASTs expressed in some type of compact encoding and with that code tied to english descriptions, perhaps it will learn really well the semantic cross space.
- kevinlu1248 3y agoLike stochastically generate into an intermediate representation that can be procedurally compiled to code right?
- alchemist1e9 3y agoYes