5 ms·
Regarding tokenizer- You are right in the context of python (but other language are more like pure "English" No?) I'm talking about the "coding grammar" (like
by transformi 3y ago
Regarding tokenizer- You are right in the context of python (but other language are more like pure "English" No?)
I'm talking about the "coding grammar" (like for python/CPP..), in order that when he write the code- it should gain benefit from using some syntax checking
- sp332 3y agoYeah, the LLM outputs a distribution of likely next tokens. It is up to the decoder to select one, and it can use a grammar to enforce certain rules on the output. https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio https://github.com/Hellisotherpeople/Constrained-Text-Genera... or https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github.com/ggerganov/llama.cpp/blob/master/grammars/... for example.