3 ms·
To build tokenizers that use hashed identifiers rather than identifiers as plain English? e.g, "NullPointerException" can be a single kanji. Current LLM proces
by numpad0 2y ago
To build tokenizers that use hashed identifiers rather than identifiers as plain English?
e.g, "NullPointerException" can be a single kanji. Current LLM processes it like "N, "ull", "P", "oint", er", "Excep", "tion". This lets them make up "PullDrawerException", which is only useful outside code.
That kind of creativity is not useful in code, in which identifiers are just friendly names for pointer addresses.
I guess real question is how much business sense such a solution would make. "S in $buzzword stands for security" kind of thing.
- nkmnz 2y agoWhy not train an LAM, a Large AST Model?