3 ms·

Great repo, but there was this for openAI which was bit hard to find: https://github.com/cogentapps/chat-with-gpt/blob/main/app/src/core/tokenizer/bpe.ts https:

Great repo, but there was this for openAI which was bit hard to find: https://github.com/cogentapps/chat-with-gpt/blob/main/app/src/core/tokenizer/bpe.ts https://github.com/cogentapps/chat-with-gpt/blob/main/app/sr.... Not completely sure, but I think it will likely work as it is for llama as both are BPE following same rules.

belladoreai 3y ago
I think it would take quite a bit of work for someone to grab that BPE implementation and make it work for LLaMA. Less work than rewriting the whole tokenizer from scratch, for sure, but a non trivial amount of work anyway.