3 ms·
Node.js Open Source LLM Tokenizer
- jakecyr 3y agoThe first (that I found) tokenizer for open source LLMs. It retrieves config files from HuggingFace and can encode / decode text and tokens. Was created in an hour and can definitely use some work. Would love contributions and feedback!