tokfan

4 ms·

1. 1.▲
   Tokenizers(huggingface.co)

   2 points by tokfan 1y ago|0 comments
2. 2.▲
   LLM Tokenization Demo(github.com)

   2 points by tokfan 1y ago|0 comments
3. 3.▲
   by tokfan 1y ago

   Same, same. However I have more of Markdown file where I have different headings for different projects and then use the "check" function of Markdown.
4. 4.▲
   Sampling – Model Context Protocol(modelcontextprotocol.io)

   3 points by tokfan 1y ago|0 comments
5. 5.▲
   What Is LLM Tokenization and Why Is It Important?(medium.com)

   1 points by tokfan 1y ago|0 comments
6. 6.▲
   All you need to know about Tokenization in LLMs(medium.com)

   2 points by tokfan 1y ago|0 comments
7. 7.▲
   by tokfan 1y ago

   Seems to be paywalled? I get a weird thing when scrolling down.
8. 8.▲
   Tokenization in Large Language Models(seantrott.substack.com)

   2 points by tokfan 1y ago|0 comments
9. 9.▲
   by tokfan 1y ago

   Haha, this is hilarious. Its interesting to see how even with "Phd level skills" LLMs often fail to answer such basic questions