4 ms·
> All 7 books come to ~1.75M tokens How do you know? Each word is one token?
by siwatanejo 8mo ago
> All 7 books come to ~1.75M tokens
How do you know? Each word is one token?
- koakuma-chan 8mo agoYou can download the books and run them through a tokenizer. I did that half a year ago and got ~2M.