2 ms·
Here's the Observable notebook I built to explore how the tokenizers work: https://observablehq.com/@simonw/gpt-tokenizer https://observablehq.com/@simonw/gpt-t
by simonw 3y ago
Here's the Observable notebook I built to explore how the tokenizers work: https://observablehq.com/@simonw/gpt-tokenizer https://observablehq.com/@simonw/gpt-tokenizer