3 ms·
How does this compare to the BPE crate [1]? Its main selling point is support for incrementally re-tokenising text, but it's also faster than tiktoken. [1] htt
by p0 1y ago
How does this compare to the BPE crate [1]? Its main selling point is support for incrementally re-tokenising text, but it's also faster than tiktoken.
[1] https://crates.io/crates/bpe https://crates.io/crates/bpe
- matthewolfe 1y agoI'm working on incremental re-tokenizing next. Then I'll run some benchmarks against this crate too.