3 ms·
> If you know of any good UTF-8 corpora for common use cases, I'd be happy to benchmark them. I think a reasonable approach is the first N bytes of wikipedia f
by BeeOnRope 7y ago
> If you know of any good UTF-8 corpora for common use cases, I'd be happy to benchmark them.
I think a reasonable approach is the first N bytes of wikipedia for various interesting languages which stress the UTF-8 continuum, e.g., English, French, maybe something in Cyrillic, Greek, Korean, Japanese, Mandarin.
Those are in html IIRC so you should have a good mix of ASCII plus UTF-8.
- zwegner 7y agoDefinitely a good idea. I had already downloaded a Mandarin Wikipedia page to look for test sequences, but hadn't benchmarked them. Getting some more pages there is a great place to start.