2 ms·
Do the delimiters have to be single bytes? e.g. Japanese full stop (IDEOGRAPHIC FULL STOP) is 3 bytes in UTF-8.
by ks2048 10mo ago
Do the delimiters have to be single bytes? e.g. Japanese full stop (IDEOGRAPHIC FULL STOP) is 3 bytes in UTF-8.
- deleted 10mo ago[deleted]
- snyy 10mo agoNo, delimiters can be multiple bytes. They have to be passed as a pattern. // With multi-byte pattern let metaspace = "<japanese_full_stop>".as_bytes(); let chunks: Vec<&[u8]> = chunk(text).pattern(metaspace).prefix().collect();