4 ms·
It'd be interesting to benchmark this on texts from different languages. English text will be very friendly to the branch predictor because it's all 1-byte code
by andrewf 2y ago
It'd be interesting to benchmark this on texts from different languages. English text will be very friendly to the branch predictor because it's all 1-byte codepoints. I think a language dominated by either 2-byte or 3-byte codepoints would be as well, mixing codepoint lengths is what would trip things up.
- nitwit005 2y agoBeen a while, but I found it was best to assume all the text is ASCII, and fall back to a slower code path only if that turned out not to be the case. A lot of tools that create JSON, XML, and so forth will escape any non ASCII characters by default. Python's json package, for example.
- torstenvl 2y agoAgreed. I saw measurable speed increases when defaulting to ASCII (via macro) and only actually calling the decoder function if the value wasn't < 128. #define cdpt(s) ((s[0] < 128) ? s[0] : cdpt_func(s)) https://github.com/torstenvl/trex/blob/master/src/trex.c https://github.com/torstenvl/trex/blob/master/src/trex.c
- dzaima 2y agoAn all-ASCII fast-path is basically obligatory for anything that cares about performance. And then the "slow path" gets a guarantee that there will be varying-length data, and, as you'll want it to not actually be slow, you'll still need to avoid branching on the now-guaranteed-variable width.