3 ms·

I made Go bindings for that library https://github.com/myfreeweb/go-base64-simd https://github.com/myfreeweb/go-base64-simd … …to use in a mail indexer project

I made Go bindings for that library https://github.com/myfreeweb/go-base64-simd https://github.com/myfreeweb/go-base64-simd … …to use in a mail indexer project… …only to discover that half of my mail was not being decoded, because that decoder does not skip whitespace.

fishywang 9y ago
that sounds like 90% of base64 encoded mails not decoded, not half.
thorel 9y ago
To be fair, the original paper does mention the problem of whitespaces (see Appendix B on page 21). It seems that the recommended solution is to use this fast space remover: https://github.com/lemire/despacer https://github.com/lemire/despacer
jorangreef 9y ago
Did you find any bugs in the library when implementing your binding? Daniel Lemire has done brilliant work. His https://github.com/lemire/despacer https://github.com/lemire/despacer looks great. I wonder though if their Base64 SIMD algorithms could be tweaked to do the decode in a single pass? Like you, also email related, I have written a native Base64 addon for Node in C optimized for decoding MIME wrapped Base64 in a single pass (https://github.com/ronomon/base64 https://github.com/ronomon/base64). It's not yet using SIMD instructions but it's 1.5-2x as fast as Node's routines with top speeds of 1398.18 MB/s decoding and 1048.58 MB/s encoding on a single core (Intel Xeon E5-1620 v3 @ 3.50GHz). Here is an explanation of the difference: https://github.com/nodejs/node/issues/12114 https://github.com/nodejs/node/issues/12114 The Ronomon Base64 addon also ships with a fuzz test (https://github.com/ronomon/base64/blob/master/test.js https://github.com/ronomon/base64/blob/master/test.js) which you are welcome to try out on your Go binding. It picked up an interesting bug in Node's decode routine where some white space was actually decoded as data (https://github.com/nodejs/node/issues/11987 https://github.com/nodejs/node/issues/11987).

fishywang 9y ago

that sounds like 90% of base64 encoded mails not decoded, not half.

thorel 9y ago

To be fair, the original paper does mention the problem of whitespaces (see Appendix B on page 21). It seems that the recommended solution is to use this fast space remover: https://github.com/lemire/despacer https://github.com/lemire/despacer

jorangreef 9y ago

Did you find any bugs in the library when implementing your binding? Daniel Lemire has done brilliant work. His https://github.com/lemire/despacer https://github.com/lemire/despacer looks great. I wonder though if their Base64 SIMD algorithms could be tweaked to do the decode in a single pass? Like you, also email related, I have written a native Base64 addon for Node in C optimized for decoding MIME wrapped Base64 in a single pass (https://github.com/ronomon/base64 https://github.com/ronomon/base64). It's not yet using SIMD instructions but it's 1.5-2x as fast as Node's routines with top speeds of 1398.18 MB/s decoding and 1048.58 MB/s encoding on a single core (Intel Xeon E5-1620 v3 @ 3.50GHz). Here is an explanation of the difference: https://github.com/nodejs/node/issues/12114 https://github.com/nodejs/node/issues/12114 The Ronomon Base64 addon also ships with a fuzz test (https://github.com/ronomon/base64/blob/master/test.js https://github.com/ronomon/base64/blob/master/test.js) which you are welcome to try out on your Go binding. It picked up an interesting bug in Node's decode routine where some white space was actually decoded as data (https://github.com/nodejs/node/issues/11987 https://github.com/nodejs/node/issues/11987).