9 ms·

Fast base64 encoding and decoding

210 points by joeyespo 9y ago

dragontamer 9y ago
https://arxiv.org/abs/1704.00605 https://arxiv.org/abs/1704.00605 In short, they spend roughly (EDIT: Herp-derp) 1/5th a cycle per byte of base64 through the use of AVX2 instructions to encode / decode base64. So one-cycle every 5 bytes. This includes robust error checking (allegedly) --------- Because SIMD-based execution was so successful in this example, I do wonder if a GPGPU implementation would be worthwhile. If some very big data were Base64-encoded / decoded (like MBs worth), then would it be worthwhile to spend a costly PCIe Transaction to transfer the data to the GPU and back? Something like an email-attachment is Base64 encoded for example. I'd expect most web programs to be only a few kilobytes (at most) of base64 data however. A few hundred bytes for Cookies and the like. So the typical web case of base64 doesn't seem "big" enough to warrant the use of a GPU, so the AVX2 methodology would be ideal.
- jandrese 9y ago
  I suspect you would bottleneck on the PCIe bus before you saw improvement. GPUs usually aren't great at problems where you have a huge stream of data to work with and not a huge amount of processing needed per byte. If your data is already in GPU memory for some reason then yeah, it's going to blast through the data at an insane rate, but getting it there in the first place is the problem.
  - dragontamer 9y ago
    > I suspect you would bottleneck on the PCIe bus before you saw improvement. Heh, you're right. Its not even close. PCIe x16 is slightly less than 16GB/s. And that's a 1-way transfer, the 2nd transfer back effectively halves the speed. At 4GHz, this Base64 encoding / decoding scheme is doing 20GB/s of encoding (round numbers for simplicity). So literally, it is slower to transfer the data to the GPU than to use those AVX2 instructions. Heck, its slower on the one-way trip to the GPU, before it even comes back (and before the GPU even touches the data!)
dchest 9y ago
Base64 is commonly used in cryptography to exchange keys. Note that if you're encoding or decoding secret data, such as keys, you most likely want a constant-time encoder (example: https://boringssl.googlesource.com/boringssl/+/master/crypto/base64/base64.c https://boringssl.googlesource.com/boringssl/+/master/crypto...), not the fastest one.
- sd8dgf8ds8g8dsg 9y ago
  You seem to be confusing encoding (base64 for example), and encryption (AES for example). Where a side channel attack studying the decryption speed/power use can reveal information, the encoding of said encrypted data can not as far as I ever heard about. (It will still be encrypted after base64 decoding has taken place). Please correct me if I'm wrong. EDIT: I see you updated your original message with different wording, please ignore this rant.
  - CiPHPerCoder 9y ago
    Side-channel attacks against base64 encoding have yet to be proven, but constant-time implementations are available just in case. My contribution is here: https://github.com/paragonie/constant_time_encoding https://github.com/paragonie/constant_time_encoding
    - nickpsecurity 9y ago
      I'll add a high-assurance implementation from Galois to that which is probably not constant time. Their blog and Github has quite a few useful tools. https://galois.com/blog/2013/09/high-assurance-base64/ https://galois.com/blog/2013/09/high-assurance-base64/ Also, anyone wanting constant time implementation might just run a verified implementation through something like FaCT or Jasmin: https://cseweb.ucsd.edu/~dstefan/pubs/cauligi:2017:fact.pdf https://cseweb.ucsd.edu/~dstefan/pubs/cauligi:2017:fact.pdf https://acmccs.github.io/papers/p1807-almeidaA.pdf https://acmccs.github.io/papers/p1807-almeidaA.pdf
  - dchest 9y ago
    You seem to be confusing encoding (base64 for example), and encryption (AES for example). I don't. Where a side channel attack studying the decryption speed/power use can reveal information, the encoding of said encrypted data can not as far as I ever heard about. (It will still be encrypted after base64 decoding has taken place). Encoding/decoding a value to/from base64 depends on the value, and many implementations use a lookup table or branches. While the table is small, so it fits in CPU cache thus making timing differences harder to detect, it's important for code dealing with secret values to avoid any timing differences by using simple branchless instructions that we know are constant-time on CPUs.
tzahola 9y ago
I hate base64. If you have to embed binary data within human-readable text, then you’re either A: doing something horribly wrong B: or forced to use a monkey-patched legacy standard Otherwise just use hexadecimal digits. At least it will be somewhat readable. I know, I know... Base64 encodes 3 bytes on 4 characters, while hex encodes 2 bytes on 4. Big fucking deal. It will almost surely be gzipped down the line, and if you’re storing amounts of data where this would count, then see point “A” above. I hope one day we will laugh at the mere idea of base64 encoding like how we do with EBCDIC or Windows-12XX codepages.
- vvanders 9y ago
  Not everything goes over the web and is gzipped. I've been working on a project lately with 1200bps radio links that are making REST calls on the other end and 2 vs 3 bytes makes a massive difference.
  - tzahola 9y ago
    And why don’t you just send it in binary? REST doesn’t mean you have to use JSON for the HTTP body.
    - vvanders 9y ago
      Yeah, that would be ideal. Some of the protocols we talk over(APRS) are ASCII only(which is a bit of point B from above). Transmissions are also signed at the radio so that they can be authenticated down the line which means that you can't easily change the payload envelope once it leaves the radio. Our main goal is adoption and talking base64 means that lots of APIs that would take an intermediate server to do the translation aren't needed. This lowers to barrier to use and means we have a larger surface area of use cases.
    - sd8dgf8ds8g8dsg 9y ago
      It sounds like her use case is badly suited for raw binary transmission due to the high possibility of data loss, much like the internet used to be when base64 was originally introduced.
cperciva 9y ago
Note that this is considerably faster than the amount of memory bandwidth available per core.
- dragontamer 9y ago
  No its not, at least if we're talking L2 cache or faster. Intel processors support two loads + one write per core. That can be 2x256-bit reads + 1x256-bit write. Each clock, every clock. At least to L1 and L2 (the only levels of cache which have that level of bandwidth). AMD Ryzen supports 2x128-bit reads OR 1x128-bit write (1-read+1write is fine). ---------- The kicker is that you must use AVX2 registers to achieve this bandwidth. If you're loading / storing 64-bit registers like RAX, you can't get there. A raw "copy" between locations (assuming they're entirely in L2 memory or faster) can be done by Skylake at 32b per cycle. The speed of this Base64 encoder however is "just" at 5bytes per cycle. I'm curious how their code would perform on AMD's Ryzen platform, because of the lower-bandwidth of Ryzen. Ryzen's AVX is also split into 2x128-bit ports, instead of Intel's 2x256-bit ports. I'd expect AMD's Ryzen to do worse, but maybe there are other bottlenecks in the code????
  - cperciva 9y ago
    I said memory bandwidth, not cache bandwidth.
    - dragontamer 9y ago
      In which case, you may be surprised how fast RAM can be today. A stick of 3200MHz DDR4 RAM has roughly 25.6 GB/s of bandwidth. You'll need to slightly overclock the memory controller to get there, but it wouldn't be insane. There are 4600MHz sticks of RAM btw (https://www.tweaktown.com/reviews/8432/corsair-vengeance-lpx-ddr4-4600-16gb-memory-kit-review/index.html https://www.tweaktown.com/reviews/8432/corsair-vengeance-lpx...), so 3200MHz+ is kind of "typical" now, despite requiring the overclock. The cheapest systems are dual channel, giving 51.2 GB/s of bandwidth to main-memory. Some higher-end workstations are quad-channel for 102.4 GB/s of bandwidth. Your total RAM bandwidth per say... 4GHz clock is ~12.5 bytes/clock on dual-channel and ~25 bytes/clock on quad-channel. Divide it further into 4-cores (on dual-channel case) and 8-cores (on quad-channel processors) we're looking at ~3-bytes main-memory bandwidth per clock per core. A lot slower than L2 cache for sure, but not nearly as bad as it seems. So you're right, ~3 bytes per clock is still slower than the base64 implementation (which is 5-bytes per clock), but the bottleneck isn't nearly that bad. Especially if "other cores" aren't hitting the memory controller so hard.
top_post 9y ago
https://github.com/monobeard/libasmb64 https://github.com/monobeard/libasmb64
- Twirrim 9y ago
  I'm not sure what your point is here? I mean, we could all go and dig up links to base64 encoders and decoders, but the article you're commenting on is specifically about how fast they've been able to get using vector instructions on modern processors.
  - faragon 9y ago
    They provide an optimized non-SIMD generic implementation. The version I posted is similar, speed-wise, and it supports in-place base64 encoding/decoding (i.e. allowing aliasing, supporting the case of using one buffer for both input and output).
pmarcelll 9y ago
Last fall I tried to figure out why Rust is slower than D in a base64 benchmark (https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks). It turned out that decoding in Rust is really fast but encoding is a bit slow. I looked at the assembly output of both programs and I could more or less follow the steps in the Rust version, but the D version was only about 5 SIMD instructions (without a single bit shift). I also checked the source code, the Rust version is fairly long, because a part of the hot loop is manually unrolled, but the D version looked like the simple, naive implementation in C. It might be a result of newer LLVM version (LDC was already using LLVM 5 and rustc is still on LLVM 4) or rustc still might have issues with optimizer settings (both binaries were compiled for Intel Haswell).
- jnordwick 9y ago
  Do you have LLVM IR output? There are definitely places where rust's hinting isn't optimal, and that might not be too difficult to fix.
  - pmarcelll 9y ago
    I didn't look at the LLVM IR (since I'm not really familiar with it), but I might investigate it a bit further in the near future.
    - jnordwick 9y ago
      I would encourage it. LLVM IR is fairly easy to understand. One of my first forays into rustc was looking at the IR for iterators to see why one version was show (turned out to be that the bounds on the index variable couldn't proven tight enough to eliminate the bounds check, iirc). Try writing the same simple code as D and compare the two IR representations. And no matter how much I slag on rust being difficult, the community is really good about helping and teaching.
floatboth 9y ago
I made Go bindings for that library https://github.com/myfreeweb/go-base64-simd https://github.com/myfreeweb/go-base64-simd … …to use in a mail indexer project… …only to discover that half of my mail was not being decoded, because that decoder does not skip whitespace.
- fishywang 9y ago
  that sounds like 90% of base64 encoded mails not decoded, not half.
- thorel 9y ago
  To be fair, the original paper does mention the problem of whitespaces (see Appendix B on page 21). It seems that the recommended solution is to use this fast space remover: https://github.com/lemire/despacer https://github.com/lemire/despacer
- jorangreef 9y ago
  Did you find any bugs in the library when implementing your binding? Daniel Lemire has done brilliant work. His https://github.com/lemire/despacer https://github.com/lemire/despacer looks great. I wonder though if their Base64 SIMD algorithms could be tweaked to do the decode in a single pass? Like you, also email related, I have written a native Base64 addon for Node in C optimized for decoding MIME wrapped Base64 in a single pass (https://github.com/ronomon/base64 https://github.com/ronomon/base64). It's not yet using SIMD instructions but it's 1.5-2x as fast as Node's routines with top speeds of 1398.18 MB/s decoding and 1048.58 MB/s encoding on a single core (Intel Xeon E5-1620 v3 @ 3.50GHz). Here is an explanation of the difference: https://github.com/nodejs/node/issues/12114 https://github.com/nodejs/node/issues/12114 The Ronomon Base64 addon also ships with a fuzz test (https://github.com/ronomon/base64/blob/master/test.js https://github.com/ronomon/base64/blob/master/test.js) which you are welcome to try out on your Go binding. It picked up an interesting bug in Node's decode routine where some white space was actually decoded as data (https://github.com/nodejs/node/issues/11987 https://github.com/nodejs/node/issues/11987).
nwmcsween 9y ago
The benchmarks shows the 'generic' aka SWAR version w or w/o openmp is the best for almost all purposes. The only thing I would change is the func sig to size_t base64_enc(const char * src, char * dst, size_t len) to ret written size.
skerit 9y ago
It's a shame nobody is really using yEnc instead of base64. yEnc only has a overhead cost of about 2%, compared to 33% for base64, and would be perfect for use on webpages.
- ensiferum 9y ago
  Usenet uses yEnc
- NL807 9y ago
  yEnc has a fair share of problems related to reliable encoding and decoding for some character sets (UTF-8 for example). Normally this is solved with the MIME types, but yEnc is not officially registered. Nor does it have an official RFC standard. base64 on the other hand "just works".
- fpoling 9y ago
  To embed binary data into web-pages one needs encoding that is friendly to the text encoding that the page uses and uses characters that only the source syntax allows. Any of this rules out yEnc. Besides, web pages where one embeds the binary data should be transmitted compressed and gzip removes the overhead of base64 down to few percent.
- HugoDaniel 9y ago
  base64 is supported in browsers with atob() and btoa(), no need for an external lib
- Nyan 9y ago
  yEnc is very much designed specifically for Usenet though. It gets its low overhead largely from using almost all characters which aren't special in NNTP (and also adapts allowed characters depending on NNTP context), which isn't appropriate for many cases where base64 is used. There's encodings which may be more appropriate, which use more than 64 codepoints, such as base85 (https://en.wikipedia.org/wiki/Ascii85 https://en.wikipedia.org/wiki/Ascii85), assuming that these are allowed. As an aside, I've implemented a fast SIMD accelerated yEnc encoder/decoder here (https://github.com/animetosho/node-yencode https://github.com/animetosho/node-yencode), for anyone interested.
faragon 9y ago
Another fast implementation (plain C99, without using SIMD), supporting buffer aliasing (i.e. allowing in-place encoding/decoding, check senc_b64() and sdec_b64() functions): https://github.com/faragon/libsrt/blob/master/src/saux/senc.c https://github.com/faragon/libsrt/blob/master/src/saux/senc....
powturbo 9y ago
In my benchmarks only the AVX2 is faster than the portable scalar "turbobase64" https://github.com/powturbo/TurboBase64 https://github.com/powturbo/TurboBase64 SIMD does not always mean faster.

210 points by joeyespo 9y ago

https://arxiv.org/abs/1704.00605 https://arxiv.org/abs/1704.00605 In short, they spend roughly (EDIT: Herp-derp) 1/5th a cycle per byte of base64 through the use of AVX2 instructions to encode / decode base64. So one-cycle every 5 bytes. This includes robust error checking (allegedly) --------- Because SIMD-based execution was so successful in this example, I do wonder if a GPGPU implementation would be worthwhile. If some very big data were Base64-encoded / decoded (like MBs worth), then would it be worthwhile to spend a costly PCIe Transaction to transfer the data to the GPU and back? Something like an email-attachment is Base64 encoded for example. I'd expect most web programs to be only a few kilobytes (at most) of base64 data however. A few hundred bytes for Cookies and the like. So the typical web case of base64 doesn't seem "big" enough to warrant the use of a GPU, so the AVX2 methodology would be ideal.

jandrese 9y ago
I suspect you would bottleneck on the PCIe bus before you saw improvement. GPUs usually aren't great at problems where you have a huge stream of data to work with and not a huge amount of processing needed per byte. If your data is already in GPU memory for some reason then yeah, it's going to blast through the data at an insane rate, but getting it there in the first place is the problem.
- dragontamer 9y ago
  > I suspect you would bottleneck on the PCIe bus before you saw improvement. Heh, you're right. Its not even close. PCIe x16 is slightly less than 16GB/s. And that's a 1-way transfer, the 2nd transfer back effectively halves the speed. At 4GHz, this Base64 encoding / decoding scheme is doing 20GB/s of encoding (round numbers for simplicity). So literally, it is slower to transfer the data to the GPU than to use those AVX2 instructions. Heck, its slower on the one-way trip to the GPU, before it even comes back (and before the GPU even touches the data!)

dchest 9y ago

Base64 is commonly used in cryptography to exchange keys. Note that if you're encoding or decoding secret data, such as keys, you most likely want a constant-time encoder (example: https://boringssl.googlesource.com/boringssl/+/master/crypto/base64/base64.c https://boringssl.googlesource.com/boringssl/+/master/crypto...), not the fastest one.

sd8dgf8ds8g8dsg 9y ago
You seem to be confusing encoding (base64 for example), and encryption (AES for example). Where a side channel attack studying the decryption speed/power use can reveal information, the encoding of said encrypted data can not as far as I ever heard about. (It will still be encrypted after base64 decoding has taken place). Please correct me if I'm wrong. EDIT: I see you updated your original message with different wording, please ignore this rant.
- CiPHPerCoder 9y ago
  Side-channel attacks against base64 encoding have yet to be proven, but constant-time implementations are available just in case. My contribution is here: https://github.com/paragonie/constant_time_encoding https://github.com/paragonie/constant_time_encoding
  - nickpsecurity 9y ago
    I'll add a high-assurance implementation from Galois to that which is probably not constant time. Their blog and Github has quite a few useful tools. https://galois.com/blog/2013/09/high-assurance-base64/ https://galois.com/blog/2013/09/high-assurance-base64/ Also, anyone wanting constant time implementation might just run a verified implementation through something like FaCT or Jasmin: https://cseweb.ucsd.edu/~dstefan/pubs/cauligi:2017:fact.pdf https://cseweb.ucsd.edu/~dstefan/pubs/cauligi:2017:fact.pdf https://acmccs.github.io/papers/p1807-almeidaA.pdf https://acmccs.github.io/papers/p1807-almeidaA.pdf
- dchest 9y ago
  You seem to be confusing encoding (base64 for example), and encryption (AES for example). I don't. Where a side channel attack studying the decryption speed/power use can reveal information, the encoding of said encrypted data can not as far as I ever heard about. (It will still be encrypted after base64 decoding has taken place). Encoding/decoding a value to/from base64 depends on the value, and many implementations use a lookup table or branches. While the table is small, so it fits in CPU cache thus making timing differences harder to detect, it's important for code dealing with secret values to avoid any timing differences by using simple branchless instructions that we know are constant-time on CPUs.

tzahola 9y ago

I hate base64. If you have to embed binary data within human-readable text, then you’re either A: doing something horribly wrong B: or forced to use a monkey-patched legacy standard Otherwise just use hexadecimal digits. At least it will be somewhat readable. I know, I know... Base64 encodes 3 bytes on 4 characters, while hex encodes 2 bytes on 4. Big fucking deal. It will almost surely be gzipped down the line, and if you’re storing amounts of data where this would count, then see point “A” above. I hope one day we will laugh at the mere idea of base64 encoding like how we do with EBCDIC or Windows-12XX codepages.

vvanders 9y ago
Not everything goes over the web and is gzipped. I've been working on a project lately with 1200bps radio links that are making REST calls on the other end and 2 vs 3 bytes makes a massive difference.
- tzahola 9y ago
  And why don’t you just send it in binary? REST doesn’t mean you have to use JSON for the HTTP body.
  - vvanders 9y ago
    Yeah, that would be ideal. Some of the protocols we talk over(APRS) are ASCII only(which is a bit of point B from above). Transmissions are also signed at the radio so that they can be authenticated down the line which means that you can't easily change the payload envelope once it leaves the radio. Our main goal is adoption and talking base64 means that lots of APIs that would take an intermediate server to do the translation aren't needed. This lowers to barrier to use and means we have a larger surface area of use cases.
  - sd8dgf8ds8g8dsg 9y ago
    It sounds like her use case is badly suited for raw binary transmission due to the high possibility of data loss, much like the internet used to be when base64 was originally introduced.

cperciva 9y ago

Note that this is considerably faster than the amount of memory bandwidth available per core.

dragontamer 9y ago
No its not, at least if we're talking L2 cache or faster. Intel processors support two loads + one write per core. That can be 2x256-bit reads + 1x256-bit write. Each clock, every clock. At least to L1 and L2 (the only levels of cache which have that level of bandwidth). AMD Ryzen supports 2x128-bit reads OR 1x128-bit write (1-read+1write is fine). ---------- The kicker is that you must use AVX2 registers to achieve this bandwidth. If you're loading / storing 64-bit registers like RAX, you can't get there. A raw "copy" between locations (assuming they're entirely in L2 memory or faster) can be done by Skylake at 32b per cycle. The speed of this Base64 encoder however is "just" at 5bytes per cycle. I'm curious how their code would perform on AMD's Ryzen platform, because of the lower-bandwidth of Ryzen. Ryzen's AVX is also split into 2x128-bit ports, instead of Intel's 2x256-bit ports. I'd expect AMD's Ryzen to do worse, but maybe there are other bottlenecks in the code????
- cperciva 9y ago
  I said memory bandwidth, not cache bandwidth.
  - dragontamer 9y ago
    In which case, you may be surprised how fast RAM can be today. A stick of 3200MHz DDR4 RAM has roughly 25.6 GB/s of bandwidth. You'll need to slightly overclock the memory controller to get there, but it wouldn't be insane. There are 4600MHz sticks of RAM btw (https://www.tweaktown.com/reviews/8432/corsair-vengeance-lpx-ddr4-4600-16gb-memory-kit-review/index.html https://www.tweaktown.com/reviews/8432/corsair-vengeance-lpx...), so 3200MHz+ is kind of "typical" now, despite requiring the overclock. The cheapest systems are dual channel, giving 51.2 GB/s of bandwidth to main-memory. Some higher-end workstations are quad-channel for 102.4 GB/s of bandwidth. Your total RAM bandwidth per say... 4GHz clock is ~12.5 bytes/clock on dual-channel and ~25 bytes/clock on quad-channel. Divide it further into 4-cores (on dual-channel case) and 8-cores (on quad-channel processors) we're looking at ~3-bytes main-memory bandwidth per clock per core. A lot slower than L2 cache for sure, but not nearly as bad as it seems. So you're right, ~3 bytes per clock is still slower than the base64 implementation (which is 5-bytes per clock), but the bottleneck isn't nearly that bad. Especially if "other cores" aren't hitting the memory controller so hard.

top_post 9y ago

https://github.com/monobeard/libasmb64 https://github.com/monobeard/libasmb64

Twirrim 9y ago
I'm not sure what your point is here? I mean, we could all go and dig up links to base64 encoders and decoders, but the article you're commenting on is specifically about how fast they've been able to get using vector instructions on modern processors.
- faragon 9y ago
  They provide an optimized non-SIMD generic implementation. The version I posted is similar, speed-wise, and it supports in-place base64 encoding/decoding (i.e. allowing aliasing, supporting the case of using one buffer for both input and output).

pmarcelll 9y ago

Last fall I tried to figure out why Rust is slower than D in a base64 benchmark (https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks). It turned out that decoding in Rust is really fast but encoding is a bit slow. I looked at the assembly output of both programs and I could more or less follow the steps in the Rust version, but the D version was only about 5 SIMD instructions (without a single bit shift). I also checked the source code, the Rust version is fairly long, because a part of the hot loop is manually unrolled, but the D version looked like the simple, naive implementation in C. It might be a result of newer LLVM version (LDC was already using LLVM 5 and rustc is still on LLVM 4) or rustc still might have issues with optimizer settings (both binaries were compiled for Intel Haswell).

jnordwick 9y ago
Do you have LLVM IR output? There are definitely places where rust's hinting isn't optimal, and that might not be too difficult to fix.
- pmarcelll 9y ago
  I didn't look at the LLVM IR (since I'm not really familiar with it), but I might investigate it a bit further in the near future.
  - jnordwick 9y ago
    I would encourage it. LLVM IR is fairly easy to understand. One of my first forays into rustc was looking at the IR for iterators to see why one version was show (turned out to be that the bounds on the index variable couldn't proven tight enough to eliminate the bounds check, iirc). Try writing the same simple code as D and compare the two IR representations. And no matter how much I slag on rust being difficult, the community is really good about helping and teaching.

floatboth 9y ago

I made Go bindings for that library https://github.com/myfreeweb/go-base64-simd https://github.com/myfreeweb/go-base64-simd … …to use in a mail indexer project… …only to discover that half of my mail was not being decoded, because that decoder does not skip whitespace.

fishywang 9y ago
that sounds like 90% of base64 encoded mails not decoded, not half.
thorel 9y ago
To be fair, the original paper does mention the problem of whitespaces (see Appendix B on page 21). It seems that the recommended solution is to use this fast space remover: https://github.com/lemire/despacer https://github.com/lemire/despacer
jorangreef 9y ago
Did you find any bugs in the library when implementing your binding? Daniel Lemire has done brilliant work. His https://github.com/lemire/despacer https://github.com/lemire/despacer looks great. I wonder though if their Base64 SIMD algorithms could be tweaked to do the decode in a single pass? Like you, also email related, I have written a native Base64 addon for Node in C optimized for decoding MIME wrapped Base64 in a single pass (https://github.com/ronomon/base64 https://github.com/ronomon/base64). It's not yet using SIMD instructions but it's 1.5-2x as fast as Node's routines with top speeds of 1398.18 MB/s decoding and 1048.58 MB/s encoding on a single core (Intel Xeon E5-1620 v3 @ 3.50GHz). Here is an explanation of the difference: https://github.com/nodejs/node/issues/12114 https://github.com/nodejs/node/issues/12114 The Ronomon Base64 addon also ships with a fuzz test (https://github.com/ronomon/base64/blob/master/test.js https://github.com/ronomon/base64/blob/master/test.js) which you are welcome to try out on your Go binding. It picked up an interesting bug in Node's decode routine where some white space was actually decoded as data (https://github.com/nodejs/node/issues/11987 https://github.com/nodejs/node/issues/11987).

nwmcsween 9y ago

The benchmarks shows the 'generic' aka SWAR version w or w/o openmp is the best for almost all purposes. The only thing I would change is the func sig to size_t base64_enc(const char * src, char * dst, size_t len) to ret written size.

skerit 9y ago

It's a shame nobody is really using yEnc instead of base64. yEnc only has a overhead cost of about 2%, compared to 33% for base64, and would be perfect for use on webpages.

ensiferum 9y ago
Usenet uses yEnc
NL807 9y ago
yEnc has a fair share of problems related to reliable encoding and decoding for some character sets (UTF-8 for example). Normally this is solved with the MIME types, but yEnc is not officially registered. Nor does it have an official RFC standard. base64 on the other hand "just works".
fpoling 9y ago
To embed binary data into web-pages one needs encoding that is friendly to the text encoding that the page uses and uses characters that only the source syntax allows. Any of this rules out yEnc. Besides, web pages where one embeds the binary data should be transmitted compressed and gzip removes the overhead of base64 down to few percent.
HugoDaniel 9y ago
base64 is supported in browsers with atob() and btoa(), no need for an external lib
Nyan 9y ago
yEnc is very much designed specifically for Usenet though. It gets its low overhead largely from using almost all characters which aren't special in NNTP (and also adapts allowed characters depending on NNTP context), which isn't appropriate for many cases where base64 is used. There's encodings which may be more appropriate, which use more than 64 codepoints, such as base85 (https://en.wikipedia.org/wiki/Ascii85 https://en.wikipedia.org/wiki/Ascii85), assuming that these are allowed. As an aside, I've implemented a fast SIMD accelerated yEnc encoder/decoder here (https://github.com/animetosho/node-yencode https://github.com/animetosho/node-yencode), for anyone interested.

faragon 9y ago

Another fast implementation (plain C99, without using SIMD), supporting buffer aliasing (i.e. allowing in-place encoding/decoding, check senc_b64() and sdec_b64() functions): https://github.com/faragon/libsrt/blob/master/src/saux/senc.c https://github.com/faragon/libsrt/blob/master/src/saux/senc....

powturbo 9y ago

In my benchmarks only the AVX2 is faster than the portable scalar "turbobase64" https://github.com/powturbo/TurboBase64 https://github.com/powturbo/TurboBase64 SIMD does not always mean faster.