5 ms·

Pretty nice. Always nice to see better optimized commonly required algorithms. If this is really faster than previous implementations (no reason to doubt, but d

Pretty nice. Always nice to see better optimized commonly required algorithms. If this is really faster than previous implementations (no reason to doubt, but don't have time to validate right now), this might save megawatts of power world wide. (Disclaimer: Would need to actually check with power measurement tools, but generally faster consumes less power.) A small nitpick: not sure whether those macros bought anything, I guess the optimizer could inline function calls to intrinsic wrappers just fine as well?

zwegner 7y ago
> this might save megawatts of power world wide That'd sure be neat! I really have no idea, though. > A small nitpick: not sure whether those macros bought anything, I guess the optimizer could inline function calls to intrinsic wrappers just fine as well? Yeah, that's a bit weird. It's my hacky generics-in-C method, that allows both AVX2 and SSE4 implementations to be compiled in one codebase, while keeping the base algorithm clean. The idea is that you can compile both versions by including multiple times, like so: #define AVX2 #include "z_validate.c" #undef AVX2 #define SSE4 #include "z_validate.c" #undef SSE4 ...which would allow a later runtime dispatch based on CPUID, etc.
- thechao 7y ago
  How much faster than a "naive" scalar implementation is this?
  - zwegner 7y ago
    It depends on the implementation. A good starting point would be looking at the various implementations benchmarked here: https://github.com/lemire/fastvalidate-utf-8 https://github.com/lemire/fastvalidate-utf-8 It looks like the main naive validator there (validate_utf8) clocks in at 1.25 cycles/byte for ASCII and 11 cycles/byte for UTF-8, which is respectively a 15.8x and 41.5x slowdown from my code.
FpUser 7y ago
...this might save megawatts of power world wide... Since when saved megawatts bothered programmers? If that was the case we would see far smaller usage of scripting languages and electron apps.
- marmada 7y ago
  It takes effort to not use electron. It doesn't take effort to have your library update its dependencies to use a faster utf-8 validator
  - F-0X 7y ago
    > It takes effort to not use electron. Why do you believe this? I don't think it is quite that ubiquitous.
    - jimsmart 7y ago
      Let's say I want to build a cross-platform app. I could use Electron (it's somewhat of a known quantity), or I could start doing some research into what else is available, and then evaluate the various offerings - not only to ensure they do what I need, but to also ensure that they use less power/resources than Electron. The effort here is (at least some kind of) unavoidable up-front cost/time. To switch existing code to use some particular UTF-8 decoding library should be as simple as changing some imports, and perhaps some other (search/replace) code tweaks or implementing a quick shim/adapter. If it doesn't work out, rollback or abandon the branch. On most codebases, even larger ones, this whole process is probably an easier thing to do, requiring less effort than finding a decent (and more power efficient) alternative to Electron — seems to be the GP's gist. And personally I'd agree.
      - FpUser 7y ago
        "Let's say I want to build a cross-platform app. I could use Electron (it's somewhat of a known quantity), or I could start doing some research into what else is available, and then evaluate the various offerings - not only to ensure they do what I need, but to also ensure that they use less power/resources than Electron. The effort here is (at least some kind of) unavoidable up-front cost/time." Yup that's right. Any sign of inconvenience and efficiency be damned. Why do we even bother trying to fix environment. This cost money and inconveniences so many people. Also it depends on your background. I have no problem doing without Electron. I would have to apply your exact logic to even look at it.

zwegner 7y ago

> this might save megawatts of power world wide That'd sure be neat! I really have no idea, though. > A small nitpick: not sure whether those macros bought anything, I guess the optimizer could inline function calls to intrinsic wrappers just fine as well? Yeah, that's a bit weird. It's my hacky generics-in-C method, that allows both AVX2 and SSE4 implementations to be compiled in one codebase, while keeping the base algorithm clean. The idea is that you can compile both versions by including multiple times, like so: #define AVX2 #include "z_validate.c" #undef AVX2 #define SSE4 #include "z_validate.c" #undef SSE4 ...which would allow a later runtime dispatch based on CPUID, etc.

thechao 7y ago
How much faster than a "naive" scalar implementation is this?
- zwegner 7y ago
  It depends on the implementation. A good starting point would be looking at the various implementations benchmarked here: https://github.com/lemire/fastvalidate-utf-8 https://github.com/lemire/fastvalidate-utf-8 It looks like the main naive validator there (validate_utf8) clocks in at 1.25 cycles/byte for ASCII and 11 cycles/byte for UTF-8, which is respectively a 15.8x and 41.5x slowdown from my code.

FpUser 7y ago

...this might save megawatts of power world wide... Since when saved megawatts bothered programmers? If that was the case we would see far smaller usage of scripting languages and electron apps.

marmada 7y ago
It takes effort to not use electron. It doesn't take effort to have your library update its dependencies to use a faster utf-8 validator
- F-0X 7y ago
  > It takes effort to not use electron. Why do you believe this? I don't think it is quite that ubiquitous.
  - jimsmart 7y ago
    Let's say I want to build a cross-platform app. I could use Electron (it's somewhat of a known quantity), or I could start doing some research into what else is available, and then evaluate the various offerings - not only to ensure they do what I need, but to also ensure that they use less power/resources than Electron. The effort here is (at least some kind of) unavoidable up-front cost/time. To switch existing code to use some particular UTF-8 decoding library should be as simple as changing some imports, and perhaps some other (search/replace) code tweaks or implementing a quick shim/adapter. If it doesn't work out, rollback or abandon the branch. On most codebases, even larger ones, this whole process is probably an easier thing to do, requiring less effort than finding a decent (and more power efficient) alternative to Electron — seems to be the GP's gist. And personally I'd agree.
    - FpUser 7y ago
      "Let's say I want to build a cross-platform app. I could use Electron (it's somewhat of a known quantity), or I could start doing some research into what else is available, and then evaluate the various offerings - not only to ensure they do what I need, but to also ensure that they use less power/resources than Electron. The effort here is (at least some kind of) unavoidable up-front cost/time." Yup that's right. Any sign of inconvenience and efficiency be damned. Why do we even bother trying to fix environment. This cost money and inconveniences so many people. Also it depends on your background. I have no problem doing without Electron. I would have to apply your exact logic to even look at it.