4 ms·

Cool performance enhancement, with an accompanying implementation in a real-world library (https://github.com/lemire/despacer https://github.com/lemire/despacer

by mdb31 4y ago

Cool performance enhancement, with an accompanying implementation in a real-world library (https://github.com/lemire/despacer https://github.com/lemire/despacer). Still, what does it signal that vector extensions are required to get better string performance on x86? Wouldn't it be better if Intel invested their AVX transistor budget into simply making existing REPB prefixes a lot faster?

janwas 4y ago
Why is a large speedup from vectors surprising? Considering that the energy required for scheduling/dispatching an instruction on OoO cores dwarfs that of the actual operation (add/mul etc), amortizing over multiple elements (=SIMD) is an obvious win.
- mdb31 4y ago
  Where do I say that the speedup is surprising? My question is whether Intel investing in AVX-512 is wise, given that: -Most existing code is not aware of AVX anyway; -Developers are especially wary of AVX-512, since they expect it to be discontinued soon. Consequently, wouldn't Intel be better off by using the silicon dedicated to AVX-512 to speed up instruction patterns that are actually used?
  - janwas 4y ago
    My point is that vector instructions are fundamentally necessary and thus "what does it signal" evaluates to "nothing surprising". Sure, REP STOSB/MOVSB make for a very compact memset/memcpy, but their performance varies depending on CPU feature flags, so you're going to want multiple codepaths anyway. And vector instructions are vastly more flexible than just those two. Also, I have not met developers who expect AVX-512 to be discontinued (the regrettable ADL situation notwithstanding; that's not a server CPU). AMD is actually adding AVX-512.
    - mdb31 4y ago
      > vector instructions are fundamentally necessary For which percentage of users? > AMD is actually adding AVX-512 Which is irrelevant to in-market support for that instruction set.
      - XorNot 4y ago
        Why would it be irrelevant? Even the paucity of availability isn't really a problem - the big winners here are server users in data centers, not desktops or laptops. How much string parsing and munging is happening ingesting big datasets right now? If running a specially optimized function set on part of your fleet reduces utilization, that's direct cost savings you realize. If the AMD is then widening that support base, you're deeply favoring expanding usage while you scale up.
        _rtld_global_ro 4y ago
        Given Intel's AVX extension could cause silent failures on servers (very high work load for prolonged time, compare to end user computers), I'm not sure it would be a big win for servers either: https://arxiv.org/pdf/2102.11245.pdf https://arxiv.org/pdf/2102.11245.pdf.
        jcranmer 4y ago
        I'm downvoting you because the assertion you're implying--that use of AVX increases soft failure rates more than using non-AVX instructions would--is not sustained by the source you use as reference.
        tialaramex 4y ago
        Indeed, I'd summarise that source as "At Facebook sometimes weird stuff happens. We postulate it's not because of all the buggy code written by Software Engineers like us, it must be hardware. As well as lots of speculation about hypothetical widespread problems that would show we're actually not writing buggy software, here's a single concrete example where it was hardware". If anything I'd say that Core 59 is one of those exceptions that prove the rule. This is such a rare phenomenon that when it does happen you can do the work to pin it down and say yup, this CPU is busted - if it was really commonplace you'd constantly trip over these bugs and get nowhere. There probably isn't really, as that paper claims, a "systemic issue across generations" except that those generations are all running Facebook's buggy code.
  - mhh__ 4y ago
    AVX-512 is not going to be discontinued. Intel's reticence/struggling with having it on desktop is irritating but it's here to stay on servers for a long time. Writing code for a specific SIMD instruction set is non-trivial, but most code will get some benefit by being compiled for the right ISA. You don't get the really fancy instructions because the pattern matching in the compiler isn't very intelligent but quite a lot of stuff is going to benefit by magic. Even without cutting people without some AVX off, you can have a fast/slow path fairly easily.
37ef_ced3 4y ago
AVX-512 is an elegant, powerful, flexible set of masked vector instructions that is useful for many purposes. For example, low-cost neural net inference (https://NN-512.com https://NN-512.com). To suggest that Intel and AMD should instead make "existing REPB prefixes a lot faster" is missing the big picture. The masked compression instructions (one of which is used in Lemire's article) are endlessly useful, not just for stripping spaces out of a string!
- mhh__ 4y ago
  Many people seem to think AVX-512 is just wider AVX, which is a shame. NN-512 is cool. I think the Go code is pretty ugly but I like the concept of the compiler a lot.
- deleted 4y ago
  [deleted]
ip26 4y ago
Is it generally possible to convert rep str sequences to AVX? Could the hardware or compiler already be doing this? AVX is just the SIMD unit. I would argue the transistors were spent on SIMD, and the hitch is simply the best way to send str commands to the SIMD hardware.
- nwmcsween 4y ago
  Why? IIRC something like 99% of string operations are on 20 chars or less. If you're hitting bottlenecks then optimize.
  - ip26 4y ago
    If you are arguing most string ops have just a few chars and therefore don’t use vectors… why do we need to spend silicon enhancing rep prefix in the first place?

by mdb31 4y ago

janwas 4y ago

Why is a large speedup from vectors surprising? Considering that the energy required for scheduling/dispatching an instruction on OoO cores dwarfs that of the actual operation (add/mul etc), amortizing over multiple elements (=SIMD) is an obvious win.

mdb31 4y ago
Where do I say that the speedup is surprising? My question is whether Intel investing in AVX-512 is wise, given that: -Most existing code is not aware of AVX anyway; -Developers are especially wary of AVX-512, since they expect it to be discontinued soon. Consequently, wouldn't Intel be better off by using the silicon dedicated to AVX-512 to speed up instruction patterns that are actually used?
- janwas 4y ago
  My point is that vector instructions are fundamentally necessary and thus "what does it signal" evaluates to "nothing surprising". Sure, REP STOSB/MOVSB make for a very compact memset/memcpy, but their performance varies depending on CPU feature flags, so you're going to want multiple codepaths anyway. And vector instructions are vastly more flexible than just those two. Also, I have not met developers who expect AVX-512 to be discontinued (the regrettable ADL situation notwithstanding; that's not a server CPU). AMD is actually adding AVX-512.
  - mdb31 4y ago
    > vector instructions are fundamentally necessary For which percentage of users? > AMD is actually adding AVX-512 Which is irrelevant to in-market support for that instruction set.
    - XorNot 4y ago
      Why would it be irrelevant? Even the paucity of availability isn't really a problem - the big winners here are server users in data centers, not desktops or laptops. How much string parsing and munging is happening ingesting big datasets right now? If running a specially optimized function set on part of your fleet reduces utilization, that's direct cost savings you realize. If the AMD is then widening that support base, you're deeply favoring expanding usage while you scale up.
      - _rtld_global_ro 4y ago
        Given Intel's AVX extension could cause silent failures on servers (very high work load for prolonged time, compare to end user computers), I'm not sure it would be a big win for servers either: https://arxiv.org/pdf/2102.11245.pdf https://arxiv.org/pdf/2102.11245.pdf.
        jcranmer 4y ago
        I'm downvoting you because the assertion you're implying--that use of AVX increases soft failure rates more than using non-AVX instructions would--is not sustained by the source you use as reference.
        tialaramex 4y ago
        Indeed, I'd summarise that source as "At Facebook sometimes weird stuff happens. We postulate it's not because of all the buggy code written by Software Engineers like us, it must be hardware. As well as lots of speculation about hypothetical widespread problems that would show we're actually not writing buggy software, here's a single concrete example where it was hardware". If anything I'd say that Core 59 is one of those exceptions that prove the rule. This is such a rare phenomenon that when it does happen you can do the work to pin it down and say yup, this CPU is busted - if it was really commonplace you'd constantly trip over these bugs and get nowhere. There probably isn't really, as that paper claims, a "systemic issue across generations" except that those generations are all running Facebook's buggy code.
- mhh__ 4y ago
  AVX-512 is not going to be discontinued. Intel's reticence/struggling with having it on desktop is irritating but it's here to stay on servers for a long time. Writing code for a specific SIMD instruction set is non-trivial, but most code will get some benefit by being compiled for the right ISA. You don't get the really fancy instructions because the pattern matching in the compiler isn't very intelligent but quite a lot of stuff is going to benefit by magic. Even without cutting people without some AVX off, you can have a fast/slow path fairly easily.

37ef_ced3 4y ago

AVX-512 is an elegant, powerful, flexible set of masked vector instructions that is useful for many purposes. For example, low-cost neural net inference (https://NN-512.com https://NN-512.com). To suggest that Intel and AMD should instead make "existing REPB prefixes a lot faster" is missing the big picture. The masked compression instructions (one of which is used in Lemire's article) are endlessly useful, not just for stripping spaces out of a string!

mhh__ 4y ago
Many people seem to think AVX-512 is just wider AVX, which is a shame. NN-512 is cool. I think the Go code is pretty ugly but I like the concept of the compiler a lot.
deleted 4y ago
[deleted]

ip26 4y ago

Is it generally possible to convert rep str sequences to AVX? Could the hardware or compiler already be doing this? AVX is just the SIMD unit. I would argue the transistors were spent on SIMD, and the hitch is simply the best way to send str commands to the SIMD hardware.

nwmcsween 4y ago
Why? IIRC something like 99% of string operations are on 20 chars or less. If you're hitting bottlenecks then optimize.
- ip26 4y ago
  If you are arguing most string ops have just a few chars and therefore don’t use vectors… why do we need to spend silicon enhancing rep prefix in the first place?