11 ms·

A 94x speed improvement demonstrated using handwritten assembly

93 points by doener 2y ago

doener 2y ago
Via https://www.tomshardware.com/pc-components/cpus/ffmpeg-devs-boast-of-up-to-94x-performance-boost-after-implementing-handwritten-avx-512-assembly-code https://www.tomshardware.com/pc-components/cpus/ffmpeg-devs-...
jsheard 2y ago
This is a slightly misleading comparison, because they're comparing naive scalar C code to hand-vectorized assembly, skipping over hand-vectorized C with vendor intrinsics. Assembly of course always wins if enough effort is put into tuning it, but intrinsics can usually get extremely close with much less effort. For reasons unknown to me the FFmpeg team has a weird vendetta against intrinsics, they require all platform-specific code to be written in assembly even if C with intrinsics would perform exactly the same. It goes without saying that assembly will be faster if you arbitrarily forbid using the fastest C constructs.
- account42 2y ago
  > This is a slightly misleading comparison, because they're comparing naive scalar C code to hand-vectorized assembly, skipping over hand-vectorized C with vendor intrinsics. Assembly of course always wins if enough effort is put into tuning it, but intrinsics can usually get extremely close with much less effort. It's a perfectly valid comparision between straightforward C and the best hand optimization you can get. > For reasons unknown to me the FFmpeg team has a weird vendetta against intrinsics, they require all platform-specific code to be written in assembly even if C with intrinsics would perform exactly the same. Wanting to standardize on a single language for low level code is absolutely reasonable. This way contributors only need to know standard C as well as assembly instead of standar C, assembly and also intel intrisics which are a fusion of the two but also not the same as either and have their own gotchas.
  - mort96 2y ago
    I mean currently, contributors who want to work on vectorized algorithms need to know C, amd64 assembly with SSE/AVX instructions, aarch64 assembly with NEON instructions and aarch64 assembly with SVE instructions (and presumably soon, or maybe already, risc-v assembly with the vector extension). I wouldn't say it's simpler than needing to know C, C with SSE/AVX instricsics, C with NEON intrinsics, C with SVE intrinsics and C with risc-v vector intrinsics.
    - astrange 2y ago
      You don't need to submit all of those, but generally these vectorized DSP functions are parts of decoding straight out of the spec, so there isn't a new way to do it. Sometimes there is of course. In that case you can get someone else to help, or one platform just runs ahead of the others.
      - mort96 2y ago
        Right, but you see my point, don't you? For every set of intrinsics you'd need to know as a contributor if the vectorized code was in C, there is an assembly language and a set of intrinsics you need to know as a contributor today. Moving to C wouldn't increase the set of things a contributor needs to know.
- Const-me 2y ago
  > FFmpeg team has a weird vendetta against intrinsics To be fair, ffmpeg is really old software. Wikipedia says they released their initial version in the end of 2000. The software landscape was very different. Back then, there were multiple competing CPU architectures. In modern world we only have two mainstream ones, AMD64 and ARM64, two legacy ones in the process of being phased out, x86 and 32-bit ARM, and very few people care about any other CPUs. Another thing, C compilers of 2000 weren’t good in terms of performance of the generated code. Clang only arrived in 2007. In 2000, neither GCC nor VC++ had particularly good optimizers or code generators. Back in 2000, it was reasonable to use assembly for performance-critical code like that. It just they never questioned that decision later, despite they should have done that many years ago.
  - astrange 2y ago
    FFmpeg developers aren't stupid and aren't doing things because of historical accident. If you don't mind writing platform specific code, it simply works better this way. Here's clang messing up x86 intrinsics code. https://x.com/ffmpeg/status/1852913590258618852 https://x.com/ffmpeg/status/1852913590258618852 The other reason to do it is that, since x86 intrinsics are named in Hungarian notation, they're so hard to read that the asm is actually more maintainable.
    - Const-me 2y ago
      > it simply works better this way That is simply not true. > clang messing up x86 intrinsics code The code is correct, and on some processors runs slightly faster than the original. Clang is the only compiler which does anything like that. And the example is irrelevant to ffmpeg because it operates on FP64 numbers, video codecs mostly do integer math. > they're so hard to read that the asm is actually more maintainable That’s subjective, I’m using SIMD intrinsics for years and I find them way better than assembly. Another thing, you can treat C as a high-level language as opposed to portable assembler. If you define structures, functions and classes in C++ which use these SIMD vectors, readability of intrinsics becomes way better than assembly. Here’s a good example of a library designed that way: https://github.com/microsoft/DirectXMath https://github.com/microsoft/DirectXMath
- lsllc 2y ago
  The intrinsics aren't portable across architectures, so using them vs inline asm (or linking with a .s/.asm file) is more about convenience/ease of use than portability (although they might be slightly more portable across different OSes for the _same_ architecture, e.g. macOS/linux for aarch64). In some ways, I prefer the "go big or go home" of asm either inline, or in a .s/.asm file, although both inline and .s/.asm have portability issues (e.g. inline asm syntax across C/C++ compilers or the flavor of the .s/.asm files depending on your assembler).
  - jsheard 2y ago
    Intrinsics do have certain advantages besides convenience, since the compiler can reason about them in ways that it can't with an opaque blob of asssembly. For example if you set the compiler to tune for a specific CPU architecture then it can re-order intrinsics according to the instruction costs of that specific hardware, whereas with assembly you would have to manually write a separate implementation for each architecture (and AFAICT FFmpeg doesn't go that far).
    - astrange 2y ago
      Those kind of optimizations aren't needed on big desktop CPUs these days, they don't need special scheduling or anything. And yes, when it's important it does do that. They aren't stupid. (An example is that unaligned SSE loads are very slow on some CPUs but faster on others.)
- flohofwoe 2y ago
  > but intrinsics can usually get extremely close with much less effort Why 'much less effort' though? Intrinsics are on the same abstraction level as an equivalent sequence of assembly instructions aren't they? And they only target one specific ISA anyway and are not portable between CPU architectures, so the difference between coding in intrinsics and assembly doesn't seem all that big. Also I wonder if MSVC and GCC/Clang intrinsics are fully compatible to each other, compiler compatibility might be another reason to use assembly.
  - jsheard 2y ago
    Instrinsics more or less map to specific assembly instructions, but the instruction scheduling and register allocation is still handled by the compiler, which you have to do by hand in raw assembly. Possibly multiple times, since the optimal instruction scheduling is hardware specific. Intrinsics can also be automatically inlined, constant folded, etc, which you don't get with assembly. Also Intel and ARM themselves specify the C intrinsics for their architectures so they're the same across MSVC, GCC and Clang, it's not like the wild west of other compiler extensions.
    - flohofwoe 2y ago
      Hmm right, makes sense. I do wonder though if actually making use of the high level convenience then can harm performance again, e.g. let's say I'm building intrinsic parameters with regular C scalar expressions, or use if-else control flow between intrinsics. Convenient to do, but can't be great for performance versus fully simdified code.
      - janwas 2y ago
        Sure, we generally don't want branches in the code. What makes intrinsics really interesting is that you can use wrapper libraries such as our Highway, then the code actually is portable and efficient.
    - astrange 2y ago
      Clang doesn't guarantee that; it runs everything through its optimizer, which means you can use an intrinsic and not get the instruction it's named after. The other big general issue is memory aliasing. If you're working on 8-bit data then C says that means it aliases everything and it's going to deoptimize your other memory accesses. Conversely if you're freely switching types like SIMD tends to do, that's against the aliasing rules.
- snvzz 2y ago
  Relevant: https://xcancel.com/FFmpeg/status/1852847683666641124 https://xcancel.com/FFmpeg/status/1852847683666641124 And my take on intrinsics, they add, not remove, complexity. For no gain. Awful, relative to writing (or reading) asm.
- unlord 2y ago
  It is actually quite a bit more misleading. I was not able to reproduce these numbers on Zen2 hardware, see https://people.videolan.org/~unlord/dav1d_6tap.png https://people.videolan.org/~unlord/dav1d_6tap.png. I spoke with the slide author and he confirmed he was using an -O0 debug build of the checkasm binary. What's more, the C code is running an 8-tap filter where the SIMD for that function (in all of SSSE3, AVX2 and AVX512) is implemented as 6-tap. Last week I posted MR !1745 (https://code.videolan.org/videolan/dav1d/-/merge_requests/1745 https://code.videolan.org/videolan/dav1d/-/merge_requests/17...) which adds 6-tap to the C code and brings improved performance to all platforms dav1d supports. This, of course, also closes the gap in these numbers but is a more accurate representation of the speed-up from hand-written assembly.
  - zbobet2012 2y ago
    The thing I found interesting in the AVX512 gains over AVX2. That's a pretty nice gain from the wider instruction set which has often been ignored in the video community.
    - jsheard 2y ago
      Sadly AVX512 is still very easy to ignore given how badly Intel has botched its rollout across consumer processors. Ironically AMD has stronger AVX512 support at this point despite the spec originating at Intel.
    - Dylan16807 2y ago
      For most operations, on most CPUs, you can get the same results with twice as many AVX2 instructions. And that's excluding the CPUs with no AVX512 at all. But the number of situations where AVX512 has a significant advantage is growing, so interest will grow alongside it.
    - wscott 2y ago
      The important thing to understand about why AVX512 is a big deal is not the width. AVX512 adds new instructions and new instruction encodings. They doubled the number of registers (16->32), and added mask registers that allow you to remove special cases at the end of loops when the array is not a multiple of the vector width. And there is piles for new permutation operations and integer operations that allow it to be useful in more cases. The part Intel struggles with is that in many places if they had the 256-bit max width but all the new operations then they could build a machine that is faster than the 512-bit version. (assuming the same code was written for both vector widths) The reason is the ALUs could be faster and you could have more of them.
mort96 2y ago
I saw this. The problem is that it's not comparing "hand-written assembly" to "not hand-writing assembly", it's comparing scalar code with vector code. If you wrote C code using AVX intrinsics, you'd get similar speed-ups without hand-writing any assembly. And the annoying part is that there are good reasons to write hand-written assembly in some particular cases. Video decoders contain a lot of assembly for a reason: you often have extremely tight loops where 1) every instruction matters, 2) the assembly is relatively straightforward, and 3) you want dependable performance which doesn't change across compilers/compiler versions/compiler settings. But those reasons don't let you make ridiculous claims like "94x improvement from hand-written assembly compared to C".
- nicce 2y ago
  > And the annoying part is that there are good reasons to write hand-written assembly in some particular cases. The major, and well-known problem is that hand-written assembly is usually 100% non-portable. Maybe it is okay if you need the boost only in few platforms. But that still requires few different implementations.
  - mort96 2y ago
    Yeah but that's kind of a given here, C with AVX intrinsics or C with NEON intrinsics or C with SVE intrinsics is also 100% non-portable.
    - dragontamer 2y ago
      But ispc is portable between AVX512 and ARM NEON. The key is that SIMD writing needs a SIMD language. OpenCL, ISPC, CUDA, HIP and C++AMP (rip) are such examples of SIMD languages. I guess a really portable SIMD language would be WebGL / HLSL and such.
      - kolbe 2y ago
        Scalar languages can go a long ways in this exercise. There are a few hiccups in trying to vectorize scalar languages, but see how well clang's latest -O3 models can do on C++. I would love to see ISPC take off. It's my favorite way of abstracting code in this new computing paradigm. But it doesn't seem to be getting much traction. I think the future is also going to be less "SIMD" and more "MIMD" (there's probably a better term for that). You even see in AVX512 with things like aggregations (which aren't SIMD), that there's no reason a single op has to be the same exact operation spread over 512/(data size) slots. You can just as easily put popular sets of operations into the tool set.
htk 2y ago
"...Intel disabled AVX-512 for its Core 12th, 13th, and 14th Generations of Core processors, leaving owners of these CPUs without them." Well known to most but news to me, I've tried to find out the reason why but couldn't come up with a definitive answer.
- mort96 2y ago
  They introduced efficiency cores, and those don't have AVX-512. Lots of software breaks if it suddenly gets moved to a core which supports different instructions, so OSes wouldn't be able to move processes between E-cores and P-cores if P-cores supported AVX-512 while E-cores didn't.
  - account42 2y ago
    Wouldn't it have been more reasonable to emulate AVX-512 on the E-cores?
    - Retr0id 2y ago
      That doesn't sound very E-friendly
      - mort96 2y ago
        Depends. If they can essentially "just" add a bit of microcode to emulate AVX-512 with a 256-bit wide vector pipeline then it shouldn't be worse. I don't know if that's feasible though or if there are other costs (would you need physical 512-bit wide registers for example?).
    - Dylan16807 2y ago
      You can double-pump a lot of the instructions, but emulating some of the rest would lead to severe slowdowns in code that should instead be using an AVX2 implementation. It's better not to fake it that hard. If those cores don't have it, don't pretend to have it. But turning it off on the P cores was dumb.
  - 2y ago
tiffanyh 2y ago
Similarly, Mike Pall of LuaJIT fame wrote (13-years ago) about the benefits of handwriting assembly. http://lua-users.org/lists/lua-l/2011-02/msg00742.html http://lua-users.org/lists/lua-l/2011-02/msg00742.html
- janwas 2y ago
  This advice seems specific to an interpreter with "hundreds of slow paths" :)
djaychela 2y ago
Seeing "handwritten assembly" always reminds me of the claims that Steve Gibson made about the code he wrote...
- vanderZwan 2y ago
  Which are what exactly?
bruh2 2y ago
https://xcancel.com/FFmpeg/status/1852542388851601913 https://xcancel.com/FFmpeg/status/1852542388851601913
rqtwteye 2y ago
I used to work on video encoding/decoding. Good times! I thoroughly enjoyed running the profiler and squeezing out a few percent here and there. It always amazed me what features CPUs have once you have an understanding of their low level architectures. They are true marvels.
deleted 2y ago
[deleted]
binary132 2y ago
I often think that our “portable assembler” isn’t really what it says on the tin any more, and that this is an opportunity for a new programming paradigm to be developed where it once stood.
renewiltord 2y ago
This is interesting. I thought one should generally avoid the vectorized AVX-512 because most processors with it get thermally throttled. I'll definitely take a closer look now. By the way, whenever I've written platform-specific code we've just #ifdef'd it and compiled it one way or the other. How does ffmpeg and other binary-distributed software do it? You can't late compile on the platform so presumably they select implementations at runtime? Install-time configuration seems too risky (people move hard drives and so on).
- wmf 2y ago
  AVX-512 throttling has improved over time so it's much safer to use now.
- cesarb 2y ago
  > I thought one should generally avoid the vectorized AVX-512 because most processors with it get thermally throttled. AFAIK, that's only the case for Intel processors; on AMD processors, you can use AVX-512 without fear. > How does ffmpeg and other binary-distributed software do it? You can't late compile on the platform so presumably they select implementations at runtime? Yes, that's what they do, either manually, automatically with compiler help (ifunc or similar), or with a full copy of the whole library which is loaded from an alternative directory whenever the necessary hardware features are present (for instance, when AVX-512 and a number of other features is available, the x86-64-v4 subdirectory will be used; run "/lib64/ld-linux-x86-64.so.2 --help" to see which ones would be used on your system).
cjensen 2y ago
To clarify, it's 94x the performance of the naive C implementation in just one type of filter. On the same filter, the table posted to twitter shows SSSE3 at 40x and AVX2 at 67x. So maybe just a case where most users were using AVX/SSE and there was no reason to optimize the C. But this is just one feature of FFmpeg. Usually the heaviest CPU user is encode and decode, which is not affected by this improvement. It's interesting and good work, but the "94x" statement is misleading.
- Temporary_31337 2y ago
  How does this compare to the naive GPU performance on an RTX GPU?
  - H8crilA 2y ago
    The GPU is a lot slower than the CPU - per core/thread. Otherwise you would need to select different core/thread counts on both sides, or set something like a power limit in watts. When it comes to watts the process node (nanometers) will largely determine the outcome.
  - astrange 2y ago
    If the entire program was on the GPU, that specific part would be extremely fast. But the rest of it is on the CPU because GPU cores aren't any good at largely serial things like video decoding. So it doesn't matter.
    - mywittyname 2y ago
      > GPU cores aren't any good at largely serial things like video decoding I'm surprised to hear this, considering GPUs are often used for video encoding and decoding. For Nvidia cards, this is called NVDEC, and AMD/Intel both have corresponding features for their video cards.
      - astrange 2y ago
        That doesn't use the GPU cores. They just happen to package dedicated video hardware alongside it, but it's more similar to a CPU with a specialized DSP attached. Any kind of decompression isn't fully parallelizable. If you've found any opportunities, that means the compression wasn't as efficient as it theoretically could be. Most codecs are merciful and eg restart the entropy coder across frames, which is why the multithreaded decoding in ffmpeg is able to work. (But it comes with a lossless video codec called ffv1 that doesn't allow this.)
bhouston 2y ago
These were on a micro-benchmark, a single function got 94x faster compared to raw C. This is similar to this other improvement that was also on a micro-benchmark: https://news.ycombinator.com/item?id=42007695 https://news.ycombinator.com/item?id=42007695
deleted 2y ago
[deleted]
CyberDildonics 2y ago
A 94x performance boost will not come from just writing some assembly instructions. It will come from changing terrible memory access patterns to be optimal, not allocating memory in a hot loop and using SIMD instructions. At 94x there could be some algorithmic changes too, like not doing redundant calculations on pixels that can have their order reversed and applied to other pixels.
cornstalks 2y ago
For the curious, this is fresh from VDD24[1]. The 94x is mildly contested since IIRC how you write the C code really matters (e.g. scalar vs autovectorized). But even with vectorized C code the speed up was something like >10x, which is still great but not as fun to say. One clarification: this is an optimization in dav1d, not FFmpeg. FFmpeg uses dav1d, so it can take advantage of this, but so can other non-FFmpeg programs that use dav1d. If you do any video handling consider adding dav1d to your arsenal! There’s currently a call for RISC-V (64-bit) optimizations in dav1d. If you want to dip your toes in RISC-V optimizations and assembly this is a great opportunity. We need more! [1]: https://www.videolan.org/videolan/events/vdd24/ https://www.videolan.org/videolan/events/vdd24/
kristianp 2y ago
Is this a case where x64 beats out Apple Silicon, (at least for AMDs Zen4 and Zen 5 and Intel servers)? What size are the SVE registers on M3/M4 [1]? Sorry for sounding inflammatory, I'm genuinely interested in the performance on Apple Silicon. I expect ffmpeg devs at least have NEON implementations, if not SVE for the newer Mx chips. [1] The svcntb intrinsic will tell you, apparently https://lemire.me/blog/2022/11/29/how-big-are-your-sve-registers-aws-graviton/ https://lemire.me/blog/2022/11/29/how-big-are-your-sve-regis...
deleted 2y ago
[deleted]
brcmthrowaway 2y ago
Why not get an LLM to convert the whole app to asm?
- smitelli 2y ago
  Or… a compiler?
- emodendroket 2y ago
  They already have a program that will convert C code into equivalent ASM instructions.
HarHarVeryFunny 2y ago
The title gives the impression that handwritten assembly code is the cause of this performance improvement, when in fact it was use of AVX-512 SIMD instructions that really made the difference. I've got to wonder how gcc's AVX-512 vectorization would compare ?
quantadev 2y ago
There was probably just one single bottleneck that got fixed by the ASM code. Often in my experience, not enough work is done to identify what the bottleneck is before trying to solve performance. It's not like every line of C takes N amount of time and every line of ASM takes a fraction of N. Most lines of C compile to the optimal ASM right off the bat. If this was one of those case like 90% of cases are, where you just need to identify the bottleneck, you can normally fix it and stay in C (not ASM), although I used to enjoy doing some inline ASM in my C code to milk out every last bit of performance in critical loops.

93 points by doener 2y ago

Via https://www.tomshardware.com/pc-components/cpus/ffmpeg-devs-boast-of-up-to-94x-performance-boost-after-implementing-handwritten-avx-512-assembly-code https://www.tomshardware.com/pc-components/cpus/ffmpeg-devs-...

jsheard 2y ago

This is a slightly misleading comparison, because they're comparing naive scalar C code to hand-vectorized assembly, skipping over hand-vectorized C with vendor intrinsics. Assembly of course always wins if enough effort is put into tuning it, but intrinsics can usually get extremely close with much less effort. For reasons unknown to me the FFmpeg team has a weird vendetta against intrinsics, they require all platform-specific code to be written in assembly even if C with intrinsics would perform exactly the same. It goes without saying that assembly will be faster if you arbitrarily forbid using the fastest C constructs.

account42 2y ago
> This is a slightly misleading comparison, because they're comparing naive scalar C code to hand-vectorized assembly, skipping over hand-vectorized C with vendor intrinsics. Assembly of course always wins if enough effort is put into tuning it, but intrinsics can usually get extremely close with much less effort. It's a perfectly valid comparision between straightforward C and the best hand optimization you can get. > For reasons unknown to me the FFmpeg team has a weird vendetta against intrinsics, they require all platform-specific code to be written in assembly even if C with intrinsics would perform exactly the same. Wanting to standardize on a single language for low level code is absolutely reasonable. This way contributors only need to know standard C as well as assembly instead of standar C, assembly and also intel intrisics which are a fusion of the two but also not the same as either and have their own gotchas.
- mort96 2y ago
  I mean currently, contributors who want to work on vectorized algorithms need to know C, amd64 assembly with SSE/AVX instructions, aarch64 assembly with NEON instructions and aarch64 assembly with SVE instructions (and presumably soon, or maybe already, risc-v assembly with the vector extension). I wouldn't say it's simpler than needing to know C, C with SSE/AVX instricsics, C with NEON intrinsics, C with SVE intrinsics and C with risc-v vector intrinsics.
  - astrange 2y ago
    You don't need to submit all of those, but generally these vectorized DSP functions are parts of decoding straight out of the spec, so there isn't a new way to do it. Sometimes there is of course. In that case you can get someone else to help, or one platform just runs ahead of the others.
    - mort96 2y ago
      Right, but you see my point, don't you? For every set of intrinsics you'd need to know as a contributor if the vectorized code was in C, there is an assembly language and a set of intrinsics you need to know as a contributor today. Moving to C wouldn't increase the set of things a contributor needs to know.
Const-me 2y ago
> FFmpeg team has a weird vendetta against intrinsics To be fair, ffmpeg is really old software. Wikipedia says they released their initial version in the end of 2000. The software landscape was very different. Back then, there were multiple competing CPU architectures. In modern world we only have two mainstream ones, AMD64 and ARM64, two legacy ones in the process of being phased out, x86 and 32-bit ARM, and very few people care about any other CPUs. Another thing, C compilers of 2000 weren’t good in terms of performance of the generated code. Clang only arrived in 2007. In 2000, neither GCC nor VC++ had particularly good optimizers or code generators. Back in 2000, it was reasonable to use assembly for performance-critical code like that. It just they never questioned that decision later, despite they should have done that many years ago.
- astrange 2y ago
  FFmpeg developers aren't stupid and aren't doing things because of historical accident. If you don't mind writing platform specific code, it simply works better this way. Here's clang messing up x86 intrinsics code. https://x.com/ffmpeg/status/1852913590258618852 https://x.com/ffmpeg/status/1852913590258618852 The other reason to do it is that, since x86 intrinsics are named in Hungarian notation, they're so hard to read that the asm is actually more maintainable.
  - Const-me 2y ago
    > it simply works better this way That is simply not true. > clang messing up x86 intrinsics code The code is correct, and on some processors runs slightly faster than the original. Clang is the only compiler which does anything like that. And the example is irrelevant to ffmpeg because it operates on FP64 numbers, video codecs mostly do integer math. > they're so hard to read that the asm is actually more maintainable That’s subjective, I’m using SIMD intrinsics for years and I find them way better than assembly. Another thing, you can treat C as a high-level language as opposed to portable assembler. If you define structures, functions and classes in C++ which use these SIMD vectors, readability of intrinsics becomes way better than assembly. Here’s a good example of a library designed that way: https://github.com/microsoft/DirectXMath https://github.com/microsoft/DirectXMath
lsllc 2y ago
The intrinsics aren't portable across architectures, so using them vs inline asm (or linking with a .s/.asm file) is more about convenience/ease of use than portability (although they might be slightly more portable across different OSes for the _same_ architecture, e.g. macOS/linux for aarch64). In some ways, I prefer the "go big or go home" of asm either inline, or in a .s/.asm file, although both inline and .s/.asm have portability issues (e.g. inline asm syntax across C/C++ compilers or the flavor of the .s/.asm files depending on your assembler).
- jsheard 2y ago
  Intrinsics do have certain advantages besides convenience, since the compiler can reason about them in ways that it can't with an opaque blob of asssembly. For example if you set the compiler to tune for a specific CPU architecture then it can re-order intrinsics according to the instruction costs of that specific hardware, whereas with assembly you would have to manually write a separate implementation for each architecture (and AFAICT FFmpeg doesn't go that far).
  - astrange 2y ago
    Those kind of optimizations aren't needed on big desktop CPUs these days, they don't need special scheduling or anything. And yes, when it's important it does do that. They aren't stupid. (An example is that unaligned SSE loads are very slow on some CPUs but faster on others.)
flohofwoe 2y ago
> but intrinsics can usually get extremely close with much less effort Why 'much less effort' though? Intrinsics are on the same abstraction level as an equivalent sequence of assembly instructions aren't they? And they only target one specific ISA anyway and are not portable between CPU architectures, so the difference between coding in intrinsics and assembly doesn't seem all that big. Also I wonder if MSVC and GCC/Clang intrinsics are fully compatible to each other, compiler compatibility might be another reason to use assembly.
- jsheard 2y ago
  Instrinsics more or less map to specific assembly instructions, but the instruction scheduling and register allocation is still handled by the compiler, which you have to do by hand in raw assembly. Possibly multiple times, since the optimal instruction scheduling is hardware specific. Intrinsics can also be automatically inlined, constant folded, etc, which you don't get with assembly. Also Intel and ARM themselves specify the C intrinsics for their architectures so they're the same across MSVC, GCC and Clang, it's not like the wild west of other compiler extensions.
  - flohofwoe 2y ago
    Hmm right, makes sense. I do wonder though if actually making use of the high level convenience then can harm performance again, e.g. let's say I'm building intrinsic parameters with regular C scalar expressions, or use if-else control flow between intrinsics. Convenient to do, but can't be great for performance versus fully simdified code.
    - janwas 2y ago
      Sure, we generally don't want branches in the code. What makes intrinsics really interesting is that you can use wrapper libraries such as our Highway, then the code actually is portable and efficient.
  - astrange 2y ago
    Clang doesn't guarantee that; it runs everything through its optimizer, which means you can use an intrinsic and not get the instruction it's named after. The other big general issue is memory aliasing. If you're working on 8-bit data then C says that means it aliases everything and it's going to deoptimize your other memory accesses. Conversely if you're freely switching types like SIMD tends to do, that's against the aliasing rules.
snvzz 2y ago
Relevant: https://xcancel.com/FFmpeg/status/1852847683666641124 https://xcancel.com/FFmpeg/status/1852847683666641124 And my take on intrinsics, they add, not remove, complexity. For no gain. Awful, relative to writing (or reading) asm.
unlord 2y ago
It is actually quite a bit more misleading. I was not able to reproduce these numbers on Zen2 hardware, see https://people.videolan.org/~unlord/dav1d_6tap.png https://people.videolan.org/~unlord/dav1d_6tap.png. I spoke with the slide author and he confirmed he was using an -O0 debug build of the checkasm binary. What's more, the C code is running an 8-tap filter where the SIMD for that function (in all of SSSE3, AVX2 and AVX512) is implemented as 6-tap. Last week I posted MR !1745 (https://code.videolan.org/videolan/dav1d/-/merge_requests/1745 https://code.videolan.org/videolan/dav1d/-/merge_requests/17...) which adds 6-tap to the C code and brings improved performance to all platforms dav1d supports. This, of course, also closes the gap in these numbers but is a more accurate representation of the speed-up from hand-written assembly.
- zbobet2012 2y ago
  The thing I found interesting in the AVX512 gains over AVX2. That's a pretty nice gain from the wider instruction set which has often been ignored in the video community.
  - jsheard 2y ago
    Sadly AVX512 is still very easy to ignore given how badly Intel has botched its rollout across consumer processors. Ironically AMD has stronger AVX512 support at this point despite the spec originating at Intel.
  - Dylan16807 2y ago
    For most operations, on most CPUs, you can get the same results with twice as many AVX2 instructions. And that's excluding the CPUs with no AVX512 at all. But the number of situations where AVX512 has a significant advantage is growing, so interest will grow alongside it.
  - wscott 2y ago
    The important thing to understand about why AVX512 is a big deal is not the width. AVX512 adds new instructions and new instruction encodings. They doubled the number of registers (16->32), and added mask registers that allow you to remove special cases at the end of loops when the array is not a multiple of the vector width. And there is piles for new permutation operations and integer operations that allow it to be useful in more cases. The part Intel struggles with is that in many places if they had the 256-bit max width but all the new operations then they could build a machine that is faster than the 512-bit version. (assuming the same code was written for both vector widths) The reason is the ALUs could be faster and you could have more of them.

mort96 2y ago

I saw this. The problem is that it's not comparing "hand-written assembly" to "not hand-writing assembly", it's comparing scalar code with vector code. If you wrote C code using AVX intrinsics, you'd get similar speed-ups without hand-writing any assembly. And the annoying part is that there are good reasons to write hand-written assembly in some particular cases. Video decoders contain a lot of assembly for a reason: you often have extremely tight loops where 1) every instruction matters, 2) the assembly is relatively straightforward, and 3) you want dependable performance which doesn't change across compilers/compiler versions/compiler settings. But those reasons don't let you make ridiculous claims like "94x improvement from hand-written assembly compared to C".

nicce 2y ago
> And the annoying part is that there are good reasons to write hand-written assembly in some particular cases. The major, and well-known problem is that hand-written assembly is usually 100% non-portable. Maybe it is okay if you need the boost only in few platforms. But that still requires few different implementations.
- mort96 2y ago
  Yeah but that's kind of a given here, C with AVX intrinsics or C with NEON intrinsics or C with SVE intrinsics is also 100% non-portable.
  - dragontamer 2y ago
    But ispc is portable between AVX512 and ARM NEON. The key is that SIMD writing needs a SIMD language. OpenCL, ISPC, CUDA, HIP and C++AMP (rip) are such examples of SIMD languages. I guess a really portable SIMD language would be WebGL / HLSL and such.
    - kolbe 2y ago
      Scalar languages can go a long ways in this exercise. There are a few hiccups in trying to vectorize scalar languages, but see how well clang's latest -O3 models can do on C++. I would love to see ISPC take off. It's my favorite way of abstracting code in this new computing paradigm. But it doesn't seem to be getting much traction. I think the future is also going to be less "SIMD" and more "MIMD" (there's probably a better term for that). You even see in AVX512 with things like aggregations (which aren't SIMD), that there's no reason a single op has to be the same exact operation spread over 512/(data size) slots. You can just as easily put popular sets of operations into the tool set.

htk 2y ago

"...Intel disabled AVX-512 for its Core 12th, 13th, and 14th Generations of Core processors, leaving owners of these CPUs without them." Well known to most but news to me, I've tried to find out the reason why but couldn't come up with a definitive answer.

mort96 2y ago
They introduced efficiency cores, and those don't have AVX-512. Lots of software breaks if it suddenly gets moved to a core which supports different instructions, so OSes wouldn't be able to move processes between E-cores and P-cores if P-cores supported AVX-512 while E-cores didn't.
- account42 2y ago
  Wouldn't it have been more reasonable to emulate AVX-512 on the E-cores?
  - Retr0id 2y ago
    That doesn't sound very E-friendly
    - mort96 2y ago
      Depends. If they can essentially "just" add a bit of microcode to emulate AVX-512 with a 256-bit wide vector pipeline then it shouldn't be worse. I don't know if that's feasible though or if there are other costs (would you need physical 512-bit wide registers for example?).
  - Dylan16807 2y ago
    You can double-pump a lot of the instructions, but emulating some of the rest would lead to severe slowdowns in code that should instead be using an AVX2 implementation. It's better not to fake it that hard. If those cores don't have it, don't pretend to have it. But turning it off on the P cores was dumb.
- 2y ago

tiffanyh 2y ago

Similarly, Mike Pall of LuaJIT fame wrote (13-years ago) about the benefits of handwriting assembly. http://lua-users.org/lists/lua-l/2011-02/msg00742.html http://lua-users.org/lists/lua-l/2011-02/msg00742.html

janwas 2y ago
This advice seems specific to an interpreter with "hundreds of slow paths" :)

djaychela 2y ago

Seeing "handwritten assembly" always reminds me of the claims that Steve Gibson made about the code he wrote...

vanderZwan 2y ago
Which are what exactly?

bruh2 2y ago

https://xcancel.com/FFmpeg/status/1852542388851601913 https://xcancel.com/FFmpeg/status/1852542388851601913

rqtwteye 2y ago

I used to work on video encoding/decoding. Good times! I thoroughly enjoyed running the profiler and squeezing out a few percent here and there. It always amazed me what features CPUs have once you have an understanding of their low level architectures. They are true marvels.

deleted 2y ago

[deleted]

binary132 2y ago

I often think that our “portable assembler” isn’t really what it says on the tin any more, and that this is an opportunity for a new programming paradigm to be developed where it once stood.

renewiltord 2y ago

This is interesting. I thought one should generally avoid the vectorized AVX-512 because most processors with it get thermally throttled. I'll definitely take a closer look now. By the way, whenever I've written platform-specific code we've just #ifdef'd it and compiled it one way or the other. How does ffmpeg and other binary-distributed software do it? You can't late compile on the platform so presumably they select implementations at runtime? Install-time configuration seems too risky (people move hard drives and so on).

wmf 2y ago
AVX-512 throttling has improved over time so it's much safer to use now.
cesarb 2y ago
> I thought one should generally avoid the vectorized AVX-512 because most processors with it get thermally throttled. AFAIK, that's only the case for Intel processors; on AMD processors, you can use AVX-512 without fear. > How does ffmpeg and other binary-distributed software do it? You can't late compile on the platform so presumably they select implementations at runtime? Yes, that's what they do, either manually, automatically with compiler help (ifunc or similar), or with a full copy of the whole library which is loaded from an alternative directory whenever the necessary hardware features are present (for instance, when AVX-512 and a number of other features is available, the x86-64-v4 subdirectory will be used; run "/lib64/ld-linux-x86-64.so.2 --help" to see which ones would be used on your system).

cjensen 2y ago

To clarify, it's 94x the performance of the naive C implementation in just one type of filter. On the same filter, the table posted to twitter shows SSSE3 at 40x and AVX2 at 67x. So maybe just a case where most users were using AVX/SSE and there was no reason to optimize the C. But this is just one feature of FFmpeg. Usually the heaviest CPU user is encode and decode, which is not affected by this improvement. It's interesting and good work, but the "94x" statement is misleading.

Temporary_31337 2y ago
How does this compare to the naive GPU performance on an RTX GPU?
- H8crilA 2y ago
  The GPU is a lot slower than the CPU - per core/thread. Otherwise you would need to select different core/thread counts on both sides, or set something like a power limit in watts. When it comes to watts the process node (nanometers) will largely determine the outcome.
- astrange 2y ago
  If the entire program was on the GPU, that specific part would be extremely fast. But the rest of it is on the CPU because GPU cores aren't any good at largely serial things like video decoding. So it doesn't matter.
  - mywittyname 2y ago
    > GPU cores aren't any good at largely serial things like video decoding I'm surprised to hear this, considering GPUs are often used for video encoding and decoding. For Nvidia cards, this is called NVDEC, and AMD/Intel both have corresponding features for their video cards.
    - astrange 2y ago
      That doesn't use the GPU cores. They just happen to package dedicated video hardware alongside it, but it's more similar to a CPU with a specialized DSP attached. Any kind of decompression isn't fully parallelizable. If you've found any opportunities, that means the compression wasn't as efficient as it theoretically could be. Most codecs are merciful and eg restart the entropy coder across frames, which is why the multithreaded decoding in ffmpeg is able to work. (But it comes with a lossless video codec called ffv1 that doesn't allow this.)

bhouston 2y ago

These were on a micro-benchmark, a single function got 94x faster compared to raw C. This is similar to this other improvement that was also on a micro-benchmark: https://news.ycombinator.com/item?id=42007695 https://news.ycombinator.com/item?id=42007695

deleted 2y ago

[deleted]

CyberDildonics 2y ago

A 94x performance boost will not come from just writing some assembly instructions. It will come from changing terrible memory access patterns to be optimal, not allocating memory in a hot loop and using SIMD instructions. At 94x there could be some algorithmic changes too, like not doing redundant calculations on pixels that can have their order reversed and applied to other pixels.

cornstalks 2y ago

For the curious, this is fresh from VDD24[1]. The 94x is mildly contested since IIRC how you write the C code really matters (e.g. scalar vs autovectorized). But even with vectorized C code the speed up was something like >10x, which is still great but not as fun to say. One clarification: this is an optimization in dav1d, not FFmpeg. FFmpeg uses dav1d, so it can take advantage of this, but so can other non-FFmpeg programs that use dav1d. If you do any video handling consider adding dav1d to your arsenal! There’s currently a call for RISC-V (64-bit) optimizations in dav1d. If you want to dip your toes in RISC-V optimizations and assembly this is a great opportunity. We need more! [1]: https://www.videolan.org/videolan/events/vdd24/ https://www.videolan.org/videolan/events/vdd24/

kristianp 2y ago

Is this a case where x64 beats out Apple Silicon, (at least for AMDs Zen4 and Zen 5 and Intel servers)? What size are the SVE registers on M3/M4 [1]? Sorry for sounding inflammatory, I'm genuinely interested in the performance on Apple Silicon. I expect ffmpeg devs at least have NEON implementations, if not SVE for the newer Mx chips. [1] The svcntb intrinsic will tell you, apparently https://lemire.me/blog/2022/11/29/how-big-are-your-sve-registers-aws-graviton/ https://lemire.me/blog/2022/11/29/how-big-are-your-sve-regis...

deleted 2y ago

[deleted]

brcmthrowaway 2y ago

Why not get an LLM to convert the whole app to asm?

smitelli 2y ago
Or… a compiler?
emodendroket 2y ago
They already have a program that will convert C code into equivalent ASM instructions.

HarHarVeryFunny 2y ago

The title gives the impression that handwritten assembly code is the cause of this performance improvement, when in fact it was use of AVX-512 SIMD instructions that really made the difference. I've got to wonder how gcc's AVX-512 vectorization would compare ?

quantadev 2y ago

There was probably just one single bottleneck that got fixed by the ASM code. Often in my experience, not enough work is done to identify what the bottleneck is before trying to solve performance. It's not like every line of C takes N amount of time and every line of ASM takes a fraction of N. Most lines of C compile to the optimal ASM right off the bat. If this was one of those case like 90% of cases are, where you just need to identify the bottleneck, you can normally fix it and stay in C (not ASM), although I used to enjoy doing some inline ASM in my C code to milk out every last bit of performance in critical loops.