4 ms·

Neat tool. It is interesting how often SIMD stuff is discussed on here. Are people really directly dealing with SIMD calls a lot? I get the draw -- this sort

Neat tool. It is interesting how often SIMD stuff is discussed on here. Are people really directly dealing with SIMD calls a lot? I get the draw -- this sort of to-the-metal hyper-optimization is legitimately fun and intellectually rewarding -- but I suspect that in the overwhelming majority of cases simply using the appropriate library, ideally one that is cross-platform and utilizes what SIMD a given target hosts, is a far better choice than bothering with the esoterica or every platform and generation of SIMD offerings.

vectorcamp 1y ago
I agree, it's always best to use something that already exists and is optimized for your platform, unless it doesn't exist or you need extra features that are not covered. In those cases you need to read large ISA manuals, use each vendor's intrinsic site or use our tool SIMD.info :)
sophacles 1y ago
I kinda agree with the main point, but keep in mind those libraries with SIMD optimizations don't just appear out of nowhere... people write those. Also it's pretty common for someone to write software for an org that thas 10^5 or more identical cores running in a datacenter (or datacenters)... some specialized optimization can easily be cost-effective in those situations. Then there's crazy distributed systems stuff, where a small latency reduction in the right place can have significant impact for an entire cluster. And on and on.... Point being, while not everyone is in a position this stuff is relevant (and not everyone who sometimes finds this stuff relevant can say it's relevant often), it's more widely applicable than you're suggesting.
- llm_nerd 1y ago
  For sure there are obviously developers building those computation libraries like numpy, compilers, R, and so on. These people exist and are grinding out great code and abstractions for the rest of us to use, and many of them are regulars on HN. But these people are seldom the target of the "learn SIMD" content that appears on here regularly. If someone is an average developer building a game or a corporate information or financial system or even a neural network implementation, if you are touching SIMD code directly you're probably approaching things in a less than optimal fashion and there are much better ways to utilize whatever features your hardware, or hardware in the future, may offer up.
  - Pannoniae 1y ago
    This is not entirely accurate... think about it this way. Every time you issue a floating-point addition or multiplication, you're using an 8th or a 16th of your CPU's theoretical performance. Of course, it's a bit more complicated than that, but that's the general gist of it. Compilers won't generate SIMD code for you (autovectorisation) except for the simplest cases, and they certainly won't do the necessary transformations (AoS to SoA or AoSoA) necessary to efficiently use SIMD. Now of course, many of these transformations can be wrapped in a higher-level API (think of sums over an array, reductions, string length, string encoding, etc.) but not all of them. Of course, multithreading also exists to improve performance, but for many tasks, it's more worth it to run it on one core without the sync overhead, especially with data-parallel algorithms where you're doing the exact same thing on all of the data and you have a fairly large dataset. Or even better, you can combine the two, partition the data between multiple cores into a few small sets then use a SIMD "kernel" to process it. With extremely embarrassingly parallel problems, you can achieve 1000x speedups this way, not exaggerating. A typical speedup is much smaller but still, using your machine well can easily produce an order-of-magnitude difference in performance. If you read up on ISPC benchmarks, you'll find that for even any kind of existing, very branchy and not SIMD-friendly code, they regularly had a free 4x speedup without changing the behaviour or the result of the program. Seriously, it's not that if you use SIMD-powered libraries then you're set for performance, if you have a holistic view of your system's performance, you can do really amazing things.
    - llm_nerd 1y ago
      I'm not advocating against using vectorization/SIMD. Of course people should eek every bit of performance out. The point is that the average developers job in achieving this is not knowing every SIMD primitive for every platform. It is almost always folly to ever touch a platform's SIMD instructions unless you're one of the aforementioned library/compiler developers. Instead the ordinary developer is usually best served by using equipped libraries and understand how to structure code where it can enable vectorization. Whether it's Eigen, or even the C++ compiler (which, in my experience, is often excellent at vectorizing code), or the Accelerate library on Apple platforms. These things mean that your code can adapt and exploit every possible platform they're deployed to, versus Jimmy SIMD who carefully made his SSE 3.2 instructions and his code was then lost in time.

vectorcamp 1y ago

I agree, it's always best to use something that already exists and is optimized for your platform, unless it doesn't exist or you need extra features that are not covered. In those cases you need to read large ISA manuals, use each vendor's intrinsic site or use our tool SIMD.info :)

sophacles 1y ago

I kinda agree with the main point, but keep in mind those libraries with SIMD optimizations don't just appear out of nowhere... people write those. Also it's pretty common for someone to write software for an org that thas 10^5 or more identical cores running in a datacenter (or datacenters)... some specialized optimization can easily be cost-effective in those situations. Then there's crazy distributed systems stuff, where a small latency reduction in the right place can have significant impact for an entire cluster. And on and on.... Point being, while not everyone is in a position this stuff is relevant (and not everyone who sometimes finds this stuff relevant can say it's relevant often), it's more widely applicable than you're suggesting.

llm_nerd 1y ago
For sure there are obviously developers building those computation libraries like numpy, compilers, R, and so on. These people exist and are grinding out great code and abstractions for the rest of us to use, and many of them are regulars on HN. But these people are seldom the target of the "learn SIMD" content that appears on here regularly. If someone is an average developer building a game or a corporate information or financial system or even a neural network implementation, if you are touching SIMD code directly you're probably approaching things in a less than optimal fashion and there are much better ways to utilize whatever features your hardware, or hardware in the future, may offer up.
- Pannoniae 1y ago
  This is not entirely accurate... think about it this way. Every time you issue a floating-point addition or multiplication, you're using an 8th or a 16th of your CPU's theoretical performance. Of course, it's a bit more complicated than that, but that's the general gist of it. Compilers won't generate SIMD code for you (autovectorisation) except for the simplest cases, and they certainly won't do the necessary transformations (AoS to SoA or AoSoA) necessary to efficiently use SIMD. Now of course, many of these transformations can be wrapped in a higher-level API (think of sums over an array, reductions, string length, string encoding, etc.) but not all of them. Of course, multithreading also exists to improve performance, but for many tasks, it's more worth it to run it on one core without the sync overhead, especially with data-parallel algorithms where you're doing the exact same thing on all of the data and you have a fairly large dataset. Or even better, you can combine the two, partition the data between multiple cores into a few small sets then use a SIMD "kernel" to process it. With extremely embarrassingly parallel problems, you can achieve 1000x speedups this way, not exaggerating. A typical speedup is much smaller but still, using your machine well can easily produce an order-of-magnitude difference in performance. If you read up on ISPC benchmarks, you'll find that for even any kind of existing, very branchy and not SIMD-friendly code, they regularly had a free 4x speedup without changing the behaviour or the result of the program. Seriously, it's not that if you use SIMD-powered libraries then you're set for performance, if you have a holistic view of your system's performance, you can do really amazing things.
  - llm_nerd 1y ago
    I'm not advocating against using vectorization/SIMD. Of course people should eek every bit of performance out. The point is that the average developers job in achieving this is not knowing every SIMD primitive for every platform. It is almost always folly to ever touch a platform's SIMD instructions unless you're one of the aforementioned library/compiler developers. Instead the ordinary developer is usually best served by using equipped libraries and understand how to structure code where it can enable vectorization. Whether it's Eigen, or even the C++ compiler (which, in my experience, is often excellent at vectorizing code), or the Accelerate library on Apple platforms. These things mean that your code can adapt and exploit every possible platform they're deployed to, versus Jimmy SIMD who carefully made his SSE 3.2 instructions and his code was then lost in time.