4 ms·

The argument seems to be that C isn't suited to SIMD because some functions (exp, for instance) are not. But it then talks about how exp is implemented in hard

The argument seems to be that C isn't suited to SIMD because some functions (exp, for instance) are not. But it then talks about how exp is implemented in hardware, in a way that forces it to be a non-SIMD calculation. That's not a C issue, that's a CPU issue. No language can save you there (unless it uses a software-only function for exp). Is this article really confused, or did I misunderstand it? The thing that makes C/C++ a good language for SIMD is how easily it lets you control memory alignment.

high_na_euv 2y ago
Even high level languages like c# allow you to express such a thing https://stackoverflow.com/questions/73269625/create-memory-aligned-array-in-c-sharp https://stackoverflow.com/questions/73269625/create-memory-a... Also structlayout and fieldoffset
leecarraher 2y ago
i think the article is confused. i write quite a bit of SIMD code for HPC and c is my goto for low level computing because of the low level memory allocation in c. In fact that tends to be the lion share of work to implement simd operations where memory isn't the bottleneck. if only it were as simple as calling vfmadd132ps on two datatypes.
woooooo 2y ago
Not in the article, but I've read that the way C does pointers, a single address with length implicit, makes it hard for compilers to assert that 2 arrays don't overlap/alias and this is an obstacle for generating SIMD instructions.
- 1718627440 2y ago
  Not really. Pointers from different allocations are guaranteed to never alias. If they do, it is UB.
  - AnimalMuppet 2y ago
    Right, but the compiler in general has no way to know that pointers are from different allocations - especially several function calls away from the allocations, with some pointer arithmetic in between.
    - 1718627440 2y ago
      Yes, that is why restrict was introduced.
    - bee_rider 2y ago
      In general, the information as to whether or not the pointers will alias might not even be known, right? Like we could be compiling a library that will take in some pointers from the user.
- wayoverthecloud 2y ago
  In C++ there is the align_alloc specificier. https://en.cppreference.com/w/c/memory/aligned_alloc https://en.cppreference.com/w/c/memory/aligned_alloc Not sure for C
  - ryanpetrich 2y ago
    aligned_alloc is a C function. It doesn't help the compiler prove that two pointers can't alias. restrict is the keyword that does.
    - bobmcnamara 2y ago
      The compiler is allowed to assume the results of malloc and certain other allocation functions do not alias. Restrict does too.
- alkh 2y ago
  I thought you could simply add `restrict`[1] to indicate that only one pointer points to a specific object? Shouldn't this help? [1]https://en.cppreference.com/w/c/language/restrict https://en.cppreference.com/w/c/language/restrict
  - aidenn0 2y ago
    You can add that qualifier, but nothing will enforce that objects don't overlap. So you replaced slow correct code with fast incorrect code.
    - TheNewAndy 2y ago
      If you believed the pointers didn't alias and they actually did then you replaced slow incorrect code with fast incorrect code
      - aidenn0 2y ago
        That doesn't follow? If you replaced every call of memcpy with memmove (to use an example from the standard library), then your program is no less correct than it was before (and possibly more). The converse is that adding "restrict" qualifiers to existing functions can only make your program less correct, not more.
        ablob 2y ago
        In this context it follows. If you believe the pointers dont alias (and want to use simd) then there should be no data-dependencies inbetween indices. (If there are data-dependencies it is quite likely that you have designed your algorithm to accomodate that, i.e. make it suitable for vectorised access. parallel prefix-sum would be an example of this). And without data-dependencies vectorized code is exactly equal to unvectorized code in all circumstances. If however, you do have data-dependencies, then your algorithm is wrong. Vectorizing can surface this error, while keeping it unvectorized may hide it. In my previous example (parallel prefix-sum) if you had a direct data-dependency to the previous index your program would've been wrong - regardless of vectorization. In short, while your statement is true in general, I believe that it is not applicable in the context of the discussion.
- drysine 2y ago
  That's what the restrict type qualifier[0] is for. [0] https://en.cppreference.com/w/c/language/restrict https://en.cppreference.com/w/c/language/restrict
- bee_rider 2y ago
  When compiling a library, it is generally impossible for the compiler to know whether or not the pointers will be aliased, right? That decision is made after the library is already compiled.
  - drwu 2y ago
    The function declaration in the header file of the library can carry the required 'restrict'. It works for c++ invokers too, as most c++ compilers also support and check the __restrict__ for old plain pointer types.
adgjlsfhk1 2y ago
> But it then talks about how exp is implemented in hardware, in a way that forces it to be a non-SIMD calculation. This is especially surprising given that very few architectures have an `exp` in hardware. It's almost always done in software.
stabbles 2y ago
I think the interpretation is more conceptual about functions, reusability and composability. The function exp has an abstract mathematical definition as mapping from numbers to numbers, and you could implement that generically if the language allows for it, but in C you cannot because it's bound to the signature `double exp(double)` and fixed like that at compile time. You cannot use this function in a different context and pass e.g. __m256d. Basically because C does not have function dispatch, it's ill-suited for generic programming. You cannot write an algorithm on scalar, and now pass arrays instead.
- Someone 2y ago
  > You cannot write an algorithm on scalar, and now pass arrays instead. It is limited to cases where you know all overloads beforehand and limited because of C’s weak type system (can’t have an overload for arrays of int and one for pointers to int, for example), but you can. https://en.cppreference.com/w/c/language/generic https://en.cppreference.com/w/c/language/generic: #include <math.h> #include <stdio.h> // Possible implementation of the tgmath.h macro cbrt #define cbrt(X) _Generic((X), \ long double: cbrtl, \ default: cbrt, \ float: cbrtf \ )(X) int main(void) { double x = 8.0; const float y = 3.375; printf("cbrt(8.0) = %f\n", cbrt(x)); // selects the default cbrt printf("cbrtf(3.375) = %f\n", cbrt(y)); // converts const float to float, // then selects cbrtf } It also, IMO, is a bit ugly, but that fits in nice with the rest of C, as seen through modern eyes. There also is a drawback for lots of cases that you cannot overload operators. Implementing vector addition as ‘+’ isn’t possible, for example. Many compilers partially support that as a language extension, though, see for example https://clang.llvm.org/docs/LanguageExtensions.html#vectors-and-extended-vectors https://clang.llvm.org/docs/LanguageExtensions.html#vectors-....
  - uecker 2y ago
    I am not sure what you mean by "weak" type system. C's type system is not weak IMHO and also differentiates between arrays and pointers: _Generic((typeof(x), int*: 1, int[]: 2) \_Generic with type argument is a new feature, it is a bit less elegant without it: _Generic((typeof(x)*)0, typeof(int*)*: 1, typeof(int[])\*: 2)
ta3401 2y ago
> But it then talks about how exp is implemented in hardware, in a way that forces it to be a non-SIMD calculation. The fact that exp is implemented in hardware is not the argument. The argument is that exp is a library function, compiled separately, and thus the compiler cannot inline the function and fuse it with an array-wide loop, to detect later on an opportunity to generate the SIMD instructions. It is true however that exp is implemented in hardware in the X86 world, and to be fair, perhaps a C compiler only needs to represent that function as an intrinsic instead, to give itself a chance to later replace it either with the function call or some SIMD instruction; but I guess that the standard doesn't provide that possibility?
- AlotOfReading 2y ago
  The compiler can (and does) know enough about the stdlib functions to treat them specially. The C standard doesn't require that libm exist as a separately linked library either, that's just an implementation detail from an earlier time that got enshrined in POSIX.
- vmchale 2y ago
  > perhaps a C compiler only needs to represent that function as an intrinsic instead, to give itself a chance to later replace it either with the function call or some SIMD instruction I gesture to this in the blog post: > In C, one writes a function, and it is exported in an object file. To appreciate why this is special, consider sum :: Num a => [a] -> a in Haskell. This function exists only in the context of GHC. > ... > perhaps there are more fluent methods for compilation (and better data structures for export à la object files).
  - ta3401 2y ago
    I now realize that I somewhat rephrased what was said in the blog. How strange!

high_na_euv 2y ago

Even high level languages like c# allow you to express such a thing https://stackoverflow.com/questions/73269625/create-memory-aligned-array-in-c-sharp https://stackoverflow.com/questions/73269625/create-memory-a... Also structlayout and fieldoffset

leecarraher 2y ago

i think the article is confused. i write quite a bit of SIMD code for HPC and c is my goto for low level computing because of the low level memory allocation in c. In fact that tends to be the lion share of work to implement simd operations where memory isn't the bottleneck. if only it were as simple as calling vfmadd132ps on two datatypes.

woooooo 2y ago

Not in the article, but I've read that the way C does pointers, a single address with length implicit, makes it hard for compilers to assert that 2 arrays don't overlap/alias and this is an obstacle for generating SIMD instructions.

1718627440 2y ago
Not really. Pointers from different allocations are guaranteed to never alias. If they do, it is UB.
- AnimalMuppet 2y ago
  Right, but the compiler in general has no way to know that pointers are from different allocations - especially several function calls away from the allocations, with some pointer arithmetic in between.
  - 1718627440 2y ago
    Yes, that is why restrict was introduced.
  - bee_rider 2y ago
    In general, the information as to whether or not the pointers will alias might not even be known, right? Like we could be compiling a library that will take in some pointers from the user.
wayoverthecloud 2y ago
In C++ there is the align_alloc specificier. https://en.cppreference.com/w/c/memory/aligned_alloc https://en.cppreference.com/w/c/memory/aligned_alloc Not sure for C
- ryanpetrich 2y ago
  aligned_alloc is a C function. It doesn't help the compiler prove that two pointers can't alias. restrict is the keyword that does.
  - bobmcnamara 2y ago
    The compiler is allowed to assume the results of malloc and certain other allocation functions do not alias. Restrict does too.
alkh 2y ago
I thought you could simply add `restrict`[1] to indicate that only one pointer points to a specific object? Shouldn't this help? [1]https://en.cppreference.com/w/c/language/restrict https://en.cppreference.com/w/c/language/restrict
- aidenn0 2y ago
  You can add that qualifier, but nothing will enforce that objects don't overlap. So you replaced slow correct code with fast incorrect code.
  - TheNewAndy 2y ago
    If you believed the pointers didn't alias and they actually did then you replaced slow incorrect code with fast incorrect code
    - aidenn0 2y ago
      That doesn't follow? If you replaced every call of memcpy with memmove (to use an example from the standard library), then your program is no less correct than it was before (and possibly more). The converse is that adding "restrict" qualifiers to existing functions can only make your program less correct, not more.
      - ablob 2y ago
        In this context it follows. If you believe the pointers dont alias (and want to use simd) then there should be no data-dependencies inbetween indices. (If there are data-dependencies it is quite likely that you have designed your algorithm to accomodate that, i.e. make it suitable for vectorised access. parallel prefix-sum would be an example of this). And without data-dependencies vectorized code is exactly equal to unvectorized code in all circumstances. If however, you do have data-dependencies, then your algorithm is wrong. Vectorizing can surface this error, while keeping it unvectorized may hide it. In my previous example (parallel prefix-sum) if you had a direct data-dependency to the previous index your program would've been wrong - regardless of vectorization. In short, while your statement is true in general, I believe that it is not applicable in the context of the discussion.
drysine 2y ago
That's what the restrict type qualifier[0] is for. [0] https://en.cppreference.com/w/c/language/restrict https://en.cppreference.com/w/c/language/restrict
bee_rider 2y ago
When compiling a library, it is generally impossible for the compiler to know whether or not the pointers will be aliased, right? That decision is made after the library is already compiled.
- drwu 2y ago
  The function declaration in the header file of the library can carry the required 'restrict'. It works for c++ invokers too, as most c++ compilers also support and check the __restrict__ for old plain pointer types.

adgjlsfhk1 2y ago

> But it then talks about how exp is implemented in hardware, in a way that forces it to be a non-SIMD calculation. This is especially surprising given that very few architectures have an `exp` in hardware. It's almost always done in software.

stabbles 2y ago

I think the interpretation is more conceptual about functions, reusability and composability. The function exp has an abstract mathematical definition as mapping from numbers to numbers, and you could implement that generically if the language allows for it, but in C you cannot because it's bound to the signature `double exp(double)` and fixed like that at compile time. You cannot use this function in a different context and pass e.g. __m256d. Basically because C does not have function dispatch, it's ill-suited for generic programming. You cannot write an algorithm on scalar, and now pass arrays instead.

Someone 2y ago
> You cannot write an algorithm on scalar, and now pass arrays instead. It is limited to cases where you know all overloads beforehand and limited because of C’s weak type system (can’t have an overload for arrays of int and one for pointers to int, for example), but you can. https://en.cppreference.com/w/c/language/generic https://en.cppreference.com/w/c/language/generic: #include <math.h> #include <stdio.h> // Possible implementation of the tgmath.h macro cbrt #define cbrt(X) _Generic((X), \ long double: cbrtl, \ default: cbrt, \ float: cbrtf \ )(X) int main(void) { double x = 8.0; const float y = 3.375; printf("cbrt(8.0) = %f\n", cbrt(x)); // selects the default cbrt printf("cbrtf(3.375) = %f\n", cbrt(y)); // converts const float to float, // then selects cbrtf } It also, IMO, is a bit ugly, but that fits in nice with the rest of C, as seen through modern eyes. There also is a drawback for lots of cases that you cannot overload operators. Implementing vector addition as ‘+’ isn’t possible, for example. Many compilers partially support that as a language extension, though, see for example https://clang.llvm.org/docs/LanguageExtensions.html#vectors-and-extended-vectors https://clang.llvm.org/docs/LanguageExtensions.html#vectors-....
- uecker 2y ago
  I am not sure what you mean by "weak" type system. C's type system is not weak IMHO and also differentiates between arrays and pointers: _Generic((typeof(x), int*: 1, int[]: 2) \_Generic with type argument is a new feature, it is a bit less elegant without it: _Generic((typeof(x)*)0, typeof(int*)*: 1, typeof(int[])\*: 2)

ta3401 2y ago

> But it then talks about how exp is implemented in hardware, in a way that forces it to be a non-SIMD calculation. The fact that exp is implemented in hardware is not the argument. The argument is that exp is a library function, compiled separately, and thus the compiler cannot inline the function and fuse it with an array-wide loop, to detect later on an opportunity to generate the SIMD instructions. It is true however that exp is implemented in hardware in the X86 world, and to be fair, perhaps a C compiler only needs to represent that function as an intrinsic instead, to give itself a chance to later replace it either with the function call or some SIMD instruction; but I guess that the standard doesn't provide that possibility?

AlotOfReading 2y ago
The compiler can (and does) know enough about the stdlib functions to treat them specially. The C standard doesn't require that libm exist as a separately linked library either, that's just an implementation detail from an earlier time that got enshrined in POSIX.
vmchale 2y ago
> perhaps a C compiler only needs to represent that function as an intrinsic instead, to give itself a chance to later replace it either with the function call or some SIMD instruction I gesture to this in the blog post: > In C, one writes a function, and it is exported in an object file. To appreciate why this is special, consider sum :: Num a => [a] -> a in Haskell. This function exists only in the context of GHC. > ... > perhaps there are more fluent methods for compilation (and better data structures for export à la object files).
- ta3401 2y ago
  I now realize that I somewhat rephrased what was said in the blog. How strange!