I finally found a tool to apparently compile standard C++ CPU code to GPU code without #pragma, language extensions, etc called NVC++, ironically from Nvidia:

I finally found a tool to apparently compile standard C++ CPU code to GPU code without #pragma, language extensions, etc called NVC++, ironically from Nvidia: https://developer.nvidia.com/blog/accelerating-standard-c-with-gpus-using-stdpar/ https://developer.nvidia.com/blog/accelerating-standard-c-wi... NVC++ can compile Standard C++ algorithms with the parallel execution policies std::execution::par or std::execution::par_unseq for execution on NVIDIA GPUs. An NVC++ command-line option, -stdpar, is used to enable GPU-accelerated C++ Parallel Algorithms. Lambdas, including generic lambdas, are fully supported in parallel algorithm invocations. No language extensions or non-standard libraries are required to enable GPU acceleration. All data movement between host memory and GPU device memory is performed implicitly and automatically under the control of CUDA Unified Memory. Unfortunately they appear to have implemented it wrong by requiring that std::() methods have std::execution::par* or std::execution::par_unseq as their first argument. I can't tell if it's their fault or the fault of the C++ standards committee for choosing to do it that way, rather than enabling par and par_unseq application-wide. It's probably due to C++'s emphasis on mutability, so to really do it right, we'd need a way to enforce const variables and pass-by-value everywhere, which just isn't feasible with C++. This is why I consider it a dead-end language and abandoned it around 2010. At least we can use copy-by-value with lambdas manually: In the earlier example, the function parameter a is captured by reference. The code within the body of the lambda, which is running on the GPU, tries to access a, which is in the CPU stack memory. This results in a memory violation and undefined behavior. In this case, the problem can easily be fixed by changing the lambda to capture by value: void saxpy(float* x, float* y, int N, float a) { std::transform(std::execution::par_unseq, x, x + N, y, y, [=](float xi, float yi){ return a * xi + yi; }); } With this one-character change, the lambda makes a copy of a, which is then copied to the GPU, and there are no attempts to reference CPU stack memory from GPU code. PHP uses copy-on-write to pass arrays by value, and is one of the only languages to get that right. Clojure uses reduction (apologies if this metaphor is wrong) for its state management, a bit like Redux, and is one of the only languages to get that right as well. So it may be possible to compile them with NVC++ somehow but that would probably take a lot of finesse. A bit of cleverness that might redeem NVC++ is function objects: Function pointers can’t be passed to C++ Parallel Algorithms to be run on the GPU, and functions may not be called through a function pointer within GPU code. For example, the following code example won’t work correctly: void square(int& x) { x = x * x; } void square_all(std::vector<int>& v) { std::for_each(std::execution::par_unseq, v.begin(), v.end(), &square); } It passes a pointer to the CPU version of the function square to a parallel for_each algorithm invocation. When the algorithm is parallelized and offloaded to the GPU, the program fails to resolve the function pointer to the GPU version of square. You can often solve this issue by using a function object, which is an object with a function call operator. The function object’s call operator is resolved at compile time to the GPU version of the function, instead of being resolved at run time to the incorrect CPU version of the function as in the previous example. For example, the following code example works: struct squared { void operator()(int& x) const { x = x * x; } }; void square_all(std::vector<int>& v) { std::for_each(std::execution::par_unseq, v.begin(), v.end(), squared{}); } Assuming this stuff "just works", I'd expect an order of magnitude speedup per nested loop, due to having to copy memory CPU<->GPU which exacerbates Amdahl's Law. I usually use an order somewhere between 2 < e (2.71828) < 10 and find 3 to be a good rule of thumb. So ~3x speedup for a loop, ~10x for a nested loop, and most software doesn't nest deeper than that. So a far cry from the 1000+x speedup that the number of CUDA cores suggests (6912 in the A100). A bit like the ~2x speedup seen by GPU database extensions: https://news.ycombinator.com/item?id=10151632 https://news.ycombinator.com/item?id=10151632 https://news.ycombinator.com/item?id=43964505 https://news.ycombinator.com/item?id=43964505 Unfortunately NVC++ requires independent thread scheduling but Nvidia took down the link explaining it in https://developer.nvidia.com/blog/inside-volta/ https://developer.nvidia.com/blog/inside-volta/ and I had to find it on Internet Archive: https://web.archive.org/web/20241102165838/https://developer.nvidia.com/blog/inside-volta/ https://web.archive.org/web/20241102165838/https://developer... https://en.wikipedia.org/wiki/Pascal_(microarchitecture) https://en.wikipedia.org/wiki/Pascal_(microarchitecture) (limited support for NVC++) https://en.wikipedia.org/wiki/Volta_(microarchitecture) https://en.wikipedia.org/wiki/Volta_(microarchitecture) https://en.wikipedia.org/wiki/Turing_(microarchitecture) https://en.wikipedia.org/wiki/Turing_(microarchitecture) https://en.wikipedia.org/wiki/Ampere_(microarchitecture) https://en.wikipedia.org/wiki/Ampere_(microarchitecture) (A100) It's been hard for me to watch this play out over the decades, because GPUs like the A100 are approaching the "big dumb" multicore trajectory we were on in the 1990s on a random walk. NVC++ requires CUDA 10.1. I asked Google AI and it looks like macOS stopped at CUDA 10.2 which ended after macOS 10.13 (High Sierra). https://umatechnology.org/nvidia-confirms-macos-will-no-longer-support-developing-cuda-apps/ https://umatechnology.org/nvidia-confirms-macos-will-no-long... So with the status quo divided on the future of GPUs and all attention turning towards LLMs/TPUs/low-precision-floats, I'd predict little movement on this independent thread scheduling trend until around 2030. Which is a tragedy on one hand but a unique opportunity for someone with resources who sees what I'm getting at. I may attempt to write a flavor of C/C++ having const everywhere and copy-on-write using macros/preprocessing or by transpiling it to C/C++ for NVC++. I don't know how it would block use of references and pointers though. This still avoids the real work of statically analyzing the abstract syntax tree's intermediate code (icode) and running it in parallel at the theoretical limit. That's when we get back on the Moore's Law trendline and see a 100x speedup of CPU code per decade, which stopped around 2007 when low-cost/low-power mobile applications took the market.