4 ms·

Erlang figured this out so much you've described the BEAM. You can have hundreds of thousands of small "processes" communicating through BEAM mailboxes on a mod

by ralphc 1y ago

Erlang figured this out so much you've described the BEAM. You can have hundreds of thousands of small "processes" communicating through BEAM mailboxes on a modern machine. Elixir has protocols and macros, improvements on Erlang, and if you really require lisp there is LFE, Lisp Flavored Erlang, that has true lisp homoiconicity.

zackmorris 1y ago
Wow LFE is really cool, thanks for showing me another new thing! Admittedly I got pretty depressed about the lack of progress on this stuff and kinda checked out over the years. So my knowledge has gaps since the 90s. - I'll use this as an excuse to talk about another learning model besides NNs and GAs: Boltzmann Machines (BMs). For anyone unfamiliar with them, they can be thought of as a temperature minimization or gas diffusion model. Rather than gradient descent, simpler formulas like summations and random sampling can be used, so they are "big dumb algorithms" like GAs as opposed to "fickle algorithms" like NNs. They borrow concepts from Markov chains and simulated annealing. I'm having trouble finding a single approachable summary though: https://en.wikipedia.org/wiki/Boltzmann_machine https://en.wikipedia.org/wiki/Boltzmann_machine https://www.geeksforgeeks.org/types-of-boltzmann-machines/ https://www.geeksforgeeks.org/types-of-boltzmann-machines/ https://medium.com/@soumallya160/a-complete-guide-to-boltzmann-machine-deep-learning-7f7ce29b9e09 https://medium.com/@soumallya160/a-complete-guide-to-boltzma... https://blog.paperspace.com/beginners-guide-to-boltzmann-machines-pytorch/ https://blog.paperspace.com/beginners-guide-to-boltzmann-mac... https://deepai.org/machine-learning-glossary-and-terms/boltzmann-machine https://deepai.org/machine-learning-glossary-and-terms/boltz... The main problem is that their fully-connected nature makes them difficult to train beyond a certain size. Restricted Boltzmann Machines (RBMs) try to overcome that, but their increased complexity makes them more like NNs. I think that GAs could be useful for training fully-connected BMs by using evolution to decide which weights to update, similarly to your original question of whether GAs could be used in place of gradient descent. Unfortunately I can't find succinct articles about it because it's pretty fringe: Using GAs to train BMs: http://www.icj-e.org/download/ICJE-5-10-108-116.pdf http://www.icj-e.org/download/ICJE-5-10-108-116.pdf Using BMs to train GAs: https://doc.lagout.org/science/0_Computer%20Science/2_Algorithms/Practical%20Handbook%20of%20GENETIC%20ALGORITHMS%2c%20Volume%20II/ganf5.pdf https://doc.lagout.org/science/0_Computer%20Science/2_Algori... Towards combining GAs with BMs and possibly NNs: https://www.sciencedirect.com/science/article/abs/pii/0375960187901496 https://www.sciencedirect.com/science/article/abs/pii/037596... https://www.sciencedirect.com/science/article/abs/pii/016781919400071H https://www.sciencedirect.com/science/article/abs/pii/016781... Note the dates on those last papers: 1987 and 1995. These are not new ideas. Unfortunately they're paywalled and I can't find online PDFs for them. I think what went wrong with LLMs is their fixation on NNs. The code smell for that is their exorbitant energy cost of training. I'd compare it to the high energy cost of proof-of-work crypto like Bitcoin vs proof-of-stake crypto like Ethereum. Or the high mental load of React vs Vue or especially htmx. A programmer's productivity is proportional to the level of abstraction, in other words how much mental load can be offloaded to the tooling. Which means that programming started regressing when we abandoned web for native mobile apps and doubled down on imperative programming with single-page applications around 2007, probably as a result of the Dot Bomb and lean approaches like Agile which resemble austerity, as opposed to those like Waterfall which resemble central planning. Understandable from a business perspective, but tragic for the cause of pure research and human progress towards achieving self-actualization through freedom from labor. Since efficiency didn't pan out, I see that as a sign that big dumb approaches like GAs deserve further study. My feeling is that all machine learning methods are actually equivalent and interconvertible, like with change of coordinates in math and transpiling. So we should use the conceptually simplest ones to avoid the complexity pitfalls of more complex ones that often lead to poor performance. - To put this in perspective, a neuron can perform about 1000 operations per second, or 1/1000 mips (megaflops), which is about as powerful as ENIAC: https://jetpress.org/volume1/moravec.htm https://jetpress.org/volume1/moravec.htm Our mind is about 100 billion neurons that have the potential to connect to others very far away. So a 100 million megaflops or 100 teraflops (10^14 flops) computer should be able to simulate aspects of the human brain. We just reached 1 exaflops (10^18 flops) supercomputers 3 years ago, which is about 10,000 times faster than needed: https://en.wikipedia.org/wiki/List_of_fastest_computers https://en.wikipedia.org/wiki/List_of_fastest_computers So something has gone terribly wrong with the way we utilize computing power. I blame it on our fixation with narrow single-threaded performance leading to an inability to see outside-the-box solutions. For a tiny fraction of the effort we've expended, we could have built wide multi-threaded CPUs like the ones I mentioned, that more naturally handle the parallel computation of GAs and the human brain. This is analogous to the problems we're facing when 1 billionaire has the wealth and resources of 1 million people. Even if they have an IQ of 200, they're still orders of magnitude less effective than all of those people solving a problem in a unified fashion. That's why systems-level problems like infant mortality, mass-starvation, global warming, etc aren't being solved. When we look deep enough, we start to see that everything is connected, and that it's impossible to talk about technological progress without political progress. But I digress. - Where I'm going with this is that whatever machine learning algorithm we settle on will have an underlying concept like evolution at its core. Each neuron is doing the best it can to find connection with the others and minimize its workload to maximize the resources available to it. Just like ants and fungi and human beings. I believe that this evolution away from entropy towards complex structure is the essence of life, and that instinct, awareness, feeling and meaning - along with problem-solving, thinking, doing and reason - are two sides of the same coin. Currently tech mainly addresses the right (masculine?) side of that coin, while the left (feminine?) side was barely touched on by companies like Apple and Atari early on. I mean masculine/feminine in the divine sense in all of us, not gender. Now we're immersed in a world of technological magic with virtually no understanding of our magical nature or how to wield that power responsibly. Basically we're so distracted that we don't realize that we're divine beings with the power to change our outer reality through manifestation by being mindful of where we place our attention in our inner reality. That quantum effects bubble up into the real world like the butterfly effect, how life "finds a way" and subtly shifts probability to create outcomes favorable to it. I believe that if we mimick this quantum-determinism bridge using highly-parallel processors running on the order of 100 billion (10^11) threads that machines will acquire consciousness. Because at some level it won't matter if the smallest components are made of carbon or silicon. In other words, artificial intelligence can be achieved by brute forcing sequential computing power (intellect/knowledge). But artificial consciousness can be achieved by brute forcing parallel computing power (intuition/wisdom). This is the piece of the puzzle that's missing with LLMs and most other competing approaches. A GPU is basically a hugely wide processor running in the low hundreds of threads. It may never reach a level that could be mistaken for consciousness, because it can't explore a large number of potential solutions simultaneously. Unless we build transputers from FPGAs or find a way to transpile SISD and MIMD into SIMD to truly run billions of isolated threads with something like the BEAM/LFE you mentioned. On that note, we also need advances in programming languages to play with this sort of intelligent machine. There are countless examples of running lisp in C, but almost no examples of running C in lisp. That's a huge problem, because the real world is imperative. We need to be able to tell computers our problems and the solutions we need using familar formula-style syntax like C, but have the computer work with lisp's icode tree internally like a spreadsheet so that it doesn't get lost in the weeds. This is where I would start, if I had the resources to do so. I'd write a functional imperative language that uses all const variables so that all logic is free of side effects and can be statically analyzed so that it translates directly to lisp and back, avoiding the complexities of borrow checkers like in Rust. But it would have every convenience method we've come to expect from languages like PHP and Javascript so that we can get real work done. And be decomposable into as many concurrent threads as possible from the start using a divide and conquer strategy, so that even uninspired/unoptimized code would run thousands or millions of times faster than the languages we're used to. And it would have a JIT to transpile most other languages to/from itself to create a functional/imperative bridge. Then I'd write a runtime for GPUs to run those isolated threads. I don't know if this is possible, but if it isn't, that might create demand for true multicore CPUs. Well I've completely gotten off topic and revealed some of my most heartfelt dreams here, and I doubt that anyone will read this. I wish I could be as concise as you, because you connected a lot of insights from so few words.
- zackmorris 1y ago
  I've blabbered on about this too much, but I decided to see if there's been any progress on multithreading libraries for GPU and found some for lisp: https://repository.lboro.ac.uk/articles/conference_contribution/And_now_for_something_completely_different_running_Lisp_on_GPUs/9404783 https://repository.lboro.ac.uk/articles/conference_contribut... (has pdf download) https://github.com/michelp/hillisp https://github.com/michelp/hillisp Unfortunately neither of these seem to do any heavy lifting, they just transpile lisp statements into GPU kernels like WebMonkey does. But it's a start. To get real work done in concurrency with SISD/MIMD to SIMD transpiling, we'd need a small team with a modest budget to work for perhaps 2 years on figuring out how to statically analyze unmodified lisp programs and break them down into independent fragments that can execute in parallel without side effects. Then use libraries like these to run those fragments as kernels and aggregate the results. Conceptually this would work like a spreadsheet, watching for changes in inputs and calculating the result across as many cores as possible unsupervised. Once that runtime reached MVP, work could begin on transpiling common languages like C and Javascript to lisp. This is also an open problem, but straightforward to solve. The difficulty appears to be that transpiling imperative code to lisp has only been done as a hack with impure or monadic logic that defeats the whole purpose of using functional programming. It also tends to lose human readability by using too much glue code through bindings, rather than naming variables common to both runtimes like how we name registers in C++ that can be used within asm() blocks: https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables....
  - zackmorris 1y ago
    I finally found a tool to apparently compile standard C++ CPU code to GPU code without #pragma, language extensions, etc called NVC++, ironically from Nvidia: https://developer.nvidia.com/blog/accelerating-standard-c-with-gpus-using-stdpar/ https://developer.nvidia.com/blog/accelerating-standard-c-wi... NVC++ can compile Standard C++ algorithms with the parallel execution policies std::execution::par or std::execution::par_unseq for execution on NVIDIA GPUs. An NVC++ command-line option, -stdpar, is used to enable GPU-accelerated C++ Parallel Algorithms. Lambdas, including generic lambdas, are fully supported in parallel algorithm invocations. No language extensions or non-standard libraries are required to enable GPU acceleration. All data movement between host memory and GPU device memory is performed implicitly and automatically under the control of CUDA Unified Memory. Unfortunately they appear to have implemented it wrong by requiring that std::() methods have std::execution::par* or std::execution::par_unseq as their first argument. I can't tell if it's their fault or the fault of the C++ standards committee for choosing to do it that way, rather than enabling par and par_unseq application-wide. It's probably due to C++'s emphasis on mutability, so to really do it right, we'd need a way to enforce const variables and pass-by-value everywhere, which just isn't feasible with C++. This is why I consider it a dead-end language and abandoned it around 2010. At least we can use copy-by-value with lambdas manually: In the earlier example, the function parameter a is captured by reference. The code within the body of the lambda, which is running on the GPU, tries to access a, which is in the CPU stack memory. This results in a memory violation and undefined behavior. In this case, the problem can easily be fixed by changing the lambda to capture by value: void saxpy(float* x, float* y, int N, float a) { std::transform(std::execution::par_unseq, x, x + N, y, y, [=](float xi, float yi){ return a * xi + yi; }); } With this one-character change, the lambda makes a copy of a, which is then copied to the GPU, and there are no attempts to reference CPU stack memory from GPU code. PHP uses copy-on-write to pass arrays by value, and is one of the only languages to get that right. Clojure uses reduction (apologies if this metaphor is wrong) for its state management, a bit like Redux, and is one of the only languages to get that right as well. So it may be possible to compile them with NVC++ somehow but that would probably take a lot of finesse. A bit of cleverness that might redeem NVC++ is function objects: Function pointers can’t be passed to C++ Parallel Algorithms to be run on the GPU, and functions may not be called through a function pointer within GPU code. For example, the following code example won’t work correctly: void square(int& x) { x = x * x; } void square_all(std::vector<int>& v) { std::for_each(std::execution::par_unseq, v.begin(), v.end(), &square); } It passes a pointer to the CPU version of the function square to a parallel for_each algorithm invocation. When the algorithm is parallelized and offloaded to the GPU, the program fails to resolve the function pointer to the GPU version of square. You can often solve this issue by using a function object, which is an object with a function call operator. The function object’s call operator is resolved at compile time to the GPU version of the function, instead of being resolved at run time to the incorrect CPU version of the function as in the previous example. For example, the following code example works: struct squared { void operator()(int& x) const { x = x * x; } }; void square_all(std::vector<int>& v) { std::for_each(std::execution::par_unseq, v.begin(), v.end(), squared{}); } Assuming this stuff "just works", I'd expect an order of magnitude speedup per nested loop, due to having to copy memory CPU<->GPU which exacerbates Amdahl's Law. I usually use an order somewhere between 2 < e (2.71828) < 10 and find 3 to be a good rule of thumb. So ~3x speedup for a loop, ~10x for a nested loop, and most software doesn't nest deeper than that. So a far cry from the 1000+x speedup that the number of CUDA cores suggests (6912 in the A100). A bit like the ~2x speedup seen by GPU database extensions: https://news.ycombinator.com/item?id=10151632 https://news.ycombinator.com/item?id=10151632 https://news.ycombinator.com/item?id=43964505 https://news.ycombinator.com/item?id=43964505 Unfortunately NVC++ requires independent thread scheduling but Nvidia took down the link explaining it in https://developer.nvidia.com/blog/inside-volta/ https://developer.nvidia.com/blog/inside-volta/ and I had to find it on Internet Archive: https://web.archive.org/web/20241102165838/https://developer.nvidia.com/blog/inside-volta/ https://web.archive.org/web/20241102165838/https://developer... https://en.wikipedia.org/wiki/Pascal_(microarchitecture) https://en.wikipedia.org/wiki/Pascal_(microarchitecture) (limited support for NVC++) https://en.wikipedia.org/wiki/Volta_(microarchitecture) https://en.wikipedia.org/wiki/Volta_(microarchitecture) https://en.wikipedia.org/wiki/Turing_(microarchitecture) https://en.wikipedia.org/wiki/Turing_(microarchitecture) https://en.wikipedia.org/wiki/Ampere_(microarchitecture) https://en.wikipedia.org/wiki/Ampere_(microarchitecture) (A100) It's been hard for me to watch this play out over the decades, because GPUs like the A100 are approaching the "big dumb" multicore trajectory we were on in the 1990s on a random walk. NVC++ requires CUDA 10.1. I asked Google AI and it looks like macOS stopped at CUDA 10.2 which ended after macOS 10.13 (High Sierra). https://umatechnology.org/nvidia-confirms-macos-will-no-longer-support-developing-cuda-apps/ https://umatechnology.org/nvidia-confirms-macos-will-no-long... So with the status quo divided on the future of GPUs and all attention turning towards LLMs/TPUs/low-precision-floats, I'd predict little movement on this independent thread scheduling trend until around 2030. Which is a tragedy on one hand but a unique opportunity for someone with resources who sees what I'm getting at. I may attempt to write a flavor of C/C++ having const everywhere and copy-on-write using macros/preprocessing or by transpiling it to C/C++ for NVC++. I don't know how it would block use of references and pointers though. This still avoids the real work of statically analyzing the abstract syntax tree's intermediate code (icode) and running it in parallel at the theoretical limit. That's when we get back on the Moore's Law trendline and see a 100x speedup of CPU code per decade, which stopped around 2007 when low-cost/low-power mobile applications took the market.

by ralphc 1y ago

zackmorris 1y ago

Wow LFE is really cool, thanks for showing me another new thing! Admittedly I got pretty depressed about the lack of progress on this stuff and kinda checked out over the years. So my knowledge has gaps since the 90s. - I'll use this as an excuse to talk about another learning model besides NNs and GAs: Boltzmann Machines (BMs). For anyone unfamiliar with them, they can be thought of as a temperature minimization or gas diffusion model. Rather than gradient descent, simpler formulas like summations and random sampling can be used, so they are "big dumb algorithms" like GAs as opposed to "fickle algorithms" like NNs. They borrow concepts from Markov chains and simulated annealing. I'm having trouble finding a single approachable summary though: https://en.wikipedia.org/wiki/Boltzmann_machine https://en.wikipedia.org/wiki/Boltzmann_machine https://www.geeksforgeeks.org/types-of-boltzmann-machines/ https://www.geeksforgeeks.org/types-of-boltzmann-machines/ https://medium.com/@soumallya160/a-complete-guide-to-boltzmann-machine-deep-learning-7f7ce29b9e09 https://medium.com/@soumallya160/a-complete-guide-to-boltzma... https://blog.paperspace.com/beginners-guide-to-boltzmann-machines-pytorch/ https://blog.paperspace.com/beginners-guide-to-boltzmann-mac... https://deepai.org/machine-learning-glossary-and-terms/boltzmann-machine https://deepai.org/machine-learning-glossary-and-terms/boltz... The main problem is that their fully-connected nature makes them difficult to train beyond a certain size. Restricted Boltzmann Machines (RBMs) try to overcome that, but their increased complexity makes them more like NNs. I think that GAs could be useful for training fully-connected BMs by using evolution to decide which weights to update, similarly to your original question of whether GAs could be used in place of gradient descent. Unfortunately I can't find succinct articles about it because it's pretty fringe: Using GAs to train BMs: http://www.icj-e.org/download/ICJE-5-10-108-116.pdf http://www.icj-e.org/download/ICJE-5-10-108-116.pdf Using BMs to train GAs: https://doc.lagout.org/science/0_Computer%20Science/2_Algorithms/Practical%20Handbook%20of%20GENETIC%20ALGORITHMS%2c%20Volume%20II/ganf5.pdf https://doc.lagout.org/science/0_Computer%20Science/2_Algori... Towards combining GAs with BMs and possibly NNs: https://www.sciencedirect.com/science/article/abs/pii/0375960187901496 https://www.sciencedirect.com/science/article/abs/pii/037596... https://www.sciencedirect.com/science/article/abs/pii/016781919400071H https://www.sciencedirect.com/science/article/abs/pii/016781... Note the dates on those last papers: 1987 and 1995. These are not new ideas. Unfortunately they're paywalled and I can't find online PDFs for them. I think what went wrong with LLMs is their fixation on NNs. The code smell for that is their exorbitant energy cost of training. I'd compare it to the high energy cost of proof-of-work crypto like Bitcoin vs proof-of-stake crypto like Ethereum. Or the high mental load of React vs Vue or especially htmx. A programmer's productivity is proportional to the level of abstraction, in other words how much mental load can be offloaded to the tooling. Which means that programming started regressing when we abandoned web for native mobile apps and doubled down on imperative programming with single-page applications around 2007, probably as a result of the Dot Bomb and lean approaches like Agile which resemble austerity, as opposed to those like Waterfall which resemble central planning. Understandable from a business perspective, but tragic for the cause of pure research and human progress towards achieving self-actualization through freedom from labor. Since efficiency didn't pan out, I see that as a sign that big dumb approaches like GAs deserve further study. My feeling is that all machine learning methods are actually equivalent and interconvertible, like with change of coordinates in math and transpiling. So we should use the conceptually simplest ones to avoid the complexity pitfalls of more complex ones that often lead to poor performance. - To put this in perspective, a neuron can perform about 1000 operations per second, or 1/1000 mips (megaflops), which is about as powerful as ENIAC: https://jetpress.org/volume1/moravec.htm https://jetpress.org/volume1/moravec.htm Our mind is about 100 billion neurons that have the potential to connect to others very far away. So a 100 million megaflops or 100 teraflops (10^14 flops) computer should be able to simulate aspects of the human brain. We just reached 1 exaflops (10^18 flops) supercomputers 3 years ago, which is about 10,000 times faster than needed: https://en.wikipedia.org/wiki/List_of_fastest_computers https://en.wikipedia.org/wiki/List_of_fastest_computers So something has gone terribly wrong with the way we utilize computing power. I blame it on our fixation with narrow single-threaded performance leading to an inability to see outside-the-box solutions. For a tiny fraction of the effort we've expended, we could have built wide multi-threaded CPUs like the ones I mentioned, that more naturally handle the parallel computation of GAs and the human brain. This is analogous to the problems we're facing when 1 billionaire has the wealth and resources of 1 million people. Even if they have an IQ of 200, they're still orders of magnitude less effective than all of those people solving a problem in a unified fashion. That's why systems-level problems like infant mortality, mass-starvation, global warming, etc aren't being solved. When we look deep enough, we start to see that everything is connected, and that it's impossible to talk about technological progress without political progress. But I digress. - Where I'm going with this is that whatever machine learning algorithm we settle on will have an underlying concept like evolution at its core. Each neuron is doing the best it can to find connection with the others and minimize its workload to maximize the resources available to it. Just like ants and fungi and human beings. I believe that this evolution away from entropy towards complex structure is the essence of life, and that instinct, awareness, feeling and meaning - along with problem-solving, thinking, doing and reason - are two sides of the same coin. Currently tech mainly addresses the right (masculine?) side of that coin, while the left (feminine?) side was barely touched on by companies like Apple and Atari early on. I mean masculine/feminine in the divine sense in all of us, not gender. Now we're immersed in a world of technological magic with virtually no understanding of our magical nature or how to wield that power responsibly. Basically we're so distracted that we don't realize that we're divine beings with the power to change our outer reality through manifestation by being mindful of where we place our attention in our inner reality. That quantum effects bubble up into the real world like the butterfly effect, how life "finds a way" and subtly shifts probability to create outcomes favorable to it. I believe that if we mimick this quantum-determinism bridge using highly-parallel processors running on the order of 100 billion (10^11) threads that machines will acquire consciousness. Because at some level it won't matter if the smallest components are made of carbon or silicon. In other words, artificial intelligence can be achieved by brute forcing sequential computing power (intellect/knowledge). But artificial consciousness can be achieved by brute forcing parallel computing power (intuition/wisdom). This is the piece of the puzzle that's missing with LLMs and most other competing approaches. A GPU is basically a hugely wide processor running in the low hundreds of threads. It may never reach a level that could be mistaken for consciousness, because it can't explore a large number of potential solutions simultaneously. Unless we build transputers from FPGAs or find a way to transpile SISD and MIMD into SIMD to truly run billions of isolated threads with something like the BEAM/LFE you mentioned. On that note, we also need advances in programming languages to play with this sort of intelligent machine. There are countless examples of running lisp in C, but almost no examples of running C in lisp. That's a huge problem, because the real world is imperative. We need to be able to tell computers our problems and the solutions we need using familar formula-style syntax like C, but have the computer work with lisp's icode tree internally like a spreadsheet so that it doesn't get lost in the weeds. This is where I would start, if I had the resources to do so. I'd write a functional imperative language that uses all const variables so that all logic is free of side effects and can be statically analyzed so that it translates directly to lisp and back, avoiding the complexities of borrow checkers like in Rust. But it would have every convenience method we've come to expect from languages like PHP and Javascript so that we can get real work done. And be decomposable into as many concurrent threads as possible from the start using a divide and conquer strategy, so that even uninspired/unoptimized code would run thousands or millions of times faster than the languages we're used to. And it would have a JIT to transpile most other languages to/from itself to create a functional/imperative bridge. Then I'd write a runtime for GPUs to run those isolated threads. I don't know if this is possible, but if it isn't, that might create demand for true multicore CPUs. Well I've completely gotten off topic and revealed some of my most heartfelt dreams here, and I doubt that anyone will read this. I wish I could be as concise as you, because you connected a lot of insights from so few words.

zackmorris 1y ago
I've blabbered on about this too much, but I decided to see if there's been any progress on multithreading libraries for GPU and found some for lisp: https://repository.lboro.ac.uk/articles/conference_contribution/And_now_for_something_completely_different_running_Lisp_on_GPUs/9404783 https://repository.lboro.ac.uk/articles/conference_contribut... (has pdf download) https://github.com/michelp/hillisp https://github.com/michelp/hillisp Unfortunately neither of these seem to do any heavy lifting, they just transpile lisp statements into GPU kernels like WebMonkey does. But it's a start. To get real work done in concurrency with SISD/MIMD to SIMD transpiling, we'd need a small team with a modest budget to work for perhaps 2 years on figuring out how to statically analyze unmodified lisp programs and break them down into independent fragments that can execute in parallel without side effects. Then use libraries like these to run those fragments as kernels and aggregate the results. Conceptually this would work like a spreadsheet, watching for changes in inputs and calculating the result across as many cores as possible unsupervised. Once that runtime reached MVP, work could begin on transpiling common languages like C and Javascript to lisp. This is also an open problem, but straightforward to solve. The difficulty appears to be that transpiling imperative code to lisp has only been done as a hack with impure or monadic logic that defeats the whole purpose of using functional programming. It also tends to lose human readability by using too much glue code through bindings, rather than naming variables common to both runtimes like how we name registers in C++ that can be used within asm() blocks: https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables....
- zackmorris 1y ago
  I finally found a tool to apparently compile standard C++ CPU code to GPU code without #pragma, language extensions, etc called NVC++, ironically from Nvidia: https://developer.nvidia.com/blog/accelerating-standard-c-with-gpus-using-stdpar/ https://developer.nvidia.com/blog/accelerating-standard-c-wi... NVC++ can compile Standard C++ algorithms with the parallel execution policies std::execution::par or std::execution::par_unseq for execution on NVIDIA GPUs. An NVC++ command-line option, -stdpar, is used to enable GPU-accelerated C++ Parallel Algorithms. Lambdas, including generic lambdas, are fully supported in parallel algorithm invocations. No language extensions or non-standard libraries are required to enable GPU acceleration. All data movement between host memory and GPU device memory is performed implicitly and automatically under the control of CUDA Unified Memory. Unfortunately they appear to have implemented it wrong by requiring that std::() methods have std::execution::par* or std::execution::par_unseq as their first argument. I can't tell if it's their fault or the fault of the C++ standards committee for choosing to do it that way, rather than enabling par and par_unseq application-wide. It's probably due to C++'s emphasis on mutability, so to really do it right, we'd need a way to enforce const variables and pass-by-value everywhere, which just isn't feasible with C++. This is why I consider it a dead-end language and abandoned it around 2010. At least we can use copy-by-value with lambdas manually: In the earlier example, the function parameter a is captured by reference. The code within the body of the lambda, which is running on the GPU, tries to access a, which is in the CPU stack memory. This results in a memory violation and undefined behavior. In this case, the problem can easily be fixed by changing the lambda to capture by value: void saxpy(float* x, float* y, int N, float a) { std::transform(std::execution::par_unseq, x, x + N, y, y, [=](float xi, float yi){ return a * xi + yi; }); } With this one-character change, the lambda makes a copy of a, which is then copied to the GPU, and there are no attempts to reference CPU stack memory from GPU code. PHP uses copy-on-write to pass arrays by value, and is one of the only languages to get that right. Clojure uses reduction (apologies if this metaphor is wrong) for its state management, a bit like Redux, and is one of the only languages to get that right as well. So it may be possible to compile them with NVC++ somehow but that would probably take a lot of finesse. A bit of cleverness that might redeem NVC++ is function objects: Function pointers can’t be passed to C++ Parallel Algorithms to be run on the GPU, and functions may not be called through a function pointer within GPU code. For example, the following code example won’t work correctly: void square(int& x) { x = x * x; } void square_all(std::vector<int>& v) { std::for_each(std::execution::par_unseq, v.begin(), v.end(), &square); } It passes a pointer to the CPU version of the function square to a parallel for_each algorithm invocation. When the algorithm is parallelized and offloaded to the GPU, the program fails to resolve the function pointer to the GPU version of square. You can often solve this issue by using a function object, which is an object with a function call operator. The function object’s call operator is resolved at compile time to the GPU version of the function, instead of being resolved at run time to the incorrect CPU version of the function as in the previous example. For example, the following code example works: struct squared { void operator()(int& x) const { x = x * x; } }; void square_all(std::vector<int>& v) { std::for_each(std::execution::par_unseq, v.begin(), v.end(), squared{}); } Assuming this stuff "just works", I'd expect an order of magnitude speedup per nested loop, due to having to copy memory CPU<->GPU which exacerbates Amdahl's Law. I usually use an order somewhere between 2 < e (2.71828) < 10 and find 3 to be a good rule of thumb. So ~3x speedup for a loop, ~10x for a nested loop, and most software doesn't nest deeper than that. So a far cry from the 1000+x speedup that the number of CUDA cores suggests (6912 in the A100). A bit like the ~2x speedup seen by GPU database extensions: https://news.ycombinator.com/item?id=10151632 https://news.ycombinator.com/item?id=10151632 https://news.ycombinator.com/item?id=43964505 https://news.ycombinator.com/item?id=43964505 Unfortunately NVC++ requires independent thread scheduling but Nvidia took down the link explaining it in https://developer.nvidia.com/blog/inside-volta/ https://developer.nvidia.com/blog/inside-volta/ and I had to find it on Internet Archive: https://web.archive.org/web/20241102165838/https://developer.nvidia.com/blog/inside-volta/ https://web.archive.org/web/20241102165838/https://developer... https://en.wikipedia.org/wiki/Pascal_(microarchitecture) https://en.wikipedia.org/wiki/Pascal_(microarchitecture) (limited support for NVC++) https://en.wikipedia.org/wiki/Volta_(microarchitecture) https://en.wikipedia.org/wiki/Volta_(microarchitecture) https://en.wikipedia.org/wiki/Turing_(microarchitecture) https://en.wikipedia.org/wiki/Turing_(microarchitecture) https://en.wikipedia.org/wiki/Ampere_(microarchitecture) https://en.wikipedia.org/wiki/Ampere_(microarchitecture) (A100) It's been hard for me to watch this play out over the decades, because GPUs like the A100 are approaching the "big dumb" multicore trajectory we were on in the 1990s on a random walk. NVC++ requires CUDA 10.1. I asked Google AI and it looks like macOS stopped at CUDA 10.2 which ended after macOS 10.13 (High Sierra). https://umatechnology.org/nvidia-confirms-macos-will-no-longer-support-developing-cuda-apps/ https://umatechnology.org/nvidia-confirms-macos-will-no-long... So with the status quo divided on the future of GPUs and all attention turning towards LLMs/TPUs/low-precision-floats, I'd predict little movement on this independent thread scheduling trend until around 2030. Which is a tragedy on one hand but a unique opportunity for someone with resources who sees what I'm getting at. I may attempt to write a flavor of C/C++ having const everywhere and copy-on-write using macros/preprocessing or by transpiling it to C/C++ for NVC++. I don't know how it would block use of references and pointers though. This still avoids the real work of statically analyzing the abstract syntax tree's intermediate code (icode) and running it in parallel at the theoretical limit. That's when we get back on the Moore's Law trendline and see a 100x speedup of CPU code per decade, which stopped around 2007 when low-cost/low-power mobile applications took the market.