4 ms·

This is cool but doesn't actually do any heavy lifting, because it runs GLSL 1.0 code directly instead of transpiling Javascript to GLSL internally. Does anyon

This is cool but doesn't actually do any heavy lifting, because it runs GLSL 1.0 code directly instead of transpiling Javascript to GLSL internally. Does anyone know of a Javascript to GLSL transpiler? My interest in this is that the world abandoned true multicore processing 30 years ago around 1995 when 3D video cards went mainstream. Had it not done that, we could have continued with Moore's law and had roughly 100-1000 CPU cores per billion transistors, along with local memories and data-driven processing using hash trees and copy-on-write provided invisibly by the runtime or even in microcode so that we wouldn't have to worry about caching. Apple's M series is the only mainstream CPU I know of that is attempting to do anything close to this, albeit poorly by still having GPU and AI cores instead of emulating single-instruction-multiple-data (SIMD) with multicore. So I've given up on the world ever offering a 1000+ core CPU for under $1000, even though it would be straightforward to design and build today. The closest approximation would be some kind of multiple-instruction-multiple-data (MIMD) transpiler that converts ordinary C-style code to something like GLSL without intrinsics, pragmas, compiler-hints, annotations, etc. In practice, that would look like simple for-loops and other conditionals being statically analyzed to detect codepaths free of side effects and auto-parallelize them for a GPU. We would never deal with SIMD or copying buffers to/from VRAM directly. The code would probably end up looking like GNU Octave, MATLAB or Julia, but we could also use stuff like scatter-gather arrays and higher-order methods like map reduce, or even green threads. Vanilla fork/join code could potentially run thousands of times faster on GPU than CPU if implemented properly. The other reason I'm so interested in this is that GPUs can't easily do genetic programming with thousands of agents acting and evolving independently in a virtual world. So we're missing out on the dozen or so other approaches to AI which are getting overshadowed by LLMs. I would compare the current situation to using React without knowing how simple the HTTP form submit model was in the 1990s, which used declarative programming and idempotent operations to avoid build processes and the imperative hell we've found ourselves in. We're all doing it the hard way with our bare hands and I don't understand why.

ralphc 1y ago
Would your 1000 core CPU do well on neuroevolution?
- zackmorris 1y ago
  Thank you, believe it or not I hadn't heard that word so it filled in a piece of the puzzle for me. https://en.wikipedia.org/wiki/Neuroevolution https://en.wikipedia.org/wiki/Neuroevolution https://medium.com/@roopal.tatiwar20/neuroevolution-evolving-neural-network-with-genetic-algorithms-8ca2165ad04c https://medium.com/@roopal.tatiwar20/neuroevolution-evolving... That was the state of the art when I got my ECE degree in 1999. They were using genetic algorithms (GAs) to evolve initial weights for neural net (NN) gradient descent right before the Dot Bomb. Then pretty much all R&D was cut/abandoned overnight, and the 2000s went to offshoring jobs and shuttering factories which led to the Housing Bubble popping. IMHO that set AI research back at least 10 years, maybe 20, at least in the US. I know it derailed my career dreams. I feel that GPUs by their SIMD nature do poorly with a large number of concurrent processes. For example, a population of GA agents running lisp with each instruction of the intermediate code (icode) tree encoded as bits in a bitfield which evolves using techniques like mutation, crossover and sexual reproduction. Some of those bits represent conditional branching. Ideally we'd want the opposite of a GPU, more like a transputer, on the order of thousands of independent arithmetic logic units (ALUs) with their own local memories, and perhaps able to use custom instructions. That way each lisp tree can execute concurrently in isolation. There may be branch-avoidance transformations from shaders that could help with this, to transpile the von Neumann SISD and MIMD code we're used to into the narrower SIMD so it can run on GPUs. Companies like Nvidia aren't touching this for some reason, and I don't know if it's just a blind spot or deliberate. This is one reason why I wanted FPGAs to take off, since they'd make great transputers. Unfortunately they stagnated due to proprietary compilers and a lack of profitable applications. Today it looks like microcontroller units (MCUs) might take their place, but they're still at least 4 orders of magnitude or more too small and 100 times too slow to be cost-effective. Imagine having a 1 GHz CPU that can run 1000 concurrent UNIX processes running lisp that only communicate through sockets. It's easy to see how mutating them is trivial, then running them as unit tests to see which ones pass is also straightforward. Erlang and Go figured that out years ago. It's so easy in fact to "just come up with the answer" this way, that I think this technique has been suppressed. With that foundation and removing the element of time, it's easy to see that this could be a drop-in replacement for gradient descent. Then the work shifts to curating large data and training sets. I suspect that this is the real work of AI research, and that the learning technique doesn't matter so much. I think of it as: GAs are good for getting to a local minimum error - finding a known-good state. NNs are better for hill climbing - exploring the local solution space. Another way to say that is, corporations spent billions of dollars to train the first LLM MVPs with gradient descent and other "hands on" techniques. When maybe they could have spent thousands of dollars if they had used GAs instead and "let go". Now that we're here, refinement is more important, so gradient descent is here to stay. Although I think of LLM blueprints as eventually fitting on a napkin and running on one chip. Then we'll orchestrate large numbers of them to solve problems in a coordinated fashion. At which point it might make sense to use techniques from GAs to create an orchestra of the mind, where each region of an artificial brain is always learning and evolving and bouncing ideas off the others.

ralphc 1y ago

Would your 1000 core CPU do well on neuroevolution?

zackmorris 1y ago
Thank you, believe it or not I hadn't heard that word so it filled in a piece of the puzzle for me. https://en.wikipedia.org/wiki/Neuroevolution https://en.wikipedia.org/wiki/Neuroevolution https://medium.com/@roopal.tatiwar20/neuroevolution-evolving-neural-network-with-genetic-algorithms-8ca2165ad04c https://medium.com/@roopal.tatiwar20/neuroevolution-evolving... That was the state of the art when I got my ECE degree in 1999. They were using genetic algorithms (GAs) to evolve initial weights for neural net (NN) gradient descent right before the Dot Bomb. Then pretty much all R&D was cut/abandoned overnight, and the 2000s went to offshoring jobs and shuttering factories which led to the Housing Bubble popping. IMHO that set AI research back at least 10 years, maybe 20, at least in the US. I know it derailed my career dreams. I feel that GPUs by their SIMD nature do poorly with a large number of concurrent processes. For example, a population of GA agents running lisp with each instruction of the intermediate code (icode) tree encoded as bits in a bitfield which evolves using techniques like mutation, crossover and sexual reproduction. Some of those bits represent conditional branching. Ideally we'd want the opposite of a GPU, more like a transputer, on the order of thousands of independent arithmetic logic units (ALUs) with their own local memories, and perhaps able to use custom instructions. That way each lisp tree can execute concurrently in isolation. There may be branch-avoidance transformations from shaders that could help with this, to transpile the von Neumann SISD and MIMD code we're used to into the narrower SIMD so it can run on GPUs. Companies like Nvidia aren't touching this for some reason, and I don't know if it's just a blind spot or deliberate. This is one reason why I wanted FPGAs to take off, since they'd make great transputers. Unfortunately they stagnated due to proprietary compilers and a lack of profitable applications. Today it looks like microcontroller units (MCUs) might take their place, but they're still at least 4 orders of magnitude or more too small and 100 times too slow to be cost-effective. Imagine having a 1 GHz CPU that can run 1000 concurrent UNIX processes running lisp that only communicate through sockets. It's easy to see how mutating them is trivial, then running them as unit tests to see which ones pass is also straightforward. Erlang and Go figured that out years ago. It's so easy in fact to "just come up with the answer" this way, that I think this technique has been suppressed. With that foundation and removing the element of time, it's easy to see that this could be a drop-in replacement for gradient descent. Then the work shifts to curating large data and training sets. I suspect that this is the real work of AI research, and that the learning technique doesn't matter so much. I think of it as: GAs are good for getting to a local minimum error - finding a known-good state. NNs are better for hill climbing - exploring the local solution space. Another way to say that is, corporations spent billions of dollars to train the first LLM MVPs with gradient descent and other "hands on" techniques. When maybe they could have spent thousands of dollars if they had used GAs instead and "let go". Now that we're here, refinement is more important, so gradient descent is here to stay. Although I think of LLM blueprints as eventually fitting on a napkin and running on one chip. Then we'll orchestrate large numbers of them to solve problems in a coordinated fashion. At which point it might make sense to use techniques from GAs to create an orchestra of the mind, where each region of an artificial brain is always learning and evolving and bouncing ideas off the others.