4 ms·

I'm late to the party due to work and holiday gatherings, but just wanted to say that this is the first glimmer of hope on the horizon in 25 years. The slowing

I'm late to the party due to work and holiday gatherings, but just wanted to say that this is the first glimmer of hope on the horizon in 25 years. The slowing down of Moore's law was well-understood by the time I was graduating from college with my Electrical and Computer Engineering (ECE) degree in 1999. The DEC Alpha was 700 MHz and they had GHz chips in labs, but most of us assumed that getting past 3-4 GHz was going to be difficult or impossible due to the switching limit of silicon with respect to pipeline stages. Also, on-chip interconnect had grown from a square profile to a ribbon shape that was taller than wide, which requires waveguide analysis. And features weren't shrinking anymore, they were just adding more layers, which isn't N^2 scaling. The end arrived just after 2007 when smartphones arrived, and the world chose affordability and efficiency over performance. Everyone hopped on the GPU bandwagon and desktop computing was largely abandoned for the next 20 years. But I've been hearing more about High-Performance Computing (HPC) recently. It's multicore computing with better languages, which is the missing link between CPU and GPU. What happened was, back in the 90s, game companies pushed for OpenGL on GPUs without doing the real work of solving auto-parallelization of code (with no pragmas/intrinsics/compiler hints etc) for multicore computers first. We should have had parallel multicore, then implemented matrix and graphics libraries over that, with OpenGL as a thin wrapper over that DSP framework. But we never got it, so everyone has been manually transpiling their general solutions to domain-specific languages (DSLs) like GLSL, HLSL and CUDA, at exorbitant effort and cost with arbitrary limitations. Established players like Nvidia don't care about this, because the status quo presents barriers to entry to competitors who might disrupt their business model. Anyway, since HPC is a general computing approach, it can be the foundation for the propriety and esoteric libraries like Vulkan, Metal, etc. Which will democratize programming and let hobbyists more easily dabble in the dozen or so alternatives to Neural Nets. Especially Genetic Algorithms (GAs), which can automatically derive the complex hand-rolled implementations like Large Language Models (LLMs) and Stable Diffusion. We'll also be able to try multiple approaches in simulation and it will be easier to run a high number of agents simultaneously so we can study their interactions and learning models in a transparent and repeatable fashion akin to using reproducible builds and declarative programming. That's how far John Koza and others got with GAs in the mid-2000s before they had to be abandon them due to cost. Now, this AmpereOne costs about an order of magnitude more than would be required for a hobbyist 256 core computer. But with inflation, $5-10,000 isn't too far off from the price of desktop computers in the 80s. So I expect that prices will come down over 3-5 years as more cores and memory are moved on-chip. Note that we never really got a System on a Chip (SoC) either, so that's some more real work that remains. We also never got reconfigurable hardware with on-chip FPGA units that could solve problems with a divide and conquer strategy, a bit like Just in Time (JIT) compilers. I had detailed plans in my head for all of this by around 2010. Especially on the software side, with ideas for better languages that combine the power of Octave/MATLAB with the approachability of Go/Erlang. The hardware's pretty easy other than needing a fab, which presents a $1-10 billion barrier. That's why it's so important that AmpereOne exists as a model to copy. The real lesson for me is that our imaginations can manifest anything, but the bigger the dream, the longer it takes for the universe to deliver it. On another timeline, I might have managed it in 10 years if I had won the internet lottery, but instead I had to wait 25 as I ran the rat race to make rent. Now I'm too old and busted so it's probably too late for me. But if HPC takes off, I might find renewed passion for tech. Since this is the way we deliver Universal Basic Income (UBI) and moneyless resources through 3D printing etc outside of subscription-based AI schemes.

AnimalMuppet 2y ago
1. What languages do you think are good ones for HPC? 2. Re GAs, Koza, and cost: In Koza's book on GAs, in one of the editions, he mentions that they scaled the performance of their research system by five orders of magnitude in a decade. What he didn't mention was that they did it by going from one Lisp Machine to 1000 PCs. They only got two orders of magnitude from per-unit performance; the rest came from growing the number of units. Of course they couldn't keep scaling that way for cost reasons. Cost, and space, and power, and several other limiting factors. They weren't going to have a million machines a decade later, or a billion machines a decade after that. To the degree that GAs (or at least their approach to them) required that in order to keep working, to that degree their approach was not workable.
- deleted 2y ago
  [deleted]
- zackmorris 2y ago
  > 2. Re GAs, Koza, and cost: Ya that's a really good point about the linear scaling limits of genetic algorithms, so let me provide some context: Where I disagree with the chip industry is that I grew up on a Mac Plus with a 1979 Motorola 68000 processor with 68,000 transistors (not a typo) that ran at 8 MHz and could get real work done - as far as spreadsheets and desktop publishing - arguably more easily than today. So to me, computers waste most of their potential now: https://en.wikipedia.org/wiki/Transistor_count https://en.wikipedia.org/wiki/Transistor_count As of 2023, Apple's M2 Max had 67 billion transistors running at 3.7 GHz and it's not even the biggest on the list. Although it's bigger than Nvidia's A100 (GA100 Ampere) at 54 billion, which is actually pretty impressive. If we assume this is all roughly linear, we have: year count speed 1979 6.8e4 8e6 2023 6.7e10 3.7e9 So the M2 should have (~1 million) * (~500) = 500 million times the computing power of the 68000 over 44 years, or Moore's law applied 29 times (44 years/18 months ~= 29). Are computers today 500 million times faster than a Mac Plus? It's a ridiculous question and the answer is self-evidently no. Without the video card, they are more like 500 times faster in practice, and almost no faster at all in real-world tasks like surfing the web. This leads to some inconvenient questions: - Where did the factor of 1 million+ speedup go? - Why have CPUs seemingly not even tried to keep up with GPUs? - If GPUs are so much better able to recruit their transistor counts, why can't they even run general purpose C-style code and mainstream software like Docker yet? Like you said and Koza found, the per-unit speed only increased 100 times in the decade of the 2000s by Moore's law because 2^(10 years/18 months) ~= 100. If it went up that much again in the 2010s (it didn't), then that would be a 10,000 times speedup by 2020, and we could extrapolate that each unit today would run about 2^(24 years/18 months) ~= 100,000 times faster than that original lisp machine. A 1000 unit cluster would run 100 million or 10^8 times faster, so research on genetic algorithms could have continued. But say we wanted to get back into genetic algorithms and the industry gave computer engineers access to real resources. We would design a 100 billion transistor CPU with around 1 million cores having 100,000 transistors each, costing $1000. Or if you like, 10,000 Pentium Pro or PowerPC 604 cores on a chip at 10 million transistors each. It would also have a content-addressable memory so core-local memory appears as a contiguous address space. It would share data between cores like how BitTorrent works. So no changes to code would be needed to process data in-cluster or distributed around the web. So that's my answer: computers today run thousands or even millions of times slower than they should for the price, but nobody cares because there's no market for real multicore. Computers were "good enough" by 2007 when smartphones arrived, so that's when their evolution ended. Had their evolution continued, then the linear scaling limits would have fallen away under the exponential growth of Moore's law, and we wouldn't be having this conversation. > 1. What languages do you think are good ones for HPC? That's still an open question. IMHO little or no research has happened there, because we didn't have the real multicore CPUs mentioned above. For a little context: Here are some aspects of good programming languages; https://www.linkedin.com/pulse/key-qualities-good-programming-code-nabeel-javaid https://www.linkedin.com/pulse/key-qualities-good-programmin... https://www.chakray.com/programming-languages-types-and-features/ https://www.chakray.com/programming-languages-types-and-feat... Abstractability Approachability Brevity Capability Consistency Correctness Efficiency Interactivy Locality Maintainability Performance Portability Productivity Readability Reliability Reusability Scalability Security Simplicity Testability Most languages only shine for a handful of these. And some don't even seem to try. For example: OpenCL is trying to address parallelization in all of the wrong ways, by copying the wrong approach (CUDA). Here's a particularly poor example, the top hit on Google: Calculate X[i] = pow(X[i],2) in 200-ish lines of code: https://github.com/rsnemmen/OpenCL-examples/blob/master/Hello_World/hello.c https://github.com/rsnemmen/OpenCL-examples/blob/master/Hell... Octave/MATLAB addresses parallelization in all of the right ways, through the language of spreadsheets and matrices: Calculate X[i] = pow(X[i],2) in 1 line of code as x .^ y or power(x, y): https://docs.octave.org/v4.4.0/Arithmetic-Ops.html#XREFpower https://docs.octave.org/v4.4.0/Arithmetic-Ops.html#XREFpower But unfortunately neither OpenCL nor Octave/MATLAB currently recruit multicore or cluster computers effectively. I think there was research in the 80s and 90s on that, but the languages were esoteric or hand-rolled. Basically hand out a bunch of shell scripts to run and aggregate the results. It all died out around the same time as Beowulf cluster jokes did: https://en.wikipedia.org/wiki/Computer_cluster https://en.wikipedia.org/wiki/Computer_cluster Here's one called Emerald: https://medium.com/@mwendakelvinblog/emerald-the-language-of-distributed-object-oriented-innovation-2fd5d1df1c71 https://medium.com/@mwendakelvinblog/emerald-the-language-of... http://www.emeraldprogramminglanguage.org http://www.emeraldprogramminglanguage.org https://emeraldlang.github.io/emerald/ https://emeraldlang.github.io/emerald/ After 35 years of programming experience, here's what I want: - Multi-element operations by a single operator like in Octave/MATLAB or shader languages like HLSL - All variables const (no mutability) with get/set between one-shot executions like the suspend-the-world io of ClojureScript and REDUX state - No monads/futures/async or borrow checker like in Rust (const negates their need, just use 2x memory rather than in-place mutation) - Pass-by-value copy-on-write semantics for all arguments like with PHP arrays (pass-by-reference classes broke PHP 5+) - Auto-parallelization of flow control and loops via static analysis of intermediate code (no pragmas/intrinsics, const avoids side effects) - Functional and focused on higher-order methods like map/reduce/filter, de-emphasize Java-style object-oriented classes - Smart collections like JavaScript classes "x.y <=> x[y]" and PHP arrays "x[y] <=> x[n]", instead of "pure" set/map/array - No ban on multiple inheritance, no final keyword, let the compiler solve inheritance constraints - No imports, give us everything and the kitchen sink like PHP, let the compiler strip unused code - Parse infix/prefix/postfix notation equally with a converter like goformat, with import/export to spreadsheet and (graph) database It would be a pure functional language (impurity negates the benefits of cryptic languages like Haskell), easier to read and more consistent than JavaScript, with simple fork/join like Go/Erlang and single-operator math on array/matrix elements like Octave/Matlab. Kind of like standalone spreadsheets connected by shell pipes over the web, but as code in a file. Akin to Jupyter notebooks I guess, or Wolfram Alpha, or literate programming. Note that this list flies in the face of many best practices today. That's because I'm coming from using languages like HyperTalk that cater to the user rather than the computer. And honestly I'm trying to get away from languages. I mostly use #nocode spreadsheets and SQL now. I wish I could make cross-platform apps with spreadsheets (a bit like Airtable and Zapier) and had a functional or algebra of sets interface for databases. It would take me at least a year or two and $100,000-250,000 minimum to write an MVP of this language. It's simply too ambitious to make in my spare time. Sorry about length and delay on this!

AnimalMuppet 2y ago

1. What languages do you think are good ones for HPC? 2. Re GAs, Koza, and cost: In Koza's book on GAs, in one of the editions, he mentions that they scaled the performance of their research system by five orders of magnitude in a decade. What he didn't mention was that they did it by going from one Lisp Machine to 1000 PCs. They only got two orders of magnitude from per-unit performance; the rest came from growing the number of units. Of course they couldn't keep scaling that way for cost reasons. Cost, and space, and power, and several other limiting factors. They weren't going to have a million machines a decade later, or a billion machines a decade after that. To the degree that GAs (or at least their approach to them) required that in order to keep working, to that degree their approach was not workable.

deleted 2y ago
[deleted]
zackmorris 2y ago
> 2. Re GAs, Koza, and cost: Ya that's a really good point about the linear scaling limits of genetic algorithms, so let me provide some context: Where I disagree with the chip industry is that I grew up on a Mac Plus with a 1979 Motorola 68000 processor with 68,000 transistors (not a typo) that ran at 8 MHz and could get real work done - as far as spreadsheets and desktop publishing - arguably more easily than today. So to me, computers waste most of their potential now: https://en.wikipedia.org/wiki/Transistor_count https://en.wikipedia.org/wiki/Transistor_count As of 2023, Apple's M2 Max had 67 billion transistors running at 3.7 GHz and it's not even the biggest on the list. Although it's bigger than Nvidia's A100 (GA100 Ampere) at 54 billion, which is actually pretty impressive. If we assume this is all roughly linear, we have: year count speed 1979 6.8e4 8e6 2023 6.7e10 3.7e9 So the M2 should have (~1 million) * (~500) = 500 million times the computing power of the 68000 over 44 years, or Moore's law applied 29 times (44 years/18 months ~= 29). Are computers today 500 million times faster than a Mac Plus? It's a ridiculous question and the answer is self-evidently no. Without the video card, they are more like 500 times faster in practice, and almost no faster at all in real-world tasks like surfing the web. This leads to some inconvenient questions: - Where did the factor of 1 million+ speedup go? - Why have CPUs seemingly not even tried to keep up with GPUs? - If GPUs are so much better able to recruit their transistor counts, why can't they even run general purpose C-style code and mainstream software like Docker yet? Like you said and Koza found, the per-unit speed only increased 100 times in the decade of the 2000s by Moore's law because 2^(10 years/18 months) ~= 100. If it went up that much again in the 2010s (it didn't), then that would be a 10,000 times speedup by 2020, and we could extrapolate that each unit today would run about 2^(24 years/18 months) ~= 100,000 times faster than that original lisp machine. A 1000 unit cluster would run 100 million or 10^8 times faster, so research on genetic algorithms could have continued. But say we wanted to get back into genetic algorithms and the industry gave computer engineers access to real resources. We would design a 100 billion transistor CPU with around 1 million cores having 100,000 transistors each, costing $1000. Or if you like, 10,000 Pentium Pro or PowerPC 604 cores on a chip at 10 million transistors each. It would also have a content-addressable memory so core-local memory appears as a contiguous address space. It would share data between cores like how BitTorrent works. So no changes to code would be needed to process data in-cluster or distributed around the web. So that's my answer: computers today run thousands or even millions of times slower than they should for the price, but nobody cares because there's no market for real multicore. Computers were "good enough" by 2007 when smartphones arrived, so that's when their evolution ended. Had their evolution continued, then the linear scaling limits would have fallen away under the exponential growth of Moore's law, and we wouldn't be having this conversation. > 1. What languages do you think are good ones for HPC? That's still an open question. IMHO little or no research has happened there, because we didn't have the real multicore CPUs mentioned above. For a little context: Here are some aspects of good programming languages; https://www.linkedin.com/pulse/key-qualities-good-programming-code-nabeel-javaid https://www.linkedin.com/pulse/key-qualities-good-programmin... https://www.chakray.com/programming-languages-types-and-features/ https://www.chakray.com/programming-languages-types-and-feat... Abstractability Approachability Brevity Capability Consistency Correctness Efficiency Interactivy Locality Maintainability Performance Portability Productivity Readability Reliability Reusability Scalability Security Simplicity Testability Most languages only shine for a handful of these. And some don't even seem to try. For example: OpenCL is trying to address parallelization in all of the wrong ways, by copying the wrong approach (CUDA). Here's a particularly poor example, the top hit on Google: Calculate X[i] = pow(X[i],2) in 200-ish lines of code: https://github.com/rsnemmen/OpenCL-examples/blob/master/Hello_World/hello.c https://github.com/rsnemmen/OpenCL-examples/blob/master/Hell... Octave/MATLAB addresses parallelization in all of the right ways, through the language of spreadsheets and matrices: Calculate X[i] = pow(X[i],2) in 1 line of code as x .^ y or power(x, y): https://docs.octave.org/v4.4.0/Arithmetic-Ops.html#XREFpower https://docs.octave.org/v4.4.0/Arithmetic-Ops.html#XREFpower But unfortunately neither OpenCL nor Octave/MATLAB currently recruit multicore or cluster computers effectively. I think there was research in the 80s and 90s on that, but the languages were esoteric or hand-rolled. Basically hand out a bunch of shell scripts to run and aggregate the results. It all died out around the same time as Beowulf cluster jokes did: https://en.wikipedia.org/wiki/Computer_cluster https://en.wikipedia.org/wiki/Computer_cluster Here's one called Emerald: https://medium.com/@mwendakelvinblog/emerald-the-language-of-distributed-object-oriented-innovation-2fd5d1df1c71 https://medium.com/@mwendakelvinblog/emerald-the-language-of... http://www.emeraldprogramminglanguage.org http://www.emeraldprogramminglanguage.org https://emeraldlang.github.io/emerald/ https://emeraldlang.github.io/emerald/ After 35 years of programming experience, here's what I want: - Multi-element operations by a single operator like in Octave/MATLAB or shader languages like HLSL - All variables const (no mutability) with get/set between one-shot executions like the suspend-the-world io of ClojureScript and REDUX state - No monads/futures/async or borrow checker like in Rust (const negates their need, just use 2x memory rather than in-place mutation) - Pass-by-value copy-on-write semantics for all arguments like with PHP arrays (pass-by-reference classes broke PHP 5+) - Auto-parallelization of flow control and loops via static analysis of intermediate code (no pragmas/intrinsics, const avoids side effects) - Functional and focused on higher-order methods like map/reduce/filter, de-emphasize Java-style object-oriented classes - Smart collections like JavaScript classes "x.y <=> x[y]" and PHP arrays "x[y] <=> x[n]", instead of "pure" set/map/array - No ban on multiple inheritance, no final keyword, let the compiler solve inheritance constraints - No imports, give us everything and the kitchen sink like PHP, let the compiler strip unused code - Parse infix/prefix/postfix notation equally with a converter like goformat, with import/export to spreadsheet and (graph) database It would be a pure functional language (impurity negates the benefits of cryptic languages like Haskell), easier to read and more consistent than JavaScript, with simple fork/join like Go/Erlang and single-operator math on array/matrix elements like Octave/Matlab. Kind of like standalone spreadsheets connected by shell pipes over the web, but as code in a file. Akin to Jupyter notebooks I guess, or Wolfram Alpha, or literate programming. Note that this list flies in the face of many best practices today. That's because I'm coming from using languages like HyperTalk that cater to the user rather than the computer. And honestly I'm trying to get away from languages. I mostly use #nocode spreadsheets and SQL now. I wish I could make cross-platform apps with spreadsheets (a bit like Airtable and Zapier) and had a functional or algebra of sets interface for databases. It would take me at least a year or two and $100,000-250,000 minimum to write an MVP of this language. It's simply too ambitious to make in my spare time. Sorry about length and delay on this!