17 ms·

I want a good parallel computer

233 points by raphlinus 2y ago

armchairhacker 2y ago
> The GPU in your computer is about 10 to 100 times more powerful than the CPU, depending on workload. For real-time graphics rendering and machine learning, you are enjoying that power, and doing those workloads on a CPU is not viable. Why aren’t we exploiting that power for other workloads? What prevents a GPU from being a more general purpose computer? What other workloads would benefit from a GPU? Computers are so fast that in practice, many tasks don't need more performance. If a program that runs those tasks is slow, it's because that program's code is particularly bad, and the solution to make the code less bad is simpler than re-writing it for the GPU. For example, GUIs have been imperceptibly reactive to user input for over 20 years. If an app's GUI feels sluggish, the problem is that the app's actions and rendering aren't on separate coroutines, or the action's coroutine is blocking (maybe it needs to be on a separate thread). But the rendering part of the GUI doesn't need to be on a GPU (any more than it is today, I admit I don't know much about rendering), because responsive GUIs exist today, some even written in scripting languages. In some cases, parallelizing a task intrinsically makes it slower, because the number of sequential operations required to handle coordination mean there are more forced-sequential operations in total. In other cases, a program spawns 1000+ threads but they only run on 8-16 processors, so the program would be faster if it spawned less threads because it would still use all processors. I do think GPU programming should be made much simpler, so this work is probably useful, but mainly to ease the implementation of tasks that already use the GPU: real-time graphics and machine learning.
- wmf 2y ago
  A big one is video encoding. It seems like GPUs would be ideal for it but in practice limitations in either the hardware or programming model make it hard to efficiently run on GPU shader cores. (GPUs usually include separate fixed-function video engines but these aren't programmable to support future codecs.)
  - dist-epoch 2y ago
    Video encoding is done with fixed-function for power efficiency. A new popular codec like H26x codec appears every 5-10 years, there is no real need to support future ones.
    - nwallin 2y ago
      Video encoding is two domains. And there's surprisingly little overlap between them. You have your real time video encoding. This is video conferencing, live television broadcasts. This is done fixed-function not just for power efficiency, but also latency. The second domain is encoding at rest. This is youtube, netflix, blu-ray, etc. This is usually done in software on the CPU for compression ratio efficiency. The problem with fixed function video encoding is that the compression ratio is bad. You either have enormous data, or awful video quality, or both. The problem with software video encoding is that it's really slow. OP is asking why we can't/don't have the best of both worlds. Why can't/don't we write a video encoder in OpenCL/CUDA/ROCm. So that we have the speed of using the GPU's compute capability but compression ratio of software.
    - deleted 2y ago
      [deleted]
- raphlinus 2y ago
  Possibly compilation and linking. That's very slow for big programs like Chromium. There's really interesting work on GPU compilers (co-dfns and Voetter's work). Optimization problems like scheduling and circuit routing. Search in theorem proving (the classical parts like model checking, not just LLM). There's still a lot that is slow and should be faster, or at the very least made to run using less power. GPUs are good at that for graphics, and I'd like to see those techniques applied more broadly.
  - return_to_monke 2y ago
    All of these things you mention are "thinking", meaning they require complex algorithms with a bunch of branches and edge cases. The tasks that GPUs are good at right now - graphics, number crunching, etc - are all very simple algorithms at the core (mostly elementary linear algebra), and the problems are, in most cases, embarassingly parallel. CPUs are not very good at branching either - see all the effort being put towards getting branch prediction right - but they are way better at it than GPUs. The main appeal of GPGPU programming is, in my opinion, that if you can get the CPU to efficiently divide the larger problem into a lot of small, simple subtasks, you can achieve faster speeds. You mentioned compilers. See a related example, for reference all the work Daniel Lemire has been doing on SIMD parsing: the algorithms he (co)invented are all highly specialized to the language, and highly nontrivial. Branchless programming requires an entirely different mindset/intuition than "traditional" programming, and I wouldn't expect the average programmer to come up with such novel ideas. A GPU is a specialized tool that is useful for a particular purpose, not a silver bullet to magically speed up your code. Theree is a reason that we are using it for its current purposes.
  - hulitu 2y ago
    > Possibly compilation and linking. That's very slow for big programs like Chromium. So instead of fixing the problem (Chromium's bloat) we just trow more memory and computing power at it, hopping that the problem will go away. Maybe we shall teach programmers to programm. /s
IshKebab 2y ago
Having worked for a company that made a "hundreds of small CPUs on a single chip", I can tell you now that they're all going to fail because the programming model is too weird, and nobody will write software for them. Whatever comes next will be a GPU with extra capabilities, not a totally new architecture. Probably an nVidia GPU.
- bryanlarsen 2y ago
  While acknowledging that it's theoretically possible other approaches might succeed, it seems quite clear the author agrees with you.
- convolvatron 2y ago
  my take from reading this is more about programming abstractions than any particular hardware instantiation. the part of the Connection Machine that remains interesting is not building machines with CPUS with transistor counts in the hundreds running off a globally synchronous clock, but that there were a whole family of SIMD languages and let you do general purpose programming in parallel. And that those language were still relevant when the architecture changed to a MIMD machine with a bunch of vector units behind each CPU.
- snovymgodym 2y ago
  Reminds me of Itanium
  - CyberDildonics 2y ago
    How is that at all like Itanium except for the superficial headline level where people say they are hard to program?
    - snovymgodym 2y ago
      Because the main feature that made Itanium hard to program for was its explicit instruction-level parallelism.
svmhdvn 2y ago
I've always admired the work that the team behind https://www.greenarraychips.com/ https://www.greenarraychips.com/ does, and the GA144 chip seems like a great parallel computing innovation.
bee_rider 2y ago
It is odd that he talks about Larabee so much, but doesn’t mention the Xeon Phis. (Or is it Xeons Phi?). > As a general trend, CPU designs are diverging into those optimizing single-core performance (performance cores) and those optimizing power efficiency (efficiency cores), with cores of both types commonly present on the same chip. As E-cores become more prevalent, algorithms designed to exploit parallelism at scale may start winning, incentivizing provision of even larger numbers of increasingly efficient cores, even if underpowered for single-threaded tasks. I’ve always been slightly annoyed by the concept of E cores, because they are so close to what I want, but not quite there… I want, like, throughput cores. Let’s take E cores, give them their AVX-512 back, and give them higher throughput memory. Maybe try and pull the Phi trick of less OoO capabilities but more threads per core. Eventually the goal should be to come up with an AVX unit so big it kills iGPUs, haha.
- nullpoint420 2y ago
  I've always wondered if you could use iGPU compute cores with unified memory as "transparent" E-cores when needed. Something like OpenCL/CUDA except it works with pthreads/goroutines and other (OS) kernel threading primitives, so code doesn't need to be recompiled for it. Ideally the OS scheduler would know how to split the work, similar to how E-core and P-core scheduling works today. I don't do HPC professionally, so I assume I'm ignorant to why this isn't possible.
- Retr0id 2y ago
  Isn't Xeon Phi just an instance of Larrabee?
  - deleted 2y ago
    [deleted]
  - adrian_b 2y ago
    It is an instance of Larrabee in the same sense as AMD Zen 4 is an instance of Larrabee. The "Larrabee New Instructions" is an instruction set that has been designed before AVX and also its first hardware implementation has been introduced before AVX, in 2010 (AVX was launched in 2011, with Sandy Bridge). Unfortunately while the hardware design of Sandy Bridge with the inferior AVX ISA has been done by the Intel A team, the hardware implementations of Larrabee have been done by some C or D teams, which were also not able to design new CPU cores for it, but they had to reuse some obsolete x86 cores, initially a Pentium core and later an Atom Silvermont core, to which the Larrabee instructions were grafted. "Larrabee New Instructions" have been renamed to "Many Integrated Cores" ISA, then to AVX-512, while passing through 3 generations of chips, Knights Ferry, Knights Corner and Knights Landing. A fourth generation, Knights Mill, was only intended for machine learning/AI applications. The successor of Knights Landing has been Skylake Server, when the AVX-512 ISA has come to standard Xeons, marking the disappearance of Xeon Phi. Already in 2013, Intel Haswell has added to AVX a few of the more important instructions that were included in the Larrabee New Instructions, but which were missing in AVX, e.g. fused multiply-add and gather instructions. The 3-address FMA format, which has caused problems to AMD, who had implemented in Bulldozer a 4-address format, has also come to AVX from Larrabee, replacing the initial 4-address specification. At each generation until Skylake Server, some of the original Larrabee instructions have been deleted, by assuming that they might be needed only for graphics, which was no longer the intended market. However a few of those instructions were really useful for some applications in which I am interested, e.g. for computations with big numbers, so I regret their disappearance. Since Skylake Server, there have been no other instruction removals, with the exception of those introduced by Intel Tiger Lake, which are now supported only by AMD Zen 5. A few days ago Intel has committed to keeping complete compatibility in the future with the ISA implemented today by Granite Rapids, so there will be no other instruction deletions.
andrewstuart 2y ago
AMD Strix Halo APU is a CPU with very powerful integrated GPU. It’s faster at AI than an Nvidia RTX4090, because 96GB of the 128GB can be allocated to the GPU memory space. This means it’s doesn’t have the same swapping/memory thrashing that a discrete GPU experiences when processing large models. 16 CPU cores and 40 GPU compute units sounds pretty parallel to me. Doesn’t that fit the bill?
- dr_kiszonka 2y ago
  It looks like it will be available in the Framework Desktop! I would love to see it in a more budget mini PC at some point from another company. (Framework is great but not in my price range.)
- bigyabai 2y ago
  > It’s faster at AI than an Nvidia RTX4090, because 96GB of the 128GB can be allocated to the GPU memory space I love AMD's Ryzen chips and will recommend their laptops over an Nvidia model all day. However, this is a pretty facetious comparison that falls apart when you normalize the memory. Any chip can be memory bottlenecked, and if we take away that arbitrary precondition the Strix Halo gets trounced in terms of compute capacity. You can look at the TDP of either chip and surmise this pretty easily.
  - dist-epoch 2y ago
    > However, this is a pretty facetious comparison that falls apart when you normalize the memory Why would you normalize though? You can't buy a 96 GB RTX4090. So it's fair to compare the whole deal, slowish APU with large RAM versus very fast GPU with limited RAM.
    - Animats 2y ago
      > You can't buy a 96 GB RTX4090 You can now buy a 96 GB RTX5090.[1] NVidia gives it a "Pro" designation and charges more, but it's the same chip. [1] https://www.tomshardware.com/pc-components/gpus/nvidia-rtx-pro-6000-up-close-blackwell-rtx-workstation-max-q-workstation-and-server-variants-shown https://www.tomshardware.com/pc-components/gpus/nvidia-rtx-p...
grg0 2y ago
The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths. It's an entirely moronic programming model to be using in 2025. - You need to compile shader source/bytecode at runtime; you can't just "run" a program. - On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better design an algorithm that does not require immediate synchronization between the two. - You need to synchronize data access between CPU-GPU and GPU workloads. - You need to deal with bad and confusing APIs because there is no standardization of the underlying hardware. - You need to deal with a combinatorial turd explosion of configurations. HW vendors want to protect their turd, so drivers and specs are behind fairly tight gates. OS vendors also want to protect their turd and refuse even the software API standard altogether. And then the tooling also sucks. What I would like is a CPU with a highly parallel array of "worker cores" all addressing the same memory and speaking the same goddamn language that the CPU does. But maybe that is an inherently crappy architecture for reasons that are beyond my basic hardware knowledge.
- Grosvenor 2y ago
  What I want is a Linear Algebra interface - As Gilbert Strang taught it. I'll "program" in LinAlg, and a JIT can compile it to whatever wonky way your HW requires. I'm not willing to even know about the HW at all, the higher level my code the more opportunities for the JIT to optimize my code. What I really want is something like Mathematica that can JIT to GPU. As another commenter mentioned all the API's assume you're a discrete GPU off the end of a slow bus, without shared memory. I would kill for an APU that could freely allocate memory for GPU or CPU and change ownership with the speed of a pagefault or kernel transition.
  - RossBencina 2y ago
    > What I really want is something like Mathematica that can JIT to GPU. https://juliagpu.org/ https://juliagpu.org/
  - vgatherps 2y ago
    https://github.com/jax-ml/jax https://github.com/jax-ml/jax
helf 2y ago
[dead]
Retr0id 2y ago
Something that frustrates me a little is that my system (apple silicon) has unified memory, which in theory should negate the need to shuffle data between CPU and GPU. But, iiuc, the GPU programming APIs at my disposal all require me to pretend the memory is not unified - which makes sense because they want to be portable across different hardware configurations. But it would make my life a lot easier if I could just target the hardware I have, and ignore compatibility concerns.
- deviantbit 2y ago
  Unified memory doesn't mean unified address space. It frustrates me when no one understands unified memory.
  - morphle 2y ago
    If you fix the pages tables (partial tutorial online) you can have continuous unified address space on Apple Silicon.
    - deviantbit 2y ago
      Let’s be honest, saying “just fix the page tables” is like telling someone they can fly if they “just rewrite gravity.” Yes, on Apple Silicon, the hardware supports shared physical memory, and with enough “convincing”, you can rig up a contiguous virtual address space for both the CPU and GPU. Apple’s unified memory architecture makes that possible, but Apple’s APIs and memory managers don’t expose this easily or safely for a reason. You’re messing with MMU-level mappings on a tightly integrated system that treats memory as a first-class citizen of the security model. I can tell you never programmed on an Amiga.
      - morphle 2y ago
        Oh yes I programmed all the Amiga models, mostly in assembly level. I reprogrammed the ROMs. I also published a magazine on all the Commodore computers internals and build lots of hardware for these machines. We had the parallel Inmos Transputer systems during the heyday of the Amiga, they where much better designed than any the custom Amiga chips.
morphle 2y ago
I haven't yet read the full blog post but so far my response is you can have this good parallel computer. See my previous HN comments the past months on building an M4 Mac mini supercomputer. For example reverse engineering the Apple M3 Ultra GPU and Neural Engine instruction set and IOMMU and pages tables that prevent you from programming all processor cores in the chip (146 cores to over ten thousand depending on how you delineate what a core is) and making your own Abstract Syntax Tree to assembly compiler for these undocumented cores will unleash at least 50 trillion operations per second. I still have to benchmark this chip and make the roofline graphs for the M4 to be sure, it might be more. https://en.wikipedia.org/wiki/Roofline_model https://en.wikipedia.org/wiki/Roofline_model
Animats 2y ago
Interesting article. Other than as an exercise, it's not clear why someone would write a massively parallel 2D renderer that needs a GPU. Modern GPUs are overkill for 2D. Now, 3D renderers, we need all the help we can get. In this context, a "renderer" is something that takes in meshes, textures, materials, transforms, and objects, and generates images. It's not an entire game development engine, such as Unreal, Unity, or Bevy. Those have several more upper levels above the renderer. Game engines know what all the objects are and what they are doing. Renderers don't. Vulkan, incidentally, is a level below the renderer. Vulkan is a cross-hardware API for asking a GPU to do all the things a GPU can do. WGPU for Rust, incidentally, is an wrapper to extend that concept to cross-platform (Mac, Android, browsers, etc.) While it seems you can write a general 3D renderer that works in a wide variety of situations, that does not work well in practice. I wish Rust had one. I've tried Rend3 (abandoned), and looked at Renderling (in progress), Orbit (abandoned), and Three.rs (abandoned). They all scale up badly as scene complexity increases. There's a friction point in design here. The renderer needs more info to work efficiently than it needs to just draw in a dumb way. Modern GPSs are good enough that a dumb renderer works pretty well, until the scene complexity hits some limit. Beyond that point, problems such as lighting requiring O(lights * objects) time start to dominate. The CPU driving the GPU maxes out while the GPU is at maybe 40% utilization. The operations that can easily be parallelized have been. Now it gets hard. In Rust 3D land, everybody seems to write My First Renderer, hit this wall, and quit. The big game engines (Unreal, etc.) handle this by using the scene graph info of the game to guide the rendering process. This is visually effective, very complicated, prone to bugs, and takes a huge engine dev team to make work. Nobody has a good solution to this yet. What does the renderer need to know from its caller? A first step I'm looking at is something where, for each light, the caller provides a lambda which can iterate through the objects in range of the light. That way, the renderer can get some info from the caller's spatial data structures. May or may not be a good idea. Too early to tell. [1] https://github.com/linebender/vello/ https://github.com/linebender/vello/
- amelius 2y ago
  > Other than as an exercise, it's not clear why someone would write a massively parallel 2D renderer that needs a GPU. Modern GPUs are overkill for 2D. Depends on how complicated your artwork is.
dekhn 2y ago
There are many intertwined issues here. One of the reasons we can't have a good parallel computer is that you need to get a large number of people to adopt your device for development purposes, and they need to have a large community of people who can run their code. Great projects die all the time because a slightly worse, but more ubiquitous technology prevents flowering of new approaches. There are economies of scale that feed back into ever-improving iterations of existing systems. Simply porting existing successful codes from CPU to GPU can be a major undertaking and if there aren't any experts who can write something that drive immediate sales, a project can die on the vine. See for example https://en.wikipedia.org/wiki/Cray_MTA https://en.wikipedia.org/wiki/Cray_MTA when I was first asked to try this machine, it was pitched as "run a million threads, the system will context switch between threads when they block on memory and run them when the memory is ready". It never really made it on its own as a supercomputer, but lots of the ideas made it to GPUs. AMD and others have explored the idea of moving the GPU closer to the CPU by placing it directly onto the same memory crossbar. Instead of the GPU connecting to the PCI express controller, it gets dropped into a socket just like a CPU. I've found the best strategy is to target my development for what the high end consumers are buying in 2 years - this is similar to many games, which launch with terrible performance on the fastest commericially available card, then runs great 2 years later when the next gen of cards arrives ("Can it run crysis?")
deviantbit 2y ago
"I believe there are two main things holding it back." He really science’d the heck out of that one. I’m getting tired of seeing opinions dressed up as insight—especially when they’re this detached from how real systems actually work. I worked on the Cell processor and I can tell you it was a nightmare. It demanded an unrealistic amount of micromanagement and gave developers rope to hang themselves with. There’s a reason it didn’t survive. What amazes me more is the comment section—full of people waxing nostalgic for architectures they clearly never had to ship stable software on. They forget why we moved on. Modern systems are built with constraints like memory protection, isolation, and stability in mind. You can’t just “flatten address spaces” and ignore the consequences. That’s how you end up with security holes, random crashes, and broken multi-tasking. There's a whole generation of engineers that don't seem to realize why we architected things this way in the first place. I will take how things are today over how things used to be in a heart beat. I really believe I need to spend 2-weeks requiring students write code on an Amiga, and the programs have to run at the same time. If anyone of them crashes, they all will fail my course. A new found appreciation may flourish.
- api 2y ago
  On flattening address spaces: the road not taken here is to run everything in something akin to the JVM, CLR, or WASM. Do that stuff in software not hardware. You could also do things like having the JIT optimize the entire running system dynamically like one program, eliminating syscall and context switch overhead not to mention most MMU overhead. Would it be faster? Maybe. The JIT would have to generate its own safety and bounds checking stuff. I’m sure some work loads would benefit a lot and others not so much. What it would do is allow CPUs to be simpler, potentially resulting in cheaper lower power chips or more cores on a die with the same transistor budget. It would also make portability trivial. Port the core kernel and JIT and software doesn’t care.
  - zozbot234 2y ago
    > On flattening address spaces: the road not taken here is to run everything in something akin to the JVM, CLR, or WASM. GPU drivers take SPIR-V code (either "kernels" for OpenCL/SYCL drivers, or "shaders" for Vulkan Compute) which is not that different at least in principle. There is also a LLVM-based soft-implementation that will just compile your SPIR-V code to run directly on the CPU.
amelius 2y ago
Isn't the ONNX standard already going into the direction of programming a GPU using a computation graph? Could it be made more general?
- sitkack 2y ago
  It lacks support for the serial portions of the execution graph, but yes. You should play around with ONNX, it can be used for a lot more than just ML stuff.
  - amelius 2y ago
    What do you mean by serial portions? Aren't operations automatically serialized if there are dependencies between them?
    - sitkack 2y ago
      s/serial/scalar ONNX doesn't have the direct capabilities to be the compilation target for regular imperative code.
casey2 2y ago
I think Tim was right, it's 2025, Nvidia just released their 50 series, but I don't see any cards, let alone GPUs.
api 2y ago
I implemented some evolutionary computation stuff on the Cell BE in college. It was a really interesting machine and could be very fast for its time but it was somewhat painful to program. The main cores were PPC and the Cell cores were… a weird proprietary architecture. You had to write kernels for them like GPGPU, so in that sense it was similar. You couldn’t use them seamlessly or have mixed work loads easily. Larrabee and Xeon Phi are closer to what I’d want. I’ve always wondered about many—many-many-core CPUs too. How many tiny ARM32 cores could you put on a big modern 5nm die? Give each one local RAM and connect them with an on die network fabric. That’d be an interesting machine for certain kinds of work loads. It’d be like a 1990s or 2000s era supercomputer on a chip but with much faster clock, RAM, and network.
sitkack 2y ago
This essay needs more work. Are you arguing for a better software abstraction, a different hardware abstraction or both? Lots of esoteric machines are name dropped, but it isn't clear how that helps your argument. Why not link to Vello? https://github.com/linebender/vello https://github.com/linebender/vello I think a stronger essay would at the end give the reader a clear view of what Good means and how to decide if a machine is closer to Good than another machine and why. SIMD machines can be turned into MIMD machines. Even hardware problems still need a software solution. The hardware is there to offer the right affordances for the kinds of software you want to write. Lots of words that are in the eye of beholder. We need a checklist or that Good parallel computer won't be built.
- winwang 2y ago
  Personal opinion: it's the software (and software tooling). The hardware is good enough (even if we're only talking 10x efficiency). Part of the issue seems slightly cultural, i.e. repetitively putting down the idea of traditional task parallelism (not-super-SIMD/data parallelism) on GPUs. Obviously, one would lose a lot of efficiency if we literally ran 1 thread per warp. But it could be useful for lightly-data-parallel tasks (like typical CPU vectorization), or maybe using warp-wide semantics to implement something like a "software" microcode engine. Dumb example: implementing division with long division using multiplications and shifts. Other things a GPU gives: insanely high memory bandwidth, programmable cache (shared memory), and (relatively) great atomic operations.
  - sitkack 2y ago
    I agree. Many things in software are in the "you're doing it wrong" but that wrong way is subjective and arbitrary. > maybe using warp-wide semantics to implement something like a "software" microcode engine. https://github.com/beehive-lab/ProtonVM https://github.com/beehive-lab/ProtonVM
    - winwang 2y ago
      Thanks for the share (and reminder)! Turns out I had this bookmarked somehow, lol.
0xbadcafebee 2y ago
If we had distributed operating systems and SSI kernels, your computer could use the idle cycles of other computers [that aren't on battery power]. People talk about a grid of solar houses, but we could've had personal/professional grid computing like 15 years ago. Nobody wanted to invest in it, I guess because chips kept getting faster.
- zozbot234 2y ago
  SSI is an interesting idea, but the actual advantage is mostly to improve efficiency when running your distributed code on a single, or few nodes. You still have to write your code with some very real awareness of the relevant issues when running on many nodes, but now you are also free to "scale down" and be highly efficient on a single node, since your code is still "natively" written for running on that kind of system. You are not going to gain much by opportunistically running bad single-node codes on larger systems, since that will be quite inefficient anyway. Also, running a large multi-node SSI system means you mostly can't partition those nodes ever, otherwise the two now-separated sets of nodes could both progress in ways that cannot be cleanly reconciled later. This is not what people expect most of the time when connecting multiple computers together.
  - 0xbadcafebee 2y ago
    You could say the same thing about multiple cores or CPUs. A lot of people write apps that aren't useful past a single core or CPU. Doesn't mean we don't build OSes & hardware for multiple cores... (Remember back when nobody had an SMP kernel, because, hey, who the hell's writing their apps for more than one CPU?! Our desktops aren't big iron!) In the worst-case, your code is just running on the CPU you already have. If you have another node/CPU, you can schedule your whole process on that one, which frees up your current CPU for more work. If you design your app to be more scalable to more nodes/CPUs, you get more benefits. So even in the worst case, everything would just be... exactly the way it is today. But there are many cases that would be benefited, and once the platform is there, more people would take advantage of it. There is still a massive opportunity in general parallel computing that we haven't explored. Plenty of research, but along specific kinds of use cases, and with not nearly enough investment, so the little work that got done took decades. I think we could solve all the problems and make it generally useful, which could open up a whole new avenue of computing / applications, the way more bandwidth did. (I'm referring to consumer use-cases above, but in the server world alone, a distributed OS with simple parallel computing would transform billion-dollar markets in software, making a whole lot of complicated solutions obsolete. It might take a miracle for the code to get adopted upstream by the Linux Mafia, though)
ip26 2y ago
I believe there are two main things holding it back. One is an impoverished execution model, which makes certain tasks difficult or impossible to do efficiently; GPUs … struggle when the workload is dynamic This sacrifice is a purposeful cornerstone of what allows GPUs to be so high throughput in the first place.
dragontamer 2y ago
There's a lot here that seems to misunderstand GPUs and SIMD. Note that raytracing is a very dynamic problem, where the GPU isn't sure if a ray hits a geometry or if it misses. When it hits, the ray needs to bounce, possibly multiple times. Various implementations of raytracing, recursion, dynamic parallelism or whatever. Its all there. Now the software / compilers aren't ready (outside of specialized situations like Microsofts DirectX Raytracing, which compiles down to a very intriguing threading model). But what was accomplished with DirectX can be done in other situations. ------- Connection Machine is before my time, but there's no way I'd consider that 80s hardware to be comparable to AVX2 let alone a modern GPU. Connection Machine was a 1-bit computer for crying out loud, just 4096 of them in parallel. Xeon Phi (70 core Intel Atoms) is slower and weaker than 192 core Modern EPYC chips. ------- Today's machines are better. A lot better than the past machines. I cannot believe any serious programmer would complain about the level of parallelism we have today and wax poetic about historic and archaic computers.
- raphlinus 2y ago
  The problems I'm having are very different than those for raytracing. Sure, it's dynamic, but at a fine granularity, so the problems you run into are divergence, and often also wanting function pointers, which don't work well in a SIMT model, By contrast, the way I'm doing 2D there's basically no divergence (monoids are cool that way) but there is a need to schedule dynamically at a coarser (workgroup) level. But the biggest problem I'm having is management of buffer space for intermediate objects. That's not relevant to the core of raytracing because you're fundamentally just accumulating an integral, then writing out the answer for a single pixel at the end. The problem with the GPU raytracing work is that they built hardware and driver support for the specific problem, rather than more general primitives on which you could build not only raytracing but other applications. The same story goes for video encoding. Continuing that direction leads to unmanageable complexity. Of course today's machines are better, they have orders of magnitude more transistors, and crystallize a ton of knowledge on how to build efficient, powerful machines. But from a design aesthetic perspective, they're becoming junkheaps of special-case logic. I do think there's something we can learn from the paths not taken, even if, quite obviously, it doesn't make sense to simply duplicate older designs.
nromiun 2y ago
What about unified memory? I know these APUs are slower than traditional GPUs but still it seems like the simpler programming model will be worth it. The biggest problem is that most APUs don't even support full unified memory (system SVM in OpenCL). From my research only Apple M series, some Qualcomm Adreno and AMD APUs support them.
throwawayabcdef 2y ago
The AIE arrays on Versal and Ryzen with XDNA are a big grid of cores (400 in an 8 x 50 array) that you program with streaming work graphs. https://docs.amd.com/r/en-US/am009-versal-ai-engine/Overview https://docs.amd.com/r/en-US/am009-versal-ai-engine/Overview Each AIE tile can stream 64 Gbps in and out and perform 1024 bit SIMD operations. Each shares memory with its neighbors and the streams can be interconnected in various ways.
joshu 2y ago
Huh. The Blelloch mentioned n the thinking machines section taught my parallel algorithms class in 1994 or so.
mikewarot 2y ago
Any computing model that tries to parallelize von Neumann machines, that is, has program counters or address space, just isn't going to scale.
- imtringued 2y ago
  The problem isn't address space or program counters. It's that each processor is going to need instruction memory stored in SRAM or an extremely efficient multi port memory for a shared instruction cache. GPUs get around this limitation by executing identical instructions over multiple threads.
  - mikewarot 2y ago
    Instructions are the problem, you have to have an architecture which just operates on data flows all in parallel and all at once, like an FPGA, but without all the fiddly special sauce parts.
nickpsecurity 2y ago
There are designs like Tilera and Phalanx that have tons of cores. Then, NUMA machines used to have 128-256 sockets in one machine with coherent memory. The SGI machines let you program them like it was one machine. Languages like Chapel were designed to make parallel programming easier. Making more things like that with lowest, possible, unit prices could help a lot.
scroot 2y ago
When this topic comes up, I always think of uFork [1]. They are even working on an FPGA prototype. [1] https://ufork.org/ https://ufork.org/
pikuseru 2y ago
No mention of the Transputer :(
Quis_sum 2y ago
Clearly the author never worked with a CM2 - I did though. The CM2 was more like a co-processor which had to be controlled by a (for that age) rather beefy SUN workstation/server. The program itself ran on the workstation which then sent the data-parallel instructions to the CM2. The CM2 was an extreme form of a MIMD design (that is why it was called data parallel). You worked with a large rectangular array (I cannot recall up to how many dimensions) which had to be a multiple of the physical processors (in your partition). All cells typically performed exactly the same operation. If you wanted to perform an operation on a subset, you had to "mask" the other cells (which were essentially idling during that time). That is hardly what the author describes.
- pjmlp 2y ago
  Did you used StarLisp? It is always a bit hard to find testimonials about the experience.
eternityforest 2y ago
I wonder if CDN server applications could use something like this, if every core had a hardware TCP/TLS stack and there was a built-in IP router to balance the load, or something like that.
muziq 2y ago
I was always fascinated by the prospects of the 1024-core Epiphany-V from Parallella.. https://parallella.org/2016/10/05/epiphany-v-a-1024-core-64-bit-risc-processor/ https://parallella.org/2016/10/05/epiphany-v-a-1024-core-64-... But it seems whatever the DARPA connection was has led to it not being for scruffs like me and is likely powering god knows what military systems..
Ericson2314 2y ago
Agreed with the premise here I have never done GPU programming or graphics, but what feels frustating looking from the outside is the designs and constraints seems so arbitrary. They don't feel like they come from actual hardware constraints/problems. It just looks like pure path dependency going all the way back to the fixed-function days, with tons of accidental complexity and and half-finished generalizations ever since.
chimyy 2y ago
[flagged]
chimyy 2y ago
[flagged]
chimyy 2y ago
[flagged]
Wumpnot 2y ago
I had hoped the GPU API would go away, and the entire thing would become fully programmable, but so far we just keep using these shitty APIs and horrible shader languages. Personally I would like to use the same language I write the application in to write the rendering code(C++). Preferably with shared memory, not some separate memory system that takes forever to transfer anything too. Somelike along the lines of the new AMD 360 Max chips, but graphics written in explicit C++.
hackburg 2y ago
[dead]
SergeAx 2y ago
The thing is that most of our everyday software will not benefit from parallelism. What we really have a use for is concurrency, which is a totally different beast.

233 points by raphlinus 2y ago

> The GPU in your computer is about 10 to 100 times more powerful than the CPU, depending on workload. For real-time graphics rendering and machine learning, you are enjoying that power, and doing those workloads on a CPU is not viable. Why aren’t we exploiting that power for other workloads? What prevents a GPU from being a more general purpose computer? What other workloads would benefit from a GPU? Computers are so fast that in practice, many tasks don't need more performance. If a program that runs those tasks is slow, it's because that program's code is particularly bad, and the solution to make the code less bad is simpler than re-writing it for the GPU. For example, GUIs have been imperceptibly reactive to user input for over 20 years. If an app's GUI feels sluggish, the problem is that the app's actions and rendering aren't on separate coroutines, or the action's coroutine is blocking (maybe it needs to be on a separate thread). But the rendering part of the GUI doesn't need to be on a GPU (any more than it is today, I admit I don't know much about rendering), because responsive GUIs exist today, some even written in scripting languages. In some cases, parallelizing a task intrinsically makes it slower, because the number of sequential operations required to handle coordination mean there are more forced-sequential operations in total. In other cases, a program spawns 1000+ threads but they only run on 8-16 processors, so the program would be faster if it spawned less threads because it would still use all processors. I do think GPU programming should be made much simpler, so this work is probably useful, but mainly to ease the implementation of tasks that already use the GPU: real-time graphics and machine learning.

wmf 2y ago
A big one is video encoding. It seems like GPUs would be ideal for it but in practice limitations in either the hardware or programming model make it hard to efficiently run on GPU shader cores. (GPUs usually include separate fixed-function video engines but these aren't programmable to support future codecs.)
- dist-epoch 2y ago
  Video encoding is done with fixed-function for power efficiency. A new popular codec like H26x codec appears every 5-10 years, there is no real need to support future ones.
  - nwallin 2y ago
    Video encoding is two domains. And there's surprisingly little overlap between them. You have your real time video encoding. This is video conferencing, live television broadcasts. This is done fixed-function not just for power efficiency, but also latency. The second domain is encoding at rest. This is youtube, netflix, blu-ray, etc. This is usually done in software on the CPU for compression ratio efficiency. The problem with fixed function video encoding is that the compression ratio is bad. You either have enormous data, or awful video quality, or both. The problem with software video encoding is that it's really slow. OP is asking why we can't/don't have the best of both worlds. Why can't/don't we write a video encoder in OpenCL/CUDA/ROCm. So that we have the speed of using the GPU's compute capability but compression ratio of software.
  - deleted 2y ago
    [deleted]
raphlinus 2y ago
Possibly compilation and linking. That's very slow for big programs like Chromium. There's really interesting work on GPU compilers (co-dfns and Voetter's work). Optimization problems like scheduling and circuit routing. Search in theorem proving (the classical parts like model checking, not just LLM). There's still a lot that is slow and should be faster, or at the very least made to run using less power. GPUs are good at that for graphics, and I'd like to see those techniques applied more broadly.
- return_to_monke 2y ago
  All of these things you mention are "thinking", meaning they require complex algorithms with a bunch of branches and edge cases. The tasks that GPUs are good at right now - graphics, number crunching, etc - are all very simple algorithms at the core (mostly elementary linear algebra), and the problems are, in most cases, embarassingly parallel. CPUs are not very good at branching either - see all the effort being put towards getting branch prediction right - but they are way better at it than GPUs. The main appeal of GPGPU programming is, in my opinion, that if you can get the CPU to efficiently divide the larger problem into a lot of small, simple subtasks, you can achieve faster speeds. You mentioned compilers. See a related example, for reference all the work Daniel Lemire has been doing on SIMD parsing: the algorithms he (co)invented are all highly specialized to the language, and highly nontrivial. Branchless programming requires an entirely different mindset/intuition than "traditional" programming, and I wouldn't expect the average programmer to come up with such novel ideas. A GPU is a specialized tool that is useful for a particular purpose, not a silver bullet to magically speed up your code. Theree is a reason that we are using it for its current purposes.
- hulitu 2y ago
  > Possibly compilation and linking. That's very slow for big programs like Chromium. So instead of fixing the problem (Chromium's bloat) we just trow more memory and computing power at it, hopping that the problem will go away. Maybe we shall teach programmers to programm. /s

IshKebab 2y ago

Having worked for a company that made a "hundreds of small CPUs on a single chip", I can tell you now that they're all going to fail because the programming model is too weird, and nobody will write software for them. Whatever comes next will be a GPU with extra capabilities, not a totally new architecture. Probably an nVidia GPU.

bryanlarsen 2y ago
While acknowledging that it's theoretically possible other approaches might succeed, it seems quite clear the author agrees with you.
convolvatron 2y ago
my take from reading this is more about programming abstractions than any particular hardware instantiation. the part of the Connection Machine that remains interesting is not building machines with CPUS with transistor counts in the hundreds running off a globally synchronous clock, but that there were a whole family of SIMD languages and let you do general purpose programming in parallel. And that those language were still relevant when the architecture changed to a MIMD machine with a bunch of vector units behind each CPU.
snovymgodym 2y ago
Reminds me of Itanium
- CyberDildonics 2y ago
  How is that at all like Itanium except for the superficial headline level where people say they are hard to program?
  - snovymgodym 2y ago
    Because the main feature that made Itanium hard to program for was its explicit instruction-level parallelism.

svmhdvn 2y ago

I've always admired the work that the team behind https://www.greenarraychips.com/ https://www.greenarraychips.com/ does, and the GA144 chip seems like a great parallel computing innovation.

bee_rider 2y ago

It is odd that he talks about Larabee so much, but doesn’t mention the Xeon Phis. (Or is it Xeons Phi?). > As a general trend, CPU designs are diverging into those optimizing single-core performance (performance cores) and those optimizing power efficiency (efficiency cores), with cores of both types commonly present on the same chip. As E-cores become more prevalent, algorithms designed to exploit parallelism at scale may start winning, incentivizing provision of even larger numbers of increasingly efficient cores, even if underpowered for single-threaded tasks. I’ve always been slightly annoyed by the concept of E cores, because they are so close to what I want, but not quite there… I want, like, throughput cores. Let’s take E cores, give them their AVX-512 back, and give them higher throughput memory. Maybe try and pull the Phi trick of less OoO capabilities but more threads per core. Eventually the goal should be to come up with an AVX unit so big it kills iGPUs, haha.

nullpoint420 2y ago
I've always wondered if you could use iGPU compute cores with unified memory as "transparent" E-cores when needed. Something like OpenCL/CUDA except it works with pthreads/goroutines and other (OS) kernel threading primitives, so code doesn't need to be recompiled for it. Ideally the OS scheduler would know how to split the work, similar to how E-core and P-core scheduling works today. I don't do HPC professionally, so I assume I'm ignorant to why this isn't possible.
Retr0id 2y ago
Isn't Xeon Phi just an instance of Larrabee?
- deleted 2y ago
  [deleted]
- adrian_b 2y ago
  It is an instance of Larrabee in the same sense as AMD Zen 4 is an instance of Larrabee. The "Larrabee New Instructions" is an instruction set that has been designed before AVX and also its first hardware implementation has been introduced before AVX, in 2010 (AVX was launched in 2011, with Sandy Bridge). Unfortunately while the hardware design of Sandy Bridge with the inferior AVX ISA has been done by the Intel A team, the hardware implementations of Larrabee have been done by some C or D teams, which were also not able to design new CPU cores for it, but they had to reuse some obsolete x86 cores, initially a Pentium core and later an Atom Silvermont core, to which the Larrabee instructions were grafted. "Larrabee New Instructions" have been renamed to "Many Integrated Cores" ISA, then to AVX-512, while passing through 3 generations of chips, Knights Ferry, Knights Corner and Knights Landing. A fourth generation, Knights Mill, was only intended for machine learning/AI applications. The successor of Knights Landing has been Skylake Server, when the AVX-512 ISA has come to standard Xeons, marking the disappearance of Xeon Phi. Already in 2013, Intel Haswell has added to AVX a few of the more important instructions that were included in the Larrabee New Instructions, but which were missing in AVX, e.g. fused multiply-add and gather instructions. The 3-address FMA format, which has caused problems to AMD, who had implemented in Bulldozer a 4-address format, has also come to AVX from Larrabee, replacing the initial 4-address specification. At each generation until Skylake Server, some of the original Larrabee instructions have been deleted, by assuming that they might be needed only for graphics, which was no longer the intended market. However a few of those instructions were really useful for some applications in which I am interested, e.g. for computations with big numbers, so I regret their disappearance. Since Skylake Server, there have been no other instruction removals, with the exception of those introduced by Intel Tiger Lake, which are now supported only by AMD Zen 5. A few days ago Intel has committed to keeping complete compatibility in the future with the ISA implemented today by Granite Rapids, so there will be no other instruction deletions.

andrewstuart 2y ago

AMD Strix Halo APU is a CPU with very powerful integrated GPU. It’s faster at AI than an Nvidia RTX4090, because 96GB of the 128GB can be allocated to the GPU memory space. This means it’s doesn’t have the same swapping/memory thrashing that a discrete GPU experiences when processing large models. 16 CPU cores and 40 GPU compute units sounds pretty parallel to me. Doesn’t that fit the bill?

dr_kiszonka 2y ago
It looks like it will be available in the Framework Desktop! I would love to see it in a more budget mini PC at some point from another company. (Framework is great but not in my price range.)
bigyabai 2y ago
> It’s faster at AI than an Nvidia RTX4090, because 96GB of the 128GB can be allocated to the GPU memory space I love AMD's Ryzen chips and will recommend their laptops over an Nvidia model all day. However, this is a pretty facetious comparison that falls apart when you normalize the memory. Any chip can be memory bottlenecked, and if we take away that arbitrary precondition the Strix Halo gets trounced in terms of compute capacity. You can look at the TDP of either chip and surmise this pretty easily.
- dist-epoch 2y ago
  > However, this is a pretty facetious comparison that falls apart when you normalize the memory Why would you normalize though? You can't buy a 96 GB RTX4090. So it's fair to compare the whole deal, slowish APU with large RAM versus very fast GPU with limited RAM.
  - Animats 2y ago
    > You can't buy a 96 GB RTX4090 You can now buy a 96 GB RTX5090.[1] NVidia gives it a "Pro" designation and charges more, but it's the same chip. [1] https://www.tomshardware.com/pc-components/gpus/nvidia-rtx-pro-6000-up-close-blackwell-rtx-workstation-max-q-workstation-and-server-variants-shown https://www.tomshardware.com/pc-components/gpus/nvidia-rtx-p...

grg0 2y ago

The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths. It's an entirely moronic programming model to be using in 2025. - You need to compile shader source/bytecode at runtime; you can't just "run" a program. - On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better design an algorithm that does not require immediate synchronization between the two. - You need to synchronize data access between CPU-GPU and GPU workloads. - You need to deal with bad and confusing APIs because there is no standardization of the underlying hardware. - You need to deal with a combinatorial turd explosion of configurations. HW vendors want to protect their turd, so drivers and specs are behind fairly tight gates. OS vendors also want to protect their turd and refuse even the software API standard altogether. And then the tooling also sucks. What I would like is a CPU with a highly parallel array of "worker cores" all addressing the same memory and speaking the same goddamn language that the CPU does. But maybe that is an inherently crappy architecture for reasons that are beyond my basic hardware knowledge.

Grosvenor 2y ago
What I want is a Linear Algebra interface - As Gilbert Strang taught it. I'll "program" in LinAlg, and a JIT can compile it to whatever wonky way your HW requires. I'm not willing to even know about the HW at all, the higher level my code the more opportunities for the JIT to optimize my code. What I really want is something like Mathematica that can JIT to GPU. As another commenter mentioned all the API's assume you're a discrete GPU off the end of a slow bus, without shared memory. I would kill for an APU that could freely allocate memory for GPU or CPU and change ownership with the speed of a pagefault or kernel transition.
- RossBencina 2y ago
  > What I really want is something like Mathematica that can JIT to GPU. https://juliagpu.org/ https://juliagpu.org/
- vgatherps 2y ago
  https://github.com/jax-ml/jax https://github.com/jax-ml/jax

helf 2y ago

[dead]

Retr0id 2y ago

Something that frustrates me a little is that my system (apple silicon) has unified memory, which in theory should negate the need to shuffle data between CPU and GPU. But, iiuc, the GPU programming APIs at my disposal all require me to pretend the memory is not unified - which makes sense because they want to be portable across different hardware configurations. But it would make my life a lot easier if I could just target the hardware I have, and ignore compatibility concerns.

deviantbit 2y ago
Unified memory doesn't mean unified address space. It frustrates me when no one understands unified memory.
- morphle 2y ago
  If you fix the pages tables (partial tutorial online) you can have continuous unified address space on Apple Silicon.
  - deviantbit 2y ago
    Let’s be honest, saying “just fix the page tables” is like telling someone they can fly if they “just rewrite gravity.” Yes, on Apple Silicon, the hardware supports shared physical memory, and with enough “convincing”, you can rig up a contiguous virtual address space for both the CPU and GPU. Apple’s unified memory architecture makes that possible, but Apple’s APIs and memory managers don’t expose this easily or safely for a reason. You’re messing with MMU-level mappings on a tightly integrated system that treats memory as a first-class citizen of the security model. I can tell you never programmed on an Amiga.
    - morphle 2y ago
      Oh yes I programmed all the Amiga models, mostly in assembly level. I reprogrammed the ROMs. I also published a magazine on all the Commodore computers internals and build lots of hardware for these machines. We had the parallel Inmos Transputer systems during the heyday of the Amiga, they where much better designed than any the custom Amiga chips.

morphle 2y ago

I haven't yet read the full blog post but so far my response is you can have this good parallel computer. See my previous HN comments the past months on building an M4 Mac mini supercomputer. For example reverse engineering the Apple M3 Ultra GPU and Neural Engine instruction set and IOMMU and pages tables that prevent you from programming all processor cores in the chip (146 cores to over ten thousand depending on how you delineate what a core is) and making your own Abstract Syntax Tree to assembly compiler for these undocumented cores will unleash at least 50 trillion operations per second. I still have to benchmark this chip and make the roofline graphs for the M4 to be sure, it might be more. https://en.wikipedia.org/wiki/Roofline_model https://en.wikipedia.org/wiki/Roofline_model

Animats 2y ago

Interesting article. Other than as an exercise, it's not clear why someone would write a massively parallel 2D renderer that needs a GPU. Modern GPUs are overkill for 2D. Now, 3D renderers, we need all the help we can get. In this context, a "renderer" is something that takes in meshes, textures, materials, transforms, and objects, and generates images. It's not an entire game development engine, such as Unreal, Unity, or Bevy. Those have several more upper levels above the renderer. Game engines know what all the objects are and what they are doing. Renderers don't. Vulkan, incidentally, is a level below the renderer. Vulkan is a cross-hardware API for asking a GPU to do all the things a GPU can do. WGPU for Rust, incidentally, is an wrapper to extend that concept to cross-platform (Mac, Android, browsers, etc.) While it seems you can write a general 3D renderer that works in a wide variety of situations, that does not work well in practice. I wish Rust had one. I've tried Rend3 (abandoned), and looked at Renderling (in progress), Orbit (abandoned), and Three.rs (abandoned). They all scale up badly as scene complexity increases. There's a friction point in design here. The renderer needs more info to work efficiently than it needs to just draw in a dumb way. Modern GPSs are good enough that a dumb renderer works pretty well, until the scene complexity hits some limit. Beyond that point, problems such as lighting requiring O(lights * objects) time start to dominate. The CPU driving the GPU maxes out while the GPU is at maybe 40% utilization. The operations that can easily be parallelized have been. Now it gets hard. In Rust 3D land, everybody seems to write My First Renderer, hit this wall, and quit. The big game engines (Unreal, etc.) handle this by using the scene graph info of the game to guide the rendering process. This is visually effective, very complicated, prone to bugs, and takes a huge engine dev team to make work. Nobody has a good solution to this yet. What does the renderer need to know from its caller? A first step I'm looking at is something where, for each light, the caller provides a lambda which can iterate through the objects in range of the light. That way, the renderer can get some info from the caller's spatial data structures. May or may not be a good idea. Too early to tell. [1] https://github.com/linebender/vello/ https://github.com/linebender/vello/

amelius 2y ago
> Other than as an exercise, it's not clear why someone would write a massively parallel 2D renderer that needs a GPU. Modern GPUs are overkill for 2D. Depends on how complicated your artwork is.

dekhn 2y ago

There are many intertwined issues here. One of the reasons we can't have a good parallel computer is that you need to get a large number of people to adopt your device for development purposes, and they need to have a large community of people who can run their code. Great projects die all the time because a slightly worse, but more ubiquitous technology prevents flowering of new approaches. There are economies of scale that feed back into ever-improving iterations of existing systems. Simply porting existing successful codes from CPU to GPU can be a major undertaking and if there aren't any experts who can write something that drive immediate sales, a project can die on the vine. See for example https://en.wikipedia.org/wiki/Cray_MTA https://en.wikipedia.org/wiki/Cray_MTA when I was first asked to try this machine, it was pitched as "run a million threads, the system will context switch between threads when they block on memory and run them when the memory is ready". It never really made it on its own as a supercomputer, but lots of the ideas made it to GPUs. AMD and others have explored the idea of moving the GPU closer to the CPU by placing it directly onto the same memory crossbar. Instead of the GPU connecting to the PCI express controller, it gets dropped into a socket just like a CPU. I've found the best strategy is to target my development for what the high end consumers are buying in 2 years - this is similar to many games, which launch with terrible performance on the fastest commericially available card, then runs great 2 years later when the next gen of cards arrives ("Can it run crysis?")

deviantbit 2y ago

"I believe there are two main things holding it back." He really science’d the heck out of that one. I’m getting tired of seeing opinions dressed up as insight—especially when they’re this detached from how real systems actually work. I worked on the Cell processor and I can tell you it was a nightmare. It demanded an unrealistic amount of micromanagement and gave developers rope to hang themselves with. There’s a reason it didn’t survive. What amazes me more is the comment section—full of people waxing nostalgic for architectures they clearly never had to ship stable software on. They forget why we moved on. Modern systems are built with constraints like memory protection, isolation, and stability in mind. You can’t just “flatten address spaces” and ignore the consequences. That’s how you end up with security holes, random crashes, and broken multi-tasking. There's a whole generation of engineers that don't seem to realize why we architected things this way in the first place. I will take how things are today over how things used to be in a heart beat. I really believe I need to spend 2-weeks requiring students write code on an Amiga, and the programs have to run at the same time. If anyone of them crashes, they all will fail my course. A new found appreciation may flourish.

api 2y ago
On flattening address spaces: the road not taken here is to run everything in something akin to the JVM, CLR, or WASM. Do that stuff in software not hardware. You could also do things like having the JIT optimize the entire running system dynamically like one program, eliminating syscall and context switch overhead not to mention most MMU overhead. Would it be faster? Maybe. The JIT would have to generate its own safety and bounds checking stuff. I’m sure some work loads would benefit a lot and others not so much. What it would do is allow CPUs to be simpler, potentially resulting in cheaper lower power chips or more cores on a die with the same transistor budget. It would also make portability trivial. Port the core kernel and JIT and software doesn’t care.
- zozbot234 2y ago
  > On flattening address spaces: the road not taken here is to run everything in something akin to the JVM, CLR, or WASM. GPU drivers take SPIR-V code (either "kernels" for OpenCL/SYCL drivers, or "shaders" for Vulkan Compute) which is not that different at least in principle. There is also a LLVM-based soft-implementation that will just compile your SPIR-V code to run directly on the CPU.

amelius 2y ago

Isn't the ONNX standard already going into the direction of programming a GPU using a computation graph? Could it be made more general?

sitkack 2y ago
It lacks support for the serial portions of the execution graph, but yes. You should play around with ONNX, it can be used for a lot more than just ML stuff.
- amelius 2y ago
  What do you mean by serial portions? Aren't operations automatically serialized if there are dependencies between them?
  - sitkack 2y ago
    s/serial/scalar ONNX doesn't have the direct capabilities to be the compilation target for regular imperative code.

casey2 2y ago

I think Tim was right, it's 2025, Nvidia just released their 50 series, but I don't see any cards, let alone GPUs.

api 2y ago

I implemented some evolutionary computation stuff on the Cell BE in college. It was a really interesting machine and could be very fast for its time but it was somewhat painful to program. The main cores were PPC and the Cell cores were… a weird proprietary architecture. You had to write kernels for them like GPGPU, so in that sense it was similar. You couldn’t use them seamlessly or have mixed work loads easily. Larrabee and Xeon Phi are closer to what I’d want. I’ve always wondered about many—many-many-core CPUs too. How many tiny ARM32 cores could you put on a big modern 5nm die? Give each one local RAM and connect them with an on die network fabric. That’d be an interesting machine for certain kinds of work loads. It’d be like a 1990s or 2000s era supercomputer on a chip but with much faster clock, RAM, and network.

sitkack 2y ago

This essay needs more work. Are you arguing for a better software abstraction, a different hardware abstraction or both? Lots of esoteric machines are name dropped, but it isn't clear how that helps your argument. Why not link to Vello? https://github.com/linebender/vello https://github.com/linebender/vello I think a stronger essay would at the end give the reader a clear view of what Good means and how to decide if a machine is closer to Good than another machine and why. SIMD machines can be turned into MIMD machines. Even hardware problems still need a software solution. The hardware is there to offer the right affordances for the kinds of software you want to write. Lots of words that are in the eye of beholder. We need a checklist or that Good parallel computer won't be built.

winwang 2y ago
Personal opinion: it's the software (and software tooling). The hardware is good enough (even if we're only talking 10x efficiency). Part of the issue seems slightly cultural, i.e. repetitively putting down the idea of traditional task parallelism (not-super-SIMD/data parallelism) on GPUs. Obviously, one would lose a lot of efficiency if we literally ran 1 thread per warp. But it could be useful for lightly-data-parallel tasks (like typical CPU vectorization), or maybe using warp-wide semantics to implement something like a "software" microcode engine. Dumb example: implementing division with long division using multiplications and shifts. Other things a GPU gives: insanely high memory bandwidth, programmable cache (shared memory), and (relatively) great atomic operations.
- sitkack 2y ago
  I agree. Many things in software are in the "you're doing it wrong" but that wrong way is subjective and arbitrary. > maybe using warp-wide semantics to implement something like a "software" microcode engine. https://github.com/beehive-lab/ProtonVM https://github.com/beehive-lab/ProtonVM
  - winwang 2y ago
    Thanks for the share (and reminder)! Turns out I had this bookmarked somehow, lol.

0xbadcafebee 2y ago

If we had distributed operating systems and SSI kernels, your computer could use the idle cycles of other computers [that aren't on battery power]. People talk about a grid of solar houses, but we could've had personal/professional grid computing like 15 years ago. Nobody wanted to invest in it, I guess because chips kept getting faster.

zozbot234 2y ago
SSI is an interesting idea, but the actual advantage is mostly to improve efficiency when running your distributed code on a single, or few nodes. You still have to write your code with some very real awareness of the relevant issues when running on many nodes, but now you are also free to "scale down" and be highly efficient on a single node, since your code is still "natively" written for running on that kind of system. You are not going to gain much by opportunistically running bad single-node codes on larger systems, since that will be quite inefficient anyway. Also, running a large multi-node SSI system means you mostly can't partition those nodes ever, otherwise the two now-separated sets of nodes could both progress in ways that cannot be cleanly reconciled later. This is not what people expect most of the time when connecting multiple computers together.
- 0xbadcafebee 2y ago
  You could say the same thing about multiple cores or CPUs. A lot of people write apps that aren't useful past a single core or CPU. Doesn't mean we don't build OSes & hardware for multiple cores... (Remember back when nobody had an SMP kernel, because, hey, who the hell's writing their apps for more than one CPU?! Our desktops aren't big iron!) In the worst-case, your code is just running on the CPU you already have. If you have another node/CPU, you can schedule your whole process on that one, which frees up your current CPU for more work. If you design your app to be more scalable to more nodes/CPUs, you get more benefits. So even in the worst case, everything would just be... exactly the way it is today. But there are many cases that would be benefited, and once the platform is there, more people would take advantage of it. There is still a massive opportunity in general parallel computing that we haven't explored. Plenty of research, but along specific kinds of use cases, and with not nearly enough investment, so the little work that got done took decades. I think we could solve all the problems and make it generally useful, which could open up a whole new avenue of computing / applications, the way more bandwidth did. (I'm referring to consumer use-cases above, but in the server world alone, a distributed OS with simple parallel computing would transform billion-dollar markets in software, making a whole lot of complicated solutions obsolete. It might take a miracle for the code to get adopted upstream by the Linux Mafia, though)

ip26 2y ago

I believe there are two main things holding it back. One is an impoverished execution model, which makes certain tasks difficult or impossible to do efficiently; GPUs … struggle when the workload is dynamic This sacrifice is a purposeful cornerstone of what allows GPUs to be so high throughput in the first place.

dragontamer 2y ago

There's a lot here that seems to misunderstand GPUs and SIMD. Note that raytracing is a very dynamic problem, where the GPU isn't sure if a ray hits a geometry or if it misses. When it hits, the ray needs to bounce, possibly multiple times. Various implementations of raytracing, recursion, dynamic parallelism or whatever. Its all there. Now the software / compilers aren't ready (outside of specialized situations like Microsofts DirectX Raytracing, which compiles down to a very intriguing threading model). But what was accomplished with DirectX can be done in other situations. ------- Connection Machine is before my time, but there's no way I'd consider that 80s hardware to be comparable to AVX2 let alone a modern GPU. Connection Machine was a 1-bit computer for crying out loud, just 4096 of them in parallel. Xeon Phi (70 core Intel Atoms) is slower and weaker than 192 core Modern EPYC chips. ------- Today's machines are better. A lot better than the past machines. I cannot believe any serious programmer would complain about the level of parallelism we have today and wax poetic about historic and archaic computers.

raphlinus 2y ago
The problems I'm having are very different than those for raytracing. Sure, it's dynamic, but at a fine granularity, so the problems you run into are divergence, and often also wanting function pointers, which don't work well in a SIMT model, By contrast, the way I'm doing 2D there's basically no divergence (monoids are cool that way) but there is a need to schedule dynamically at a coarser (workgroup) level. But the biggest problem I'm having is management of buffer space for intermediate objects. That's not relevant to the core of raytracing because you're fundamentally just accumulating an integral, then writing out the answer for a single pixel at the end. The problem with the GPU raytracing work is that they built hardware and driver support for the specific problem, rather than more general primitives on which you could build not only raytracing but other applications. The same story goes for video encoding. Continuing that direction leads to unmanageable complexity. Of course today's machines are better, they have orders of magnitude more transistors, and crystallize a ton of knowledge on how to build efficient, powerful machines. But from a design aesthetic perspective, they're becoming junkheaps of special-case logic. I do think there's something we can learn from the paths not taken, even if, quite obviously, it doesn't make sense to simply duplicate older designs.

nromiun 2y ago

What about unified memory? I know these APUs are slower than traditional GPUs but still it seems like the simpler programming model will be worth it. The biggest problem is that most APUs don't even support full unified memory (system SVM in OpenCL). From my research only Apple M series, some Qualcomm Adreno and AMD APUs support them.

throwawayabcdef 2y ago

The AIE arrays on Versal and Ryzen with XDNA are a big grid of cores (400 in an 8 x 50 array) that you program with streaming work graphs. https://docs.amd.com/r/en-US/am009-versal-ai-engine/Overview https://docs.amd.com/r/en-US/am009-versal-ai-engine/Overview Each AIE tile can stream 64 Gbps in and out and perform 1024 bit SIMD operations. Each shares memory with its neighbors and the streams can be interconnected in various ways.

joshu 2y ago

Huh. The Blelloch mentioned n the thinking machines section taught my parallel algorithms class in 1994 or so.

mikewarot 2y ago

Any computing model that tries to parallelize von Neumann machines, that is, has program counters or address space, just isn't going to scale.

imtringued 2y ago
The problem isn't address space or program counters. It's that each processor is going to need instruction memory stored in SRAM or an extremely efficient multi port memory for a shared instruction cache. GPUs get around this limitation by executing identical instructions over multiple threads.
- mikewarot 2y ago
  Instructions are the problem, you have to have an architecture which just operates on data flows all in parallel and all at once, like an FPGA, but without all the fiddly special sauce parts.

nickpsecurity 2y ago

There are designs like Tilera and Phalanx that have tons of cores. Then, NUMA machines used to have 128-256 sockets in one machine with coherent memory. The SGI machines let you program them like it was one machine. Languages like Chapel were designed to make parallel programming easier. Making more things like that with lowest, possible, unit prices could help a lot.

scroot 2y ago

When this topic comes up, I always think of uFork [1]. They are even working on an FPGA prototype. [1] https://ufork.org/ https://ufork.org/

pikuseru 2y ago

No mention of the Transputer :(

Quis_sum 2y ago

Clearly the author never worked with a CM2 - I did though. The CM2 was more like a co-processor which had to be controlled by a (for that age) rather beefy SUN workstation/server. The program itself ran on the workstation which then sent the data-parallel instructions to the CM2. The CM2 was an extreme form of a MIMD design (that is why it was called data parallel). You worked with a large rectangular array (I cannot recall up to how many dimensions) which had to be a multiple of the physical processors (in your partition). All cells typically performed exactly the same operation. If you wanted to perform an operation on a subset, you had to "mask" the other cells (which were essentially idling during that time). That is hardly what the author describes.

pjmlp 2y ago
Did you used StarLisp? It is always a bit hard to find testimonials about the experience.

eternityforest 2y ago

I wonder if CDN server applications could use something like this, if every core had a hardware TCP/TLS stack and there was a built-in IP router to balance the load, or something like that.

muziq 2y ago

I was always fascinated by the prospects of the 1024-core Epiphany-V from Parallella.. https://parallella.org/2016/10/05/epiphany-v-a-1024-core-64-bit-risc-processor/ https://parallella.org/2016/10/05/epiphany-v-a-1024-core-64-... But it seems whatever the DARPA connection was has led to it not being for scruffs like me and is likely powering god knows what military systems..

Ericson2314 2y ago

Agreed with the premise here I have never done GPU programming or graphics, but what feels frustating looking from the outside is the designs and constraints seems so arbitrary. They don't feel like they come from actual hardware constraints/problems. It just looks like pure path dependency going all the way back to the fixed-function days, with tons of accidental complexity and and half-finished generalizations ever since.

chimyy 2y ago

[flagged]

chimyy 2y ago

[flagged]

chimyy 2y ago

[flagged]

Wumpnot 2y ago

I had hoped the GPU API would go away, and the entire thing would become fully programmable, but so far we just keep using these shitty APIs and horrible shader languages. Personally I would like to use the same language I write the application in to write the rendering code(C++). Preferably with shared memory, not some separate memory system that takes forever to transfer anything too. Somelike along the lines of the new AMD 360 Max chips, but graphics written in explicit C++.

hackburg 2y ago

[dead]

SergeAx 2y ago

The thing is that most of our everyday software will not benefit from parallelism. What we really have a use for is concurrency, which is a totally different beast.