> However, it’s not that hard to ignore the ecosystem I'd say this will only work if you have stable results or models to target like Whisper, that's one thing

> However, it’s not that hard to ignore the ecosystem I'd say this will only work if you have stable results or models to target like Whisper, that's one thing, and "going your own way" can help improve portability and stuff; Llama.cpp is another good example. But a lot of the software demand is not driven by that, it's driven by continuously evolving models and needs; a lot of the bloat or whatever you want to call it is a result of that. Besides that, the programming models are moving on. The open source Nvidia Linux driver now enables fully heterogeneous memory management on x86 across the CPU and GPU. This means the GPU and CPU do not need the programmer to enforce memory coherency, perform device-specific allocations, or copy memory; migrations, page table/TLB flushes, etc all work out of the box with no modifications to userspace software. So now your io_uring asynchronous loop can write training data to memory that is implicitly available to the GPU, no matter what memory allocator you're using. It basically means arbitrary CPU compute and arbitrary GPU compute is now composable using the memory substrate (and OS kernel) as a coherent transport/storage layer. On x86/Nvidia, this works on the granularity of a page, but on the Grace Hopper superchip, this is going to take place at the level of a cache line. Multiple Hopper Superchips can be NVLink'd together over infiniband so this works across the cluster. You can drive an entire rack of systems this way and it works. For people actually doing a lot of GPU-specific programming, or deploying models on servers (e.g. for API usage), this is going to be a big deal in the long run, and it started way back when they first introduced unified virtual memory. AMD is moving this way too for their compute stacks, I assume. The compute shader model just isn't evolving for these kinds of needs and it isn't clear it's going to anytime soon.