3 ms·

I'm in kind of a different place with it on the inference side. I've got these crazy tuned up CUDA kernels that are relatively straightforward to build in isol

I'm in kind of a different place with it on the inference side. I've got these crazy tuned up CUDA kernels that are relatively straightforward to build in isolation and really where all the magic happens, and there's this new CUTLASS 3 stuff and modern C++ can call it all trivially. And then there's this increasingly thin film of torch crap that's just this side of unbuildable and drags in this reference counting and broken setup.py and it's a bunch of up and down projections to the real hacker shit. I'm thinking I'm about one squeeze of the toothpaste tube from just squuezing that junk out and having a nice, clean, well-groomed C++ program that can link anything and link into anything.

pjmlp 1y ago
CUTLASS 4 has first class support for Python.
- saagarjha 1y ago
  In fact I doubt the C++ API will be getting much love moving forward
  - pjmlp 1y ago
    At GTC 2025, NVidia introduced two major changes in CUDA ecosystem. First class support for Python JIT/DSLs across the whole ecosystem. Change the way C++ is used and taught, more focused on standard C++ support and libraries, than low level CUDA extensions. So in a way, I think you're kind of right.
    - benreesman 1y ago
      Nah, their people are way involved in mdarray and ROCm is looking to have the "oh no its broken again" bit flipped off in the RDNA 4/5 cycle. NVIDIA wants Python and C++ people, they want a new thing to moat up on, and they know it has to be legitimately good to defy economic gravity on chips a lot of companies can design and fan now.
      - pjmlp 1y ago
        Intel and AMD don't have anyone but themselves to blame. And Khronos always expecting the community to do the work for their standards, filling in the missing pieces for actually excellence in developer experience.
        benreesman 1y ago
        Well I blame our legislators, our regulators, and a public who tolerates low integrity, low competence leadership. A society gets what it pays for with the standards it sets for its leaders. Out of a great many outcomes a very topical one today is semiconductors, and the outcomes rival any on anything for corrupt, incompetent, and entirely consistent with our speed run of the road to irrelevance.
        bigyabai 1y ago
        You don't seem to know what Khronos even is, at this point. It is a nonprofit consortium, joining the group for the community effort is the only reason it exists. By blaming Khronos you're really just reiterating blame on Intel and AMD with the tacit inclusion of Apple. I suppose you could blame Nvidia for not giving away their IP, but they were a staunch OpenCL supporter from the start.

pjmlp 1y ago

CUTLASS 4 has first class support for Python.

saagarjha 1y ago
In fact I doubt the C++ API will be getting much love moving forward
- pjmlp 1y ago
  At GTC 2025, NVidia introduced two major changes in CUDA ecosystem. First class support for Python JIT/DSLs across the whole ecosystem. Change the way C++ is used and taught, more focused on standard C++ support and libraries, than low level CUDA extensions. So in a way, I think you're kind of right.
  - benreesman 1y ago
    Nah, their people are way involved in mdarray and ROCm is looking to have the "oh no its broken again" bit flipped off in the RDNA 4/5 cycle. NVIDIA wants Python and C++ people, they want a new thing to moat up on, and they know it has to be legitimately good to defy economic gravity on chips a lot of companies can design and fan now.
    - pjmlp 1y ago
      Intel and AMD don't have anyone but themselves to blame. And Khronos always expecting the community to do the work for their standards, filling in the missing pieces for actually excellence in developer experience.
      - benreesman 1y ago
        Well I blame our legislators, our regulators, and a public who tolerates low integrity, low competence leadership. A society gets what it pays for with the standards it sets for its leaders. Out of a great many outcomes a very topical one today is semiconductors, and the outcomes rival any on anything for corrupt, incompetent, and entirely consistent with our speed run of the road to irrelevance.
      - bigyabai 1y ago
        You don't seem to know what Khronos even is, at this point. It is a nonprofit consortium, joining the group for the community effort is the only reason it exists. By blaming Khronos you're really just reiterating blame on Intel and AMD with the tacit inclusion of Apple. I suppose you could blame Nvidia for not giving away their IP, but they were a staunch OpenCL supporter from the start.