9 ms·

Nice to see ML folks getting weaned off of Python and using a language that can optimally exploit the underlying hardware and not require setting up a specializ

Nice to see ML folks getting weaned off of Python and using a language that can optimally exploit the underlying hardware and not require setting up a specialized environment to build and run.

xvilka 3y ago
Do you mean Julia language?
codethief 3y ago
That's a rather odd comparison to make. First of all, OP, like llama.cpp, doesn't use the GPU – in contrast to most Python ML code. It's not hard to write Python code that "optimally exploits" the GPU. You might call the GPU a "specialized environment to build and run" but it's arguably much better suited to the problem. Second, OP, like llama.cpp, produced efficient and highly specialized code after it was clear the model being specialized for (StableDiffusion / LLaMa / …) works well. Where Python shines, though, is the prototyping phase when you have yet to find an appropriate model. We have yet to see this sort of easy & convenient prototyping in C++. Now, this is not to take away anything from the fantastic work that's being done by the llama.cpp people (to whom I also count OP) in the "ML on a CPU" space. But the problems being solved are entirely different.
- PcChip 3y ago
  >You might call the GPU a "specialized environment to build and run" but it's arguably much better suited to the problem. I feel like the person you're replying to knows that the GPU is better suited than the CPU to do this task, and your argument doesn't really make sense. I think they were referring to the python venv environment with all the library dependencies as the "specialized environment"
  - jebarker 3y ago
    The point is that as awesome as this repo is it doesn't do much to ween the "ML folks" off of Python since it doesn't provide the flexibility and GPU support that people designing and training DL systems rely on.
    - waynecochran 3y ago
      I’m just encouraged when I see ML libraries not using Python w its environment kludges. Just a step in the right direction.
      - jebarker 3y ago
        I don't disagree that Python environments are a mess. I'm actually a developer on quite a prominent large scale neural network training library and a DL researcher that uses said library. With my developer hat on I like to have minimal dependencies and keep Python scripting as decoupled as possible from the CUDA C++ implementation. With my researcher hat on I don't want to be slowed down by C++ development every time I want to change my model or training pipeline. At least for me, C++ development is slower and more error prone than modifying Python. Obviously doing any heavy lifting in Python is a bad idea. But as a scripting language I think it's good, especially if you keep the environment simple. I don't think the answer for DL training is to dump Python entirely and start over in pure C/C++/Rust/Julia/whatever. Learning C/C++ is too big of an ask for everyone working on the model design and training side and it would slow down progress significantly - most of that work is actually data munging and targeted model tweaks. But I do think there's still a lot that can be done to decouple Python from the underlying engine and yield networks where inference can be run in a minimal dependency environment. There's lots of great people working on all these things.
- two_in_one 3y ago
  Add to that it's only inference code, not training.
- waynecochran 3y ago
  Yeah i make a living in the GPU space. I think my comment comes from colleagues having to hold my hand to set up their ML / Python environments with all of their picadellos. In fact its bad enough that i have to use docker to create an insular environment tailored to their specific setup. And Python is like a 1000 times slower when its not using other libs like numpy.
  - pdntspa 3y ago
    Are they not using venvs or something? It should be as simple as python -m venv venv; ./activate; pip install -r requirements.txt
    - waynecochran 3y ago
      Everyone has their own way to do this. Every step is broken by some unfamiliar dependency that requires special arcane knowledge to fix. Part of me is a grumpy old man that doesn’t gravitate to the shiny new tools that come out every week that the younger devs keep up with :)
      - pdntspa 3y ago
        pip and venv are neither shiny nor new, it's the standard way of doing things for a while. I am an outsider to python and am incredibly thankful for this standardization, because i agree getting python env set up correctly before venv was a huge pain If your guys arent on this I'd suggest you get them on it, it dramatically simplifies setup
        waynecochran 3y ago
        Here is a tiny excerpt try to get dvc to work just so I could get the training weights for deployment ... remember I don't develop much w Python... $ dvc pull Command 'dvc' not found, but can be installed with: sudo snap install dvc $ sudo snap install dvc error: This revision of snap "dvc" was published using classic confinement and thus may perform arbitrary system changes outside of the security sandbox that snaps are usually confined to, which may put your system at risk. If you understand and want to proceed repeat the command including --classic. ok I get dvc installed somehow -- don't remember. Time to get the weights... $ python3 -m dvc pull ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! Finally I just have my colleague manually copy the weights. This kind of thing went for hours.
- segfaultbuserr 3y ago
  > Where Python shines, though, is the prototyping phase when you have yet to find an appropriate model. We have yet to see this sort of easy & convenient prototyping in C++. +1. To produce a highly-optimized C/C++ kernel that utilizes the CPU to the fullest extent, it requires tremendously amount of talent and expertise. For example, not everyone can write a hand-vectorized kernel with AVX2 intrinsics (outside a few specialized applications like 3D graphics, media encoding, and the likes), and even fewer people can exploit the underlying feature of the algorithm for optimization, such as producing usable output at greatly reduced numerical precision. The power of LLM provides strong motivation to drive the brainpower of countless programmers all over the world to do just that. New techniques are proposed and implemented on a monthly basis, with people thinking and applying every possible trick on the LLM optimization problems. In this regard, moving from Python to C is totally reasonable. In comparison, right now I'm working on optimizing a niche open-source scientific simulation kernel with a naive C codebase. Before me, there were hardly any contributors in the last decade. Python has its place because not everyone has a level of resource and expertise comparable to ML. In particular, when the bulk of the data processing of a Python script is in done in a function call to a C++ or FORTRAN kernel like scipy, the differences between naive C and naive Python code (or Julia code if you're following the trend) are not that much, especially when it's a one-off project for just publishing a single paper.
  - hotstickyballs 3y ago
    It’s going to be a tf or PyTorch feature rather than going directly to writing things in C. No point solving this problem only once.
- kwant_kiddo 3y ago
  not sure what "It's not hard to write Python code that "optimally exploits" the GPU", exactly means but Python is so far from exploiting the GPU resources even with C/C++ bindings that it's not even funny. I am sure that HPC folks would have migrated way from FORTRAN and C/C++ long time ago if it was so easy.
  - codethief 3y ago
    I wasn't trying to claim that Python is great at fully exploiting GPU resources on generic GPU tasks. But in ML applications it often does, at least in my experience.
- pjmlp 3y ago
  That Python ML code is calling C++ code running in the GPU, one more reason to use C++ across the whole stack. CERN already used prototyping in C++, with ROOT and CINT, 20 years ago. https://root.cern/ https://root.cern/ Nowadays it is even usable from Netbooks via Xeus. It is more a matter of lack of exposure to C++ interpreters than anything else.
- always2slow 3y ago
  >That's a rather odd comparison to make. First of all, OP, like llama.cpp, doesn't use the GPU When was the last time you looked at llama.cpp? It has supported GPU, GPU+CPU, and distributed inference using OpenMPI for awhile now. It also supports training, as well as negative prompting and grammars! The ease of getting llama.cpp running on just about anything has already started innovation.
fassssst 3y ago
Yup. I would much prefer if every ML model had a simple C inference API that could be called directly from pretty much any language on any platform, without a mess of dependencies and environment setup.
- naillo 3y ago
  ML is such a beautiful and perfect setup for dependency free execution too. It should just be like downloading a mathematical function. I'm glad we're finally embracing that.
intelVISA 3y ago
Wasn't Python originally designed as a language to teach children how to code? Weird to see so many, otherwise intelligent, folks latch onto it. It really doesn't have any redeeming characteristics vs. Common Lisp, or Haskell, to warrant this bizarre popularity imo
- mdp2021 3y ago
  > Wasn't Python originally designed as a language to teach children how to code I think it would be very confusing for a child to start with a language so far away from low-level logic. ...And some people said BASIC was evil. At least what it is doing looks plain and direct.
  - segfaultbuserr 3y ago
    > I think it would be very confusing for a child to start with a language so far away from low-level logic. Depending on the person. For some, it would be very frustrating to start with a language so close to the implementation detail, and so far away from what you want to do. It's very possible that someone might have long lost the motivation before one can do anything non-trivial. I started from Python, to C, to assembly, to 4-layer circuit boards. Whenever I went a level deeper, it feels like opening the inner working of a blackbox that I normally only interacts with pushbuttons on its front panel, but I otherwise is roughly aware of what they do. On the other hand, much of my childhood was spent on tinkering with PCs and servers, including hosting websites and compiling packages from source, so I was already well aware of the basic concepts in computing before I started programming. So, top-down and bottom-up are both absolutely workable, under the right circumstances.
  - tester756 3y ago
    >I think it would be very confusing for a child to start with a language so far away from low-level logic. Why? I started with C++ and when they showed me C# I instantly feel in love cuz I didn't have to deal with unnecessary complexity and annoyances and could focus on pure programming, algorithms, etc.
    - 3y ago
api 3y ago
It’s interesting to me that my CPU can run some of these things in quantized form almost as fast as the GPU. Has the whole thing been all about memory bandwidth all along? In addition to compute the GPU architecture is one that somewhat colocates working memory alongside compute. Units have local memories that sync with global memory. Is that a big part of why GPUs are so good for this?
- brucethemoose2 3y ago
  > Has the whole thing been all about memory bandwidth all along Yeah, sort of. LLMs like llama at a batch size of 1 are hilariously bandwidth bound. Stable Diffusion less so. Its still bandwidth heavy on GPUs, but compute is much more of a bottleneck.
geysersam 3y ago
It's not like any performance significant component of the ML stack is actually implemented in Python. Everything is and has always been cuda, c or c++ under the hood. Python is just the extremely effective glue binding it all together.
- brucethemoose2 3y ago
  Sometimes implementations will spend a little too much time in Python interpretation, but yeah, its largely lower level code. The problem with PyTorch specifically is that (without Triton compilation) pretty much all projects run in eager mode. That's fine for experimentation and demonstrations in papers, but its crazy that its used so much for production without any compilation. It would be like using debug C binaries for production, and they only work with any kind of sane performance on a single CPU maker.
galangalalgol 3y ago
As long as we are language trolling, why would anyone start a greenfield project like this in C++ these days? The android, windows, firefox, and now chrome projects have all begun to shift towards rust and in the case of Android and Firefox, write a significant amounts of their project in rust. Migrating an existing project like that is difficult. The chrome team in particular lamented the difficulty. But starting a new project? If you have a team familiar with performant c++ the speedbump of starting a greenfield project in rust is negligible, and the ergonomic improvements in the build system and the language itself will make up for that in any project that takes more than a few months. For that speed bump you get memory and thread race safety, far better than any stack of c++ analysis tools could ever provide with a tiny fraction of the unit tests you'd write in c++. And you lose no performance.
- mnrlt 3y ago
  C++ has a standard, multiple competing implementations and a largely drama-free community. Does CUDA even have Rust bindings, and if so, are they on the same level as the C++ ones? What do you mean by "the windows projects" that shift towards Rust?
  - galangalalgol 3y ago
    MS has started implementing pieces of windows in rust. If you have windows 11 you are running rust. The cuda bindings are good for ml, but missing for cufft and similar. There are people working on better cuda support, but there are even more people working on vendor agnostic gpgpu using spirv and webgpu. It isn't there yet. Right now you are mostly left to your own bindings unless you are doing ml or blas. Edit: I can't argue about the drama part. The competing compilers will get there. A couple gcc frontends in work, and crane lift as a competing back end for llvm and full self-hosting. There is also miri I guess to emit c? People use that to get rust on the C64 or other niche processors.
    - pjmlp 3y ago
      Yes they started, yet there is enough C++ to rewrite in the 30 years of Windows NT history. Meanwhile, Visual Studio team released better tooling for Unreal in Visual C++.
A-Train 3y ago
Amen.
danybittel 3y ago
Since when does C++ optimally exploit the underlying hardware? It has no vector instructions, does not run on the GPU and is arguably too hard to make multithreaded. Which leaves you with about 0.5% performance of a current PCs.
- jcelerier 3y ago
  > does not run on the GPU both Cuda and the Metal shader language are C++, so is OpenCL since 2.0 (https://www.khronos.org/opencl/ https://www.khronos.org/opencl/), so is AMD ROCm's HIP (https://github.com/ROCm-Developer-Tools/HIP https://github.com/ROCm-Developer-Tools/HIP), so is SYCL (https://www.khronos.org/sycl/ https://www.khronos.org/sycl/)? C++ is pretty much the language that runs most on GPUs. > no vector instructions, There's a thousand different possibilities for SIMD in C++, from #pragma omp simd, to libs such as std::experimental::simd (https://en.cppreference.com/w/cpp/experimental/simd/simd https://en.cppreference.com/w/cpp/experimental/simd/simd), Eve (https://github.com/jfalcou/eve https://github.com/jfalcou/eve), Highway (https://github.com/google/highway https://github.com/google/highway), Vc (https://github.com/VcDevel/Vc https://github.com/VcDevel/Vc)...
- waynecochran 3y ago
  Vector types / instructions would be nice. The C++20 STL algorithms are very friendly to vectorization with the various parallel policies (e.g. std::execution::unsequenced_policy) that open up your code to be vectorized. Wonderful libs like Eigen handle a lot of my numeric needs for linear algebra. I think you are forgetting the CUDA is C/C++.
  - Someone 3y ago
    > Vector types / instructions would be nice It’s technically not a C++ feature, but both gcc (https://gcc.gnu.org/onlinedocs/gcc/Vector-Extensions.html https://gcc.gnu.org/onlinedocs/gcc/Vector-Extensions.html) and Clang (https://releases.llvm.org/3.1/tools/clang/docs/LanguageExtensions.html#vectors https://releases.llvm.org/3.1/tools/clang/docs/LanguageExten...) have vector types, and clang even supports the gcc way of writing them, so it gets pretty close.
aimor 3y ago
I really appreciate the people doing this work. It's the only way I've run these models without any headaches. The difference is so stark, even with CUDA and Linux it's bad, with AMD and Windows it's miserable. I'm pretty sure it's not just me..
gumby 3y ago
It's a bummer there's so little work on the training side in C++. Especially since the python training systems are mostly calls into libraries written in C++!
- pjmlp 3y ago
  Yeah, and since C++17 the language is already quite productive for scripting like workflows, the missing piece of the puzzle is that there are two few C++ REPLs aroung, ROOT/CINT being one of the few well known ones.
  - cozzyd 3y ago
    CINT is now obsolete, replaced by cling (which is way better!).

Nice to see ML folks getting weaned off of Python and using a language that can optimally exploit the underlying hardware and not require setting up a specialized environment to build and run.

xvilka 3y ago

Do you mean Julia language?

codethief 3y ago

That's a rather odd comparison to make. First of all, OP, like llama.cpp, doesn't use the GPU – in contrast to most Python ML code. It's not hard to write Python code that "optimally exploits" the GPU. You might call the GPU a "specialized environment to build and run" but it's arguably much better suited to the problem. Second, OP, like llama.cpp, produced efficient and highly specialized code after it was clear the model being specialized for (StableDiffusion / LLaMa / …) works well. Where Python shines, though, is the prototyping phase when you have yet to find an appropriate model. We have yet to see this sort of easy & convenient prototyping in C++. Now, this is not to take away anything from the fantastic work that's being done by the llama.cpp people (to whom I also count OP) in the "ML on a CPU" space. But the problems being solved are entirely different.

PcChip 3y ago
>You might call the GPU a "specialized environment to build and run" but it's arguably much better suited to the problem. I feel like the person you're replying to knows that the GPU is better suited than the CPU to do this task, and your argument doesn't really make sense. I think they were referring to the python venv environment with all the library dependencies as the "specialized environment"
- jebarker 3y ago
  The point is that as awesome as this repo is it doesn't do much to ween the "ML folks" off of Python since it doesn't provide the flexibility and GPU support that people designing and training DL systems rely on.
  - waynecochran 3y ago
    I’m just encouraged when I see ML libraries not using Python w its environment kludges. Just a step in the right direction.
    - jebarker 3y ago
      I don't disagree that Python environments are a mess. I'm actually a developer on quite a prominent large scale neural network training library and a DL researcher that uses said library. With my developer hat on I like to have minimal dependencies and keep Python scripting as decoupled as possible from the CUDA C++ implementation. With my researcher hat on I don't want to be slowed down by C++ development every time I want to change my model or training pipeline. At least for me, C++ development is slower and more error prone than modifying Python. Obviously doing any heavy lifting in Python is a bad idea. But as a scripting language I think it's good, especially if you keep the environment simple. I don't think the answer for DL training is to dump Python entirely and start over in pure C/C++/Rust/Julia/whatever. Learning C/C++ is too big of an ask for everyone working on the model design and training side and it would slow down progress significantly - most of that work is actually data munging and targeted model tweaks. But I do think there's still a lot that can be done to decouple Python from the underlying engine and yield networks where inference can be run in a minimal dependency environment. There's lots of great people working on all these things.
two_in_one 3y ago
Add to that it's only inference code, not training.
waynecochran 3y ago
Yeah i make a living in the GPU space. I think my comment comes from colleagues having to hold my hand to set up their ML / Python environments with all of their picadellos. In fact its bad enough that i have to use docker to create an insular environment tailored to their specific setup. And Python is like a 1000 times slower when its not using other libs like numpy.
- pdntspa 3y ago
  Are they not using venvs or something? It should be as simple as python -m venv venv; ./activate; pip install -r requirements.txt
  - waynecochran 3y ago
    Everyone has their own way to do this. Every step is broken by some unfamiliar dependency that requires special arcane knowledge to fix. Part of me is a grumpy old man that doesn’t gravitate to the shiny new tools that come out every week that the younger devs keep up with :)
    - pdntspa 3y ago
      pip and venv are neither shiny nor new, it's the standard way of doing things for a while. I am an outsider to python and am incredibly thankful for this standardization, because i agree getting python env set up correctly before venv was a huge pain If your guys arent on this I'd suggest you get them on it, it dramatically simplifies setup
      - waynecochran 3y ago
        Here is a tiny excerpt try to get dvc to work just so I could get the training weights for deployment ... remember I don't develop much w Python... $ dvc pull Command 'dvc' not found, but can be installed with: sudo snap install dvc $ sudo snap install dvc error: This revision of snap "dvc" was published using classic confinement and thus may perform arbitrary system changes outside of the security sandbox that snaps are usually confined to, which may put your system at risk. If you understand and want to proceed repeat the command including --classic. ok I get dvc installed somehow -- don't remember. Time to get the weights... $ python3 -m dvc pull ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! Finally I just have my colleague manually copy the weights. This kind of thing went for hours.
segfaultbuserr 3y ago
> Where Python shines, though, is the prototyping phase when you have yet to find an appropriate model. We have yet to see this sort of easy & convenient prototyping in C++. +1. To produce a highly-optimized C/C++ kernel that utilizes the CPU to the fullest extent, it requires tremendously amount of talent and expertise. For example, not everyone can write a hand-vectorized kernel with AVX2 intrinsics (outside a few specialized applications like 3D graphics, media encoding, and the likes), and even fewer people can exploit the underlying feature of the algorithm for optimization, such as producing usable output at greatly reduced numerical precision. The power of LLM provides strong motivation to drive the brainpower of countless programmers all over the world to do just that. New techniques are proposed and implemented on a monthly basis, with people thinking and applying every possible trick on the LLM optimization problems. In this regard, moving from Python to C is totally reasonable. In comparison, right now I'm working on optimizing a niche open-source scientific simulation kernel with a naive C codebase. Before me, there were hardly any contributors in the last decade. Python has its place because not everyone has a level of resource and expertise comparable to ML. In particular, when the bulk of the data processing of a Python script is in done in a function call to a C++ or FORTRAN kernel like scipy, the differences between naive C and naive Python code (or Julia code if you're following the trend) are not that much, especially when it's a one-off project for just publishing a single paper.
- hotstickyballs 3y ago
  It’s going to be a tf or PyTorch feature rather than going directly to writing things in C. No point solving this problem only once.
kwant_kiddo 3y ago
not sure what "It's not hard to write Python code that "optimally exploits" the GPU", exactly means but Python is so far from exploiting the GPU resources even with C/C++ bindings that it's not even funny. I am sure that HPC folks would have migrated way from FORTRAN and C/C++ long time ago if it was so easy.
- codethief 3y ago
  I wasn't trying to claim that Python is great at fully exploiting GPU resources on generic GPU tasks. But in ML applications it often does, at least in my experience.
pjmlp 3y ago
That Python ML code is calling C++ code running in the GPU, one more reason to use C++ across the whole stack. CERN already used prototyping in C++, with ROOT and CINT, 20 years ago. https://root.cern/ https://root.cern/ Nowadays it is even usable from Netbooks via Xeus. It is more a matter of lack of exposure to C++ interpreters than anything else.
always2slow 3y ago
>That's a rather odd comparison to make. First of all, OP, like llama.cpp, doesn't use the GPU When was the last time you looked at llama.cpp? It has supported GPU, GPU+CPU, and distributed inference using OpenMPI for awhile now. It also supports training, as well as negative prompting and grammars! The ease of getting llama.cpp running on just about anything has already started innovation.

fassssst 3y ago

Yup. I would much prefer if every ML model had a simple C inference API that could be called directly from pretty much any language on any platform, without a mess of dependencies and environment setup.

naillo 3y ago
ML is such a beautiful and perfect setup for dependency free execution too. It should just be like downloading a mathematical function. I'm glad we're finally embracing that.

intelVISA 3y ago

Wasn't Python originally designed as a language to teach children how to code? Weird to see so many, otherwise intelligent, folks latch onto it. It really doesn't have any redeeming characteristics vs. Common Lisp, or Haskell, to warrant this bizarre popularity imo

mdp2021 3y ago
> Wasn't Python originally designed as a language to teach children how to code I think it would be very confusing for a child to start with a language so far away from low-level logic. ...And some people said BASIC was evil. At least what it is doing looks plain and direct.
- segfaultbuserr 3y ago
  > I think it would be very confusing for a child to start with a language so far away from low-level logic. Depending on the person. For some, it would be very frustrating to start with a language so close to the implementation detail, and so far away from what you want to do. It's very possible that someone might have long lost the motivation before one can do anything non-trivial. I started from Python, to C, to assembly, to 4-layer circuit boards. Whenever I went a level deeper, it feels like opening the inner working of a blackbox that I normally only interacts with pushbuttons on its front panel, but I otherwise is roughly aware of what they do. On the other hand, much of my childhood was spent on tinkering with PCs and servers, including hosting websites and compiling packages from source, so I was already well aware of the basic concepts in computing before I started programming. So, top-down and bottom-up are both absolutely workable, under the right circumstances.
- tester756 3y ago
  >I think it would be very confusing for a child to start with a language so far away from low-level logic. Why? I started with C++ and when they showed me C# I instantly feel in love cuz I didn't have to deal with unnecessary complexity and annoyances and could focus on pure programming, algorithms, etc.
  - 3y ago

api 3y ago

It’s interesting to me that my CPU can run some of these things in quantized form almost as fast as the GPU. Has the whole thing been all about memory bandwidth all along? In addition to compute the GPU architecture is one that somewhat colocates working memory alongside compute. Units have local memories that sync with global memory. Is that a big part of why GPUs are so good for this?

brucethemoose2 3y ago
> Has the whole thing been all about memory bandwidth all along Yeah, sort of. LLMs like llama at a batch size of 1 are hilariously bandwidth bound. Stable Diffusion less so. Its still bandwidth heavy on GPUs, but compute is much more of a bottleneck.

geysersam 3y ago

It's not like any performance significant component of the ML stack is actually implemented in Python. Everything is and has always been cuda, c or c++ under the hood. Python is just the extremely effective glue binding it all together.

brucethemoose2 3y ago
Sometimes implementations will spend a little too much time in Python interpretation, but yeah, its largely lower level code. The problem with PyTorch specifically is that (without Triton compilation) pretty much all projects run in eager mode. That's fine for experimentation and demonstrations in papers, but its crazy that its used so much for production without any compilation. It would be like using debug C binaries for production, and they only work with any kind of sane performance on a single CPU maker.

galangalalgol 3y ago

As long as we are language trolling, why would anyone start a greenfield project like this in C++ these days? The android, windows, firefox, and now chrome projects have all begun to shift towards rust and in the case of Android and Firefox, write a significant amounts of their project in rust. Migrating an existing project like that is difficult. The chrome team in particular lamented the difficulty. But starting a new project? If you have a team familiar with performant c++ the speedbump of starting a greenfield project in rust is negligible, and the ergonomic improvements in the build system and the language itself will make up for that in any project that takes more than a few months. For that speed bump you get memory and thread race safety, far better than any stack of c++ analysis tools could ever provide with a tiny fraction of the unit tests you'd write in c++. And you lose no performance.

mnrlt 3y ago
C++ has a standard, multiple competing implementations and a largely drama-free community. Does CUDA even have Rust bindings, and if so, are they on the same level as the C++ ones? What do you mean by "the windows projects" that shift towards Rust?
- galangalalgol 3y ago
  MS has started implementing pieces of windows in rust. If you have windows 11 you are running rust. The cuda bindings are good for ml, but missing for cufft and similar. There are people working on better cuda support, but there are even more people working on vendor agnostic gpgpu using spirv and webgpu. It isn't there yet. Right now you are mostly left to your own bindings unless you are doing ml or blas. Edit: I can't argue about the drama part. The competing compilers will get there. A couple gcc frontends in work, and crane lift as a competing back end for llvm and full self-hosting. There is also miri I guess to emit c? People use that to get rust on the C64 or other niche processors.
  - pjmlp 3y ago
    Yes they started, yet there is enough C++ to rewrite in the 30 years of Windows NT history. Meanwhile, Visual Studio team released better tooling for Unreal in Visual C++.

A-Train 3y ago

Amen.

danybittel 3y ago

Since when does C++ optimally exploit the underlying hardware? It has no vector instructions, does not run on the GPU and is arguably too hard to make multithreaded. Which leaves you with about 0.5% performance of a current PCs.

jcelerier 3y ago
> does not run on the GPU both Cuda and the Metal shader language are C++, so is OpenCL since 2.0 (https://www.khronos.org/opencl/ https://www.khronos.org/opencl/), so is AMD ROCm's HIP (https://github.com/ROCm-Developer-Tools/HIP https://github.com/ROCm-Developer-Tools/HIP), so is SYCL (https://www.khronos.org/sycl/ https://www.khronos.org/sycl/)? C++ is pretty much the language that runs most on GPUs. > no vector instructions, There's a thousand different possibilities for SIMD in C++, from #pragma omp simd, to libs such as std::experimental::simd (https://en.cppreference.com/w/cpp/experimental/simd/simd https://en.cppreference.com/w/cpp/experimental/simd/simd), Eve (https://github.com/jfalcou/eve https://github.com/jfalcou/eve), Highway (https://github.com/google/highway https://github.com/google/highway), Vc (https://github.com/VcDevel/Vc https://github.com/VcDevel/Vc)...
waynecochran 3y ago
Vector types / instructions would be nice. The C++20 STL algorithms are very friendly to vectorization with the various parallel policies (e.g. std::execution::unsequenced_policy) that open up your code to be vectorized. Wonderful libs like Eigen handle a lot of my numeric needs for linear algebra. I think you are forgetting the CUDA is C/C++.
- Someone 3y ago
  > Vector types / instructions would be nice It’s technically not a C++ feature, but both gcc (https://gcc.gnu.org/onlinedocs/gcc/Vector-Extensions.html https://gcc.gnu.org/onlinedocs/gcc/Vector-Extensions.html) and Clang (https://releases.llvm.org/3.1/tools/clang/docs/LanguageExtensions.html#vectors https://releases.llvm.org/3.1/tools/clang/docs/LanguageExten...) have vector types, and clang even supports the gcc way of writing them, so it gets pretty close.

aimor 3y ago

I really appreciate the people doing this work. It's the only way I've run these models without any headaches. The difference is so stark, even with CUDA and Linux it's bad, with AMD and Windows it's miserable. I'm pretty sure it's not just me..

gumby 3y ago

It's a bummer there's so little work on the training side in C++. Especially since the python training systems are mostly calls into libraries written in C++!

pjmlp 3y ago
Yeah, and since C++17 the language is already quite productive for scripting like workflows, the missing piece of the puzzle is that there are two few C++ REPLs aroung, ROOT/CINT being one of the few well known ones.
- cozzyd 3y ago
  CINT is now obsolete, replaced by cling (which is way better!).