13 ms·

llama.cpp is great. It started off as CPU-only solution and now looks like it wants to support any computation device it can. I find it interesting that it's a

by adeon 3y ago

llama.cpp is great. It started off as CPU-only solution and now looks like it wants to support any computation device it can. I find it interesting that it's an example of an ML software that's totally detached from Python ML ecosystem and also popular. Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? Interested in hearing thoughts.

csjh 3y ago
I believe it's moreso for the (actively pursued) speed optimizations it provides. When inference is already computationally expensive any bit of performance is a big plus
plaguuuuuu 3y ago
I've been using it because there are bindings in other languages I know like .NET
nmfisher 3y ago
It's a great project and an impressive achievement, but I'm also struggling to understand what people use it for that PyTorch wasn't offering. Easy deployment on iOS I guess? I would have thought that's a pretty small use case though. Given the author hand-rolled his own FFT, I'm also guessing it's not as performant?
- civilitty 3y ago
  Easy deployment anywhere, not just iOS. I haven't used Python in years so I have no idea what package manager is the best now, completely forgot how to use virtualenv, and it only took a few weeks to completely fuck up my local Python install ("Your version of CUDA doesn't match the one used to blah blah") Python is a mess. llama.cpp was literally a git clone followed by "cd llama.cpp && make && ./main" - I can recite the commands from memory and I haven't done any C/C++ development in a long time.
  - nmfisher 3y ago
    Thanks for the response. Interesting to see a lot of other people echoing the same comments - that dependency management in Python is an absolute PITA. I know exactly what you mean, but I'm probably so inured to it by now that I've just come to accept it. Obviously not everyone feels this way!
  - cshimmin 3y ago
    For most modern ML projects in python you can just do something like `conda env create -f environment.yaml` then straight to `./main.py`. This handles very complex dependencies in a single command. The example you gave works because llama.cpp specifically strives to have no dependencies. But this is not an intrinsically useful goal; there's a reason software libraries were invented. I always have fun when I find out that the thing I'm trying to compile needs -std=C++26 and glibc 3.0, and I'm running on a HPC cluster where I can't use a system level package manager, and I don't want to be arsed to dockerize every small thing I want to run. For scientific and ML uses, conda has basically solved the whole "python packaging is a mess" that people seem to still complain about, at least on the end-user side. Sure, conda is slow as hell but there's a drop in replacement (mamba) that solves that issue.
    - josephg 3y ago
      Conda? Mamba? Or should I use Venv? What are the commands to “activate” an environment? And why do I have to do that anyway, given thats not needed in any other programming language? Which of those systems support nested dependencies properly? Do any of them support the dependency graph containing multiple, mutually incompatible copies of the same package? Coming from rust (and nodejs before that), the package management situation in python feels like a mess. It’s barely better than C and C++ - both of which are also a disaster. (Make? Autotools? CMake? Use vendored dependencies? System dependencies? Which openblas package should I install from apt? Are any of them recent enough? Kill me.) Node: npm install. npm start. Rust: cargo run. Cargo run —-release. I don’t want to pick from 18 flavours of “virtual environments” that I have to remember how to to “activate”. And I don’t want to deal with transitive dependency conflicts, and I don’t want to be wading through my distro’s packages to figure out how to manually install dependencies. I just want to run the program. Python and C both make that much more difficult than it needs to be.
- deleted 3y ago
  [deleted]
- superkuh 3y ago
  Pytorch (+GPU) dependency and python container type diversity are particularly bad. Programmers may not perceive this since they're already managing their python environment keeping all the OS/libs/containers/applications in the alignment required for things to work but it's quite complex. I couldn't do it. In comparison I could just type git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and make . And it worked. And since then I've managed to get llama.cpp clBLAS partial GPU acceleration working with my AMD RX 580 8GB. Plus with the llama.cpp CPU mmap stuff I can run multiple LLM IRC bot processes using the same model all sharing the RAM representation for free. Are there even ways to run 2 or 3 bit models in pytorch implementations like llama.cpp can do? It's pretty rad I could run a 65B llama in 27 GB of RAM on my 32GB RAM system (and still get better perplexity than 30B 8 bit).
  - manfre 3y ago
    > In comparison I could just type git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and make . And it worked. You're comparing a single, well managed project that had put effort into user onboarding against all projects of a different language and proclaiming that an entire language/ecosystem is crap. The only real take away is that many projects, independent of language, put way too little effort towards onboarding users.
    - superkuh 3y ago
      That's exactly it. Who wants to try pulling in an entire language with wide depenencies and it's ecosystem of envs/containers/etc when a single program will do it? Not people who just want to run inference on pre-made models.
- slashtom 3y ago
  llama cpp is just cpp inference on the llama model. PyTorch is a library to train neural networks. I'm not sure why people are conflating these two totally different projects..
- whywhywhywhy 3y ago
  >Easy deployment on iOS I guess? Deploying a PyTorch running Python app to anything in a way a user can just run it is a struggle. Even iOS aside that’s not a small use case that’s all local and offline ML potential. Really makes me wonder about the whole language, did they ever expect the code to have to run elsewhere than the machines the writer controls.
forgingahead 3y ago
Yes Python is incredibly annoying to use. Their dependency management is a total mess, and it's incredible how brittle packages if there are even minor point changes in versions anywhere in a stack.
- sp332 3y ago
  I have to agree. Installing dependencies for some git repos is a total crapshoot. I ended up wasting so much hard drive space with copies of pytorch. Meanwhile llama.cpp is just "make" and takes less time to build than to download one copy of pytorch.
  - cshimmin 3y ago
    So, the solution is that everyone should write code as self-contained C++ code and not use any software libraries ever. Dependency hell has been solved for all time!
    - sp332 3y ago
      Well, in the cases where the libraries are causing more work than they are solving. Then yes.
    - Tostino 3y ago
      There is a happy medium... somewhere. After following Postgres development for the better part of a decade, I think it's definitely closer to the python side of things... But man they (python) do make it hard to like using that ecosystem. The flip side is like you said... You will just have to reimplement everything yourself and then you can never worry about dependencies again! Just hope you didn't introduce some obscure security issue in your hashmap implementation .
    - josephg 3y ago
      Python was released in 1991. It obviously didn’t get package management right, just like C and C++ didn’t figure it out in the 70s. Take a look at rust’s Cargo for what a modern package manager should look like. Or deno / Go if you swing that way. Which old language gets package management right? None of them. None of them get it right. And sure - conda / venv / CMake / etc help. But last century’s bad design decisions still shine through.
zmmmmm 3y ago
yeah it's sad I guess but half the reason I am recommending this is that it "just works" so much more easily than installing half the Python ecosystem into a conda environment (ironically, just so that Python can then behave as a thin wrapper for calling the underlying native libraries ...)
slabity 3y ago
> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? Interested in hearing thoughts. For me it's less about the language and more about the dependencies. When I want to run a Python ML program, I need to go through the hassle of figuring out what the minimum version is, which package manager/distribution I need to use, and what system libraries those dependencies need to function properly. If I want to build a package for my distribution, these problems are dialed up to 11 and make it difficult to integrate (especially when using Nix). On top of that, those dependencies typically hide the juicy details of the program I actually care about. For something like C or C++? Usually the most complicated part is running `make` or `cmake` with `pkgconfig` somewhere in my path. Maybe install some missing system libraries if necessary. I just don't want to install yet another hundred copies of dependencies in a virtualenv and just hope it's set up correctly.
- taf2 3y ago
  I totally agree with you the irony being python and languages like were built in part to reduce the complexity not only of the language but also to build and run the code… I feel machine learning is a low enough level thing that it should not be tied to a high level language like python… so I can use node, ruby, php or whatever by adding a c binding etc that to me is why this is most interesting
  - josephg 3y ago
    The problem is that python is designed assuming people want to use system-wide packages. In hindsight, that has turned out to be a mistake. Conda / venv try to bridge that gap but they’re kludgy, complex hacks compared to something like cargo or even npm. Worse, because Python is a dynamic language, you also have to deal with all of that complexity at deployment time. (Vs C/C++/Zig/Rust where you can just ship the compiled binary).
    - drdaeman 3y ago
      > The problem is that python is designed assuming people want to use system-wide packages. This wasn't true for decades, `virtualenv` was de-facto standard isolation solution (now baked in as `python -m venv`, still de-facto standard), and `pip` is the package manager (we don't talk about setuptools/distutils, ssh!). If someone still used system-wide packages that was either because a) they were building a container or some single-purpose system; or b) they were sloppy or had no idea what they're doing (most likely, following some crappy tutorial). Or it was distro people creating packages to satisfy dependencies for Python programs - but that's a whole different story (and one's virtualenv shouldn't inherit system packages unless it is really really necessary and iif it makes sense to do so). The problem started when one needed some external non-Python dependencies. Python had invented binary wheels and they're around for a while (completely solving issues with e.g. PostgreSQL drivers, no one needs to worry about libpq), but I suppose depending on specific versions of kernel drivers and CUDA libraries is a more complex and nuanced subject. > Vs C/C++/Zig/Rust where you can just ship the compiled binary Only assuming that you can either statically link, or if all libraries' ABIs are stable (or if you're targeting a very specific ABI, but I've had my share of "version `GLIBC_2.xx' not found"s and not fond of those). In a similar spirit, any Python project can be distributed as one binary (Python interpreter and a ZIP archive, bundled together) plus a set of zero or more .so files.
bartwr 3y ago
Even "compiled" JAX or PyTorch can leave some performance even if you hit the common path (It's also "praying" that the compiler actually works if you do anything non-standard). But memory wise, there is almost no optimization or reuse (and it's sacrificed for performance), which leads to insane memory usage. And it's not that they are bad - but optimal compilation of a graph is combinatorially explosive problem, impossible without heuristics and guesswork (what to reuse and waste memory vs recompute). A good programmer can do a significantly better job.
b33j0r 3y ago
Not quite. Nothing to do directly with python. This was the introduction of 4 and 8 bit quantization to a large number of people. There wasn’t a python library like that anyone was used to using. Would have always been a C extension anyway. Starting with the cpu in this situation made sense, strangely. There are python wrappers now. I tried to make one for ya’ll in rust in April, but haha I had a compiler issue I never solved.
noman-land 3y ago
I don't even write python, really, but I've been interfacing with llama.cpp and whisper.cpp through it recently because it's been most convenient. Before that I was using nodejs libraries that just wrap those two cpp libs. I guess since these models are meant to be run "client side" or "at the edge" or whatever you want to call it, it helps if they can be neutrally used with just about any wrapper. Using them from Javascript instead of Python is sort of huge for moving ML off the server and into the client. I haven't really dipped my toes into the space until llama and whisper cpp came along because they dropped the barrier extremely low. The documentation is simple, and excellent. The tools that it's enabled on top like text-generation-webui are next level easy. git clone. make. download model. run. That's it.
PeterStuer 3y ago
Python isn't annoying to use, at least for me and many I know. But it isn't known for speed. And or easy multithreading.
pjmlp 3y ago
For me Python's main use cases are being BASIC replacement, and a saner alternative to Perl (I like Perl though) for OS scripting. For everything else, I rather use compiled languages with JIT/AOT toolchains, and since most "Python libraries" for machine learning are actually C and C++, any language goes, there is nothing special about Python there.
yieldcrv 3y ago
The Python apologists are more annoying than the language. Its always been obvious that ML’s marriage to python has always been credential ladened proponents in tangentially related fields following group think. As soon as we got a reason to ignore those PhDs, their gatekept moat evaporated overnight and the community of [co-]dependencies became irrelevant overnight.
- dontreact 3y ago
  As far back as 2015, it’s been common to take neural net inference and get a C++ version of it. That’s what this is. It didn’t make python obsolete then and it won’t now. Training (orchestration, architecture definition etc.) and data munging (scraping cleaning analyzing etc.) are much easier with python than C++, and so there is no chance that C++ takes over as the lingua Franca of machine learning until those activities are rare
  - emmender 3y ago
    python is syntactic sugar - the heavy lifting is done by c/c++ bindings. many ML experts are not software engineers. They just want syntax to get their job done. fair enough.
realusername 3y ago
I personally really don't like much Python, I find it as tedious to write as Go but without the added performance, typesafety and autocomplete benefits that comes with it in exchange. If I have use a dynamic language, at least make it battery included like Ruby. Sure it's also not performant but I get something back in exchange. Python sits in a very uncomfortable spot which I don't find a use for. Too verbose for a dynamic language and not performant enough compared to a more static language. The testing culture is also pretty poor in my opinion, packages rarely have proper tests, (and especially in the ML field)
- emmender 3y ago
  In addition to the above: 1) function decorators etc have made the code unreadable 2) while code is succinct, a lot of abstraction is hidden in some C/C++ language binding somewhere, so, when there is a problem, it is hard to debug 3) Pytorch has become a monolithic monster with practically no-one understanding its functionality e-2-e
nologic01 3y ago
Wow, so interesting to see the "depth" of anti-python feeling in some quarters. I guess that is the backlash from all the hordes of Python-bros. Having used both C++ and Python for some time, the idea that managing C++ dependencies is easier than venv and pip install is one of the moments you wonder how credible is HN opinion on anything. > a compelling non-Python solution appears Confusing a large ML framework like pytorch that allows you to experiment and develop any type of model with a particular optimized implementation in a low level language suggests people are not even aware of basic workflows in this space. > also popular Ofcourse its popular. As in: People are delirious with LLM FOMO but can't fork gazillions to cloud gatekeepers or NVIDIA so anybody who can alleviate that pain is like a deus-ex-machina. Ofcourse llama.cpp and its creator are great. But the exercise primarily points out that there isn't a unified platfrom to both develop and deploy ML type models in a flexible and hardware agnostic way. p.s. For julia lovers that usually jump at the "two-language problem" of Python: here is your chance to shine. There is a Llama.jl that wraps Llama.cpp. You want to develop a native one.
- mook 3y ago
  Managing C++ dependencies _is_ much easier! It's either "run this setup exe" or "extract this zip file/tarball/dmg and run". This is because most people don't care about developing the project, just using it. So they don't care what the dependencies are, just that things work. C++ might be more difficult to handle dependencies to build things, but few people will look into hacking on the code before checking to see if it's even relevant.
  - nologic01 3y ago
    Maybe this distinction explains indeed the dissonance! But it might be rather shortsighted given the state of those models and the need to tune them.
  - ric2b 3y ago
    If we're only talking about end-user "binaries" you can also package Python protects into exe files or similar format that bundle all the dependencies and are ready to run.
MrYellowP 3y ago
> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? Interested in hearing thoughts. Exploring the landscape ends up with you having 29384232938792834234 different python environments, because that one thing requires specific versions of one set of libraries, while that other thing requires different versions of the same library and there is no middle ground. It's horribly annoying and I absolutely love python!
noduerme 3y ago
I've been wondering for awhile now - what was ever the benefit to building these things in Python, other than pytorch and numpy being there to experiment on in hobbyist ways? There's no way that a serious AI is really going to be built in a scripting language, is there? Once you know what you actually want it to do, you're definitely going to rebuild it as close to the metal as you can, right? Not to mention, to really protect source code and all the sugar around the training systems, it's going to be a good investment to get out of hobby land and manage your own memory and just code them in C/C++. It strikes me that the hobby AI ethos aligns very well with scripting languages in that they both assume the availability of endless resources to push things a little dirtier and messier and see if anything interesting emerges. Which is great for hobby AI. It's probably not the future, though, unless resource availability outpaces the imagination of people to write more and more bloated scripts to accomplish what's already been proven.
- nologic01 3y ago
  incredibly condescending which always pairs well with ignorance. "Hobby AI" are the people (mathematicians, domain experts etc) that made this all possible so that you can now "just code it in C/C++". Have you ever tried to iterate developing any serious class of algorithms in C++? > Once you know what you actually want it to do When is that exactly? Even the last few months of LLM land development show very clearly how everything is rapidly evolving (and will very likely continue for quite some time). Numerical linear algebra stabilized decades ago so you do have low-level libraries in C++ (or even fortran) but there is quite some distance between an LLM and linear algebra.
  - fisf 3y ago
    > Have you ever tried to iterate developing any serious class of algorithms in C++? Yes, and it's not so bad. A lot of ML deployments have been based on C/C++ for inference anyway (with Python driving the training). So that's really nothing new. I.e. most Python research code is not deployable in terms of quality / performance.
    - 3y ago
thom 3y ago
I think Python and R are generally superior (in terms of developer experience) when you have to do end-to-end ML work, including acquiring and munging data, plotting results etc. But even then, the core algorithms are generally implemented in libraries built out of native code (C/C++/Fortran), just wrapped in friendly bindings. For LLMs, unless you're doing extensive work refactoring the inputs, there are fewer productivity gains to be had around the edges - the main gains are just speeding up training, evaluation and inference, i.e. pure performance.
kristofferR 3y ago
Yeah, most Python software seem to have pretty poor compatibility, I usually need to downgrade my Python version to get stuff to run.
v3ss0n 3y ago
Quality of HN comments are getting bad for a few months. This is nothing to do with python ML ecosystem and what you have to realize llmcpp doing is it is inferencing already built models- which is running the models. Building (training) machine learning, deep learning models are much more complex , order of magnitude complex than just running the models and doing that in C or C++ would take you years which would take just a few month with python. And complexity of `pip install` is nothing compared to that. That's why no real ETL+Deep learning, training work is done in c or c++.
- cztomsik 3y ago
  You're not entirely wrong but pretty much everything you use from python is written in C++ anyway, so what's your point?
  - ageofwant 3y ago
    The point is, as you pointed out, that you code against the appropriate level of abstraction. You write a ML workflow appropriate language like Python in something like C++/rust, and ML flows in Python. That should really not be that hard to understand.
  - v3ss0n 3y ago
    It is same argument as "Every HTML, CSS , JAVASCRIPT" development you do is written in C/C++ anyways .
    - cztomsik 3y ago
      I am not an expert, but from what I've seen, PyTorch is mostly a thin wrapper over the C++ libtorch. The same is true for DOM as well, but nobody uses DOM directly, whereas everybody uses PyTorch. The big shift is Jupyter, but that's mainly for exploratory programming. If you already know what you're doing, there's no reason why C++ should be worse than Python for training. It's likely that most ML engineers do not have experience with C++. I don't have that experience either but from what I've seen, C++ is very powerful, so once you subtract the jupyter, there's not really too much left. BTW: You cannot use DOM/CSSOM from C++, the only API is JS, so your argument is theoretical.
jtode 3y ago
I'm currently a Python dev for a living, but I spend a portion of my personal time trying to get better at C/C++, and in my case, it's strictly about the potential for writing faster code. I'm interesting in getting into DSP stuff specifically, so it's 100% necessary if I want to do that, and I would also like to get my head wrapped around OpenCV for similarly creative reasons. Python opened up the world of code to a lot more people, but there was a cost to that, and the real action as far as actual computer systems go is always gonna be at a much lower level, in much the same way that a lot of people can top up their fluids but most of us pay someone to change the oil. I could actually totally see oil changes becoming completely robotic in the future, but we would first have to establish open standards for oil pans that all automakers adhered to. The whole computers designing computers thing, outside of someone cracking cold fusion I don't think we'll ever have the juice for it. In my lifetime, a C-level programmer will never be out of work, but I suspect that the demand for Python programmers is going to slack off, while the supply continues to grow.
tension000 3y ago
Yet another TEDIOUS BATTLE: Python vs. C++/C stack. This project gained popularity due to the HIGH DEMAND for running large models with 1B+ parameters, like `llama`. Python dominates the interface and training ecosystem, but prior to llama.cpp, non-ML professionals showed little interest in a fast C++ interface library. While existing solutions like tensorflow-serving [1] in C++ were sufficiently fast with GPU support, llama.cpp took the initiative to optimize for CPU and trim unnecessary code, essentially code-golfing and sacrificing some algorithm correctness for improved performance, which isn't favored by "ML research". NOTE: In my opinion, a true pioneer was DarkNet, which implemented the YOLO model series and significantly outperformed others [2]. Same trick basically like llama.cpp [1] https://github.com/tensorflow/serving https://github.com/tensorflow/serving [2] https://github.com/pjreddie/darknet https://github.com/pjreddie/darknet
dragonwriter 3y ago
> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? The desktop/laptop LLMs use case is one where resource efficiency often makes the difference between “I can’t use this at all” rather than the more frequent “I can use it but it maybe runs a little slower”, and llama.cpp offers that. It also has offered new quantization options that the Python-based tooling hasn’t, which compounds the basic resource efficiency point. (It’s also not an “everyone” thing: plenty of people are using the Python-based toolchains for LLMs, its possible for their to be multiple popular options in a space . Not everything is all or nothing.)
deleted 3y ago
[deleted]
IshKebab 3y ago
> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Yes.

by adeon 3y ago

csjh 3y ago

I believe it's moreso for the (actively pursued) speed optimizations it provides. When inference is already computationally expensive any bit of performance is a big plus

plaguuuuuu 3y ago

I've been using it because there are bindings in other languages I know like .NET

nmfisher 3y ago

It's a great project and an impressive achievement, but I'm also struggling to understand what people use it for that PyTorch wasn't offering. Easy deployment on iOS I guess? I would have thought that's a pretty small use case though. Given the author hand-rolled his own FFT, I'm also guessing it's not as performant?

civilitty 3y ago
Easy deployment anywhere, not just iOS. I haven't used Python in years so I have no idea what package manager is the best now, completely forgot how to use virtualenv, and it only took a few weeks to completely fuck up my local Python install ("Your version of CUDA doesn't match the one used to blah blah") Python is a mess. llama.cpp was literally a git clone followed by "cd llama.cpp && make && ./main" - I can recite the commands from memory and I haven't done any C/C++ development in a long time.
- nmfisher 3y ago
  Thanks for the response. Interesting to see a lot of other people echoing the same comments - that dependency management in Python is an absolute PITA. I know exactly what you mean, but I'm probably so inured to it by now that I've just come to accept it. Obviously not everyone feels this way!
- cshimmin 3y ago
  For most modern ML projects in python you can just do something like `conda env create -f environment.yaml` then straight to `./main.py`. This handles very complex dependencies in a single command. The example you gave works because llama.cpp specifically strives to have no dependencies. But this is not an intrinsically useful goal; there's a reason software libraries were invented. I always have fun when I find out that the thing I'm trying to compile needs -std=C++26 and glibc 3.0, and I'm running on a HPC cluster where I can't use a system level package manager, and I don't want to be arsed to dockerize every small thing I want to run. For scientific and ML uses, conda has basically solved the whole "python packaging is a mess" that people seem to still complain about, at least on the end-user side. Sure, conda is slow as hell but there's a drop in replacement (mamba) that solves that issue.
  - josephg 3y ago
    Conda? Mamba? Or should I use Venv? What are the commands to “activate” an environment? And why do I have to do that anyway, given thats not needed in any other programming language? Which of those systems support nested dependencies properly? Do any of them support the dependency graph containing multiple, mutually incompatible copies of the same package? Coming from rust (and nodejs before that), the package management situation in python feels like a mess. It’s barely better than C and C++ - both of which are also a disaster. (Make? Autotools? CMake? Use vendored dependencies? System dependencies? Which openblas package should I install from apt? Are any of them recent enough? Kill me.) Node: npm install. npm start. Rust: cargo run. Cargo run —-release. I don’t want to pick from 18 flavours of “virtual environments” that I have to remember how to to “activate”. And I don’t want to deal with transitive dependency conflicts, and I don’t want to be wading through my distro’s packages to figure out how to manually install dependencies. I just want to run the program. Python and C both make that much more difficult than it needs to be.
deleted 3y ago
[deleted]
superkuh 3y ago
Pytorch (+GPU) dependency and python container type diversity are particularly bad. Programmers may not perceive this since they're already managing their python environment keeping all the OS/libs/containers/applications in the alignment required for things to work but it's quite complex. I couldn't do it. In comparison I could just type git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and make . And it worked. And since then I've managed to get llama.cpp clBLAS partial GPU acceleration working with my AMD RX 580 8GB. Plus with the llama.cpp CPU mmap stuff I can run multiple LLM IRC bot processes using the same model all sharing the RAM representation for free. Are there even ways to run 2 or 3 bit models in pytorch implementations like llama.cpp can do? It's pretty rad I could run a 65B llama in 27 GB of RAM on my 32GB RAM system (and still get better perplexity than 30B 8 bit).
- manfre 3y ago
  > In comparison I could just type git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and make . And it worked. You're comparing a single, well managed project that had put effort into user onboarding against all projects of a different language and proclaiming that an entire language/ecosystem is crap. The only real take away is that many projects, independent of language, put way too little effort towards onboarding users.
  - superkuh 3y ago
    That's exactly it. Who wants to try pulling in an entire language with wide depenencies and it's ecosystem of envs/containers/etc when a single program will do it? Not people who just want to run inference on pre-made models.
slashtom 3y ago
llama cpp is just cpp inference on the llama model. PyTorch is a library to train neural networks. I'm not sure why people are conflating these two totally different projects..
whywhywhywhy 3y ago
>Easy deployment on iOS I guess? Deploying a PyTorch running Python app to anything in a way a user can just run it is a struggle. Even iOS aside that’s not a small use case that’s all local and offline ML potential. Really makes me wonder about the whole language, did they ever expect the code to have to run elsewhere than the machines the writer controls.

forgingahead 3y ago

Yes Python is incredibly annoying to use. Their dependency management is a total mess, and it's incredible how brittle packages if there are even minor point changes in versions anywhere in a stack.

sp332 3y ago
I have to agree. Installing dependencies for some git repos is a total crapshoot. I ended up wasting so much hard drive space with copies of pytorch. Meanwhile llama.cpp is just "make" and takes less time to build than to download one copy of pytorch.
- cshimmin 3y ago
  So, the solution is that everyone should write code as self-contained C++ code and not use any software libraries ever. Dependency hell has been solved for all time!
  - sp332 3y ago
    Well, in the cases where the libraries are causing more work than they are solving. Then yes.
  - Tostino 3y ago
    There is a happy medium... somewhere. After following Postgres development for the better part of a decade, I think it's definitely closer to the python side of things... But man they (python) do make it hard to like using that ecosystem. The flip side is like you said... You will just have to reimplement everything yourself and then you can never worry about dependencies again! Just hope you didn't introduce some obscure security issue in your hashmap implementation .
  - josephg 3y ago
    Python was released in 1991. It obviously didn’t get package management right, just like C and C++ didn’t figure it out in the 70s. Take a look at rust’s Cargo for what a modern package manager should look like. Or deno / Go if you swing that way. Which old language gets package management right? None of them. None of them get it right. And sure - conda / venv / CMake / etc help. But last century’s bad design decisions still shine through.

zmmmmm 3y ago

yeah it's sad I guess but half the reason I am recommending this is that it "just works" so much more easily than installing half the Python ecosystem into a conda environment (ironically, just so that Python can then behave as a thin wrapper for calling the underlying native libraries ...)

slabity 3y ago

> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? Interested in hearing thoughts. For me it's less about the language and more about the dependencies. When I want to run a Python ML program, I need to go through the hassle of figuring out what the minimum version is, which package manager/distribution I need to use, and what system libraries those dependencies need to function properly. If I want to build a package for my distribution, these problems are dialed up to 11 and make it difficult to integrate (especially when using Nix). On top of that, those dependencies typically hide the juicy details of the program I actually care about. For something like C or C++? Usually the most complicated part is running `make` or `cmake` with `pkgconfig` somewhere in my path. Maybe install some missing system libraries if necessary. I just don't want to install yet another hundred copies of dependencies in a virtualenv and just hope it's set up correctly.

taf2 3y ago
I totally agree with you the irony being python and languages like were built in part to reduce the complexity not only of the language but also to build and run the code… I feel machine learning is a low enough level thing that it should not be tied to a high level language like python… so I can use node, ruby, php or whatever by adding a c binding etc that to me is why this is most interesting
- josephg 3y ago
  The problem is that python is designed assuming people want to use system-wide packages. In hindsight, that has turned out to be a mistake. Conda / venv try to bridge that gap but they’re kludgy, complex hacks compared to something like cargo or even npm. Worse, because Python is a dynamic language, you also have to deal with all of that complexity at deployment time. (Vs C/C++/Zig/Rust where you can just ship the compiled binary).
  - drdaeman 3y ago
    > The problem is that python is designed assuming people want to use system-wide packages. This wasn't true for decades, `virtualenv` was de-facto standard isolation solution (now baked in as `python -m venv`, still de-facto standard), and `pip` is the package manager (we don't talk about setuptools/distutils, ssh!). If someone still used system-wide packages that was either because a) they were building a container or some single-purpose system; or b) they were sloppy or had no idea what they're doing (most likely, following some crappy tutorial). Or it was distro people creating packages to satisfy dependencies for Python programs - but that's a whole different story (and one's virtualenv shouldn't inherit system packages unless it is really really necessary and iif it makes sense to do so). The problem started when one needed some external non-Python dependencies. Python had invented binary wheels and they're around for a while (completely solving issues with e.g. PostgreSQL drivers, no one needs to worry about libpq), but I suppose depending on specific versions of kernel drivers and CUDA libraries is a more complex and nuanced subject. > Vs C/C++/Zig/Rust where you can just ship the compiled binary Only assuming that you can either statically link, or if all libraries' ABIs are stable (or if you're targeting a very specific ABI, but I've had my share of "version `GLIBC_2.xx' not found"s and not fond of those). In a similar spirit, any Python project can be distributed as one binary (Python interpreter and a ZIP archive, bundled together) plus a set of zero or more .so files.

bartwr 3y ago

Even "compiled" JAX or PyTorch can leave some performance even if you hit the common path (It's also "praying" that the compiler actually works if you do anything non-standard). But memory wise, there is almost no optimization or reuse (and it's sacrificed for performance), which leads to insane memory usage. And it's not that they are bad - but optimal compilation of a graph is combinatorially explosive problem, impossible without heuristics and guesswork (what to reuse and waste memory vs recompute). A good programmer can do a significantly better job.

b33j0r 3y ago

Not quite. Nothing to do directly with python. This was the introduction of 4 and 8 bit quantization to a large number of people. There wasn’t a python library like that anyone was used to using. Would have always been a C extension anyway. Starting with the cpu in this situation made sense, strangely. There are python wrappers now. I tried to make one for ya’ll in rust in April, but haha I had a compiler issue I never solved.

noman-land 3y ago

I don't even write python, really, but I've been interfacing with llama.cpp and whisper.cpp through it recently because it's been most convenient. Before that I was using nodejs libraries that just wrap those two cpp libs. I guess since these models are meant to be run "client side" or "at the edge" or whatever you want to call it, it helps if they can be neutrally used with just about any wrapper. Using them from Javascript instead of Python is sort of huge for moving ML off the server and into the client. I haven't really dipped my toes into the space until llama and whisper cpp came along because they dropped the barrier extremely low. The documentation is simple, and excellent. The tools that it's enabled on top like text-generation-webui are next level easy. git clone. make. download model. run. That's it.

PeterStuer 3y ago

Python isn't annoying to use, at least for me and many I know. But it isn't known for speed. And or easy multithreading.

pjmlp 3y ago

For me Python's main use cases are being BASIC replacement, and a saner alternative to Perl (I like Perl though) for OS scripting. For everything else, I rather use compiled languages with JIT/AOT toolchains, and since most "Python libraries" for machine learning are actually C and C++, any language goes, there is nothing special about Python there.

yieldcrv 3y ago

The Python apologists are more annoying than the language. Its always been obvious that ML’s marriage to python has always been credential ladened proponents in tangentially related fields following group think. As soon as we got a reason to ignore those PhDs, their gatekept moat evaporated overnight and the community of [co-]dependencies became irrelevant overnight.

dontreact 3y ago
As far back as 2015, it’s been common to take neural net inference and get a C++ version of it. That’s what this is. It didn’t make python obsolete then and it won’t now. Training (orchestration, architecture definition etc.) and data munging (scraping cleaning analyzing etc.) are much easier with python than C++, and so there is no chance that C++ takes over as the lingua Franca of machine learning until those activities are rare
- emmender 3y ago
  python is syntactic sugar - the heavy lifting is done by c/c++ bindings. many ML experts are not software engineers. They just want syntax to get their job done. fair enough.

realusername 3y ago

I personally really don't like much Python, I find it as tedious to write as Go but without the added performance, typesafety and autocomplete benefits that comes with it in exchange. If I have use a dynamic language, at least make it battery included like Ruby. Sure it's also not performant but I get something back in exchange. Python sits in a very uncomfortable spot which I don't find a use for. Too verbose for a dynamic language and not performant enough compared to a more static language. The testing culture is also pretty poor in my opinion, packages rarely have proper tests, (and especially in the ML field)

emmender 3y ago
In addition to the above: 1) function decorators etc have made the code unreadable 2) while code is succinct, a lot of abstraction is hidden in some C/C++ language binding somewhere, so, when there is a problem, it is hard to debug 3) Pytorch has become a monolithic monster with practically no-one understanding its functionality e-2-e

nologic01 3y ago

Wow, so interesting to see the "depth" of anti-python feeling in some quarters. I guess that is the backlash from all the hordes of Python-bros. Having used both C++ and Python for some time, the idea that managing C++ dependencies is easier than venv and pip install is one of the moments you wonder how credible is HN opinion on anything. > a compelling non-Python solution appears Confusing a large ML framework like pytorch that allows you to experiment and develop any type of model with a particular optimized implementation in a low level language suggests people are not even aware of basic workflows in this space. > also popular Ofcourse its popular. As in: People are delirious with LLM FOMO but can't fork gazillions to cloud gatekeepers or NVIDIA so anybody who can alleviate that pain is like a deus-ex-machina. Ofcourse llama.cpp and its creator are great. But the exercise primarily points out that there isn't a unified platfrom to both develop and deploy ML type models in a flexible and hardware agnostic way. p.s. For julia lovers that usually jump at the "two-language problem" of Python: here is your chance to shine. There is a Llama.jl that wraps Llama.cpp. You want to develop a native one.

mook 3y ago
Managing C++ dependencies _is_ much easier! It's either "run this setup exe" or "extract this zip file/tarball/dmg and run". This is because most people don't care about developing the project, just using it. So they don't care what the dependencies are, just that things work. C++ might be more difficult to handle dependencies to build things, but few people will look into hacking on the code before checking to see if it's even relevant.
- nologic01 3y ago
  Maybe this distinction explains indeed the dissonance! But it might be rather shortsighted given the state of those models and the need to tune them.
- ric2b 3y ago
  If we're only talking about end-user "binaries" you can also package Python protects into exe files or similar format that bundle all the dependencies and are ready to run.

MrYellowP 3y ago

> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? Interested in hearing thoughts. Exploring the landscape ends up with you having 29384232938792834234 different python environments, because that one thing requires specific versions of one set of libraries, while that other thing requires different versions of the same library and there is no middle ground. It's horribly annoying and I absolutely love python!

noduerme 3y ago

I've been wondering for awhile now - what was ever the benefit to building these things in Python, other than pytorch and numpy being there to experiment on in hobbyist ways? There's no way that a serious AI is really going to be built in a scripting language, is there? Once you know what you actually want it to do, you're definitely going to rebuild it as close to the metal as you can, right? Not to mention, to really protect source code and all the sugar around the training systems, it's going to be a good investment to get out of hobby land and manage your own memory and just code them in C/C++. It strikes me that the hobby AI ethos aligns very well with scripting languages in that they both assume the availability of endless resources to push things a little dirtier and messier and see if anything interesting emerges. Which is great for hobby AI. It's probably not the future, though, unless resource availability outpaces the imagination of people to write more and more bloated scripts to accomplish what's already been proven.

nologic01 3y ago
incredibly condescending which always pairs well with ignorance. "Hobby AI" are the people (mathematicians, domain experts etc) that made this all possible so that you can now "just code it in C/C++". Have you ever tried to iterate developing any serious class of algorithms in C++? > Once you know what you actually want it to do When is that exactly? Even the last few months of LLM land development show very clearly how everything is rapidly evolving (and will very likely continue for quite some time). Numerical linear algebra stabilized decades ago so you do have low-level libraries in C++ (or even fortran) but there is quite some distance between an LLM and linear algebra.
- fisf 3y ago
  > Have you ever tried to iterate developing any serious class of algorithms in C++? Yes, and it's not so bad. A lot of ML deployments have been based on C/C++ for inference anyway (with Python driving the training). So that's really nothing new. I.e. most Python research code is not deployable in terms of quality / performance.
  - 3y ago

thom 3y ago

I think Python and R are generally superior (in terms of developer experience) when you have to do end-to-end ML work, including acquiring and munging data, plotting results etc. But even then, the core algorithms are generally implemented in libraries built out of native code (C/C++/Fortran), just wrapped in friendly bindings. For LLMs, unless you're doing extensive work refactoring the inputs, there are fewer productivity gains to be had around the edges - the main gains are just speeding up training, evaluation and inference, i.e. pure performance.

kristofferR 3y ago

Yeah, most Python software seem to have pretty poor compatibility, I usually need to downgrade my Python version to get stuff to run.

v3ss0n 3y ago

Quality of HN comments are getting bad for a few months. This is nothing to do with python ML ecosystem and what you have to realize llmcpp doing is it is inferencing already built models- which is running the models. Building (training) machine learning, deep learning models are much more complex , order of magnitude complex than just running the models and doing that in C or C++ would take you years which would take just a few month with python. And complexity of `pip install` is nothing compared to that. That's why no real ETL+Deep learning, training work is done in c or c++.

cztomsik 3y ago
You're not entirely wrong but pretty much everything you use from python is written in C++ anyway, so what's your point?
- ageofwant 3y ago
  The point is, as you pointed out, that you code against the appropriate level of abstraction. You write a ML workflow appropriate language like Python in something like C++/rust, and ML flows in Python. That should really not be that hard to understand.
- v3ss0n 3y ago
  It is same argument as "Every HTML, CSS , JAVASCRIPT" development you do is written in C/C++ anyways .
  - cztomsik 3y ago
    I am not an expert, but from what I've seen, PyTorch is mostly a thin wrapper over the C++ libtorch. The same is true for DOM as well, but nobody uses DOM directly, whereas everybody uses PyTorch. The big shift is Jupyter, but that's mainly for exploratory programming. If you already know what you're doing, there's no reason why C++ should be worse than Python for training. It's likely that most ML engineers do not have experience with C++. I don't have that experience either but from what I've seen, C++ is very powerful, so once you subtract the jupyter, there's not really too much left. BTW: You cannot use DOM/CSSOM from C++, the only API is JS, so your argument is theoretical.

jtode 3y ago

I'm currently a Python dev for a living, but I spend a portion of my personal time trying to get better at C/C++, and in my case, it's strictly about the potential for writing faster code. I'm interesting in getting into DSP stuff specifically, so it's 100% necessary if I want to do that, and I would also like to get my head wrapped around OpenCV for similarly creative reasons. Python opened up the world of code to a lot more people, but there was a cost to that, and the real action as far as actual computer systems go is always gonna be at a much lower level, in much the same way that a lot of people can top up their fluids but most of us pay someone to change the oil. I could actually totally see oil changes becoming completely robotic in the future, but we would first have to establish open standards for oil pans that all automakers adhered to. The whole computers designing computers thing, outside of someone cracking cold fusion I don't think we'll ever have the juice for it. In my lifetime, a C-level programmer will never be out of work, but I suspect that the demand for Python programmers is going to slack off, while the supply continues to grow.

tension000 3y ago

Yet another TEDIOUS BATTLE: Python vs. C++/C stack. This project gained popularity due to the HIGH DEMAND for running large models with 1B+ parameters, like `llama`. Python dominates the interface and training ecosystem, but prior to llama.cpp, non-ML professionals showed little interest in a fast C++ interface library. While existing solutions like tensorflow-serving [1] in C++ were sufficiently fast with GPU support, llama.cpp took the initiative to optimize for CPU and trim unnecessary code, essentially code-golfing and sacrificing some algorithm correctness for improved performance, which isn't favored by "ML research". NOTE: In my opinion, a true pioneer was DarkNet, which implemented the YOLO model series and significantly outperformed others [2]. Same trick basically like llama.cpp [1] https://github.com/tensorflow/serving https://github.com/tensorflow/serving [2] https://github.com/pjreddie/darknet https://github.com/pjreddie/darknet

dragonwriter 3y ago

> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Less hassle? Or did it take off for a different reason? The desktop/laptop LLMs use case is one where resource efficiency often makes the difference between “I can’t use this at all” rather than the more frequent “I can use it but it maybe runs a little slower”, and llama.cpp offers that. It also has offered new quantization options that the Python-based tooling hasn’t, which compounds the basic resource efficiency point. (It’s also not an “everyone” thing: plenty of people are using the Python-based toolchains for LLMs, its possible for their to be multiple popular options in a space . Not everything is all or nothing.)

deleted 3y ago

[deleted]

IshKebab 3y ago

> Is Python so annoying to use that when a compelling non-Python solution appears, everyone will love it? Yes.