5 ms·

It's a great project and an impressive achievement, but I'm also struggling to understand what people use it for that PyTorch wasn't offering. Easy deployment o

It's a great project and an impressive achievement, but I'm also struggling to understand what people use it for that PyTorch wasn't offering. Easy deployment on iOS I guess? I would have thought that's a pretty small use case though. Given the author hand-rolled his own FFT, I'm also guessing it's not as performant?

civilitty 3y ago
Easy deployment anywhere, not just iOS. I haven't used Python in years so I have no idea what package manager is the best now, completely forgot how to use virtualenv, and it only took a few weeks to completely fuck up my local Python install ("Your version of CUDA doesn't match the one used to blah blah") Python is a mess. llama.cpp was literally a git clone followed by "cd llama.cpp && make && ./main" - I can recite the commands from memory and I haven't done any C/C++ development in a long time.
- nmfisher 3y ago
  Thanks for the response. Interesting to see a lot of other people echoing the same comments - that dependency management in Python is an absolute PITA. I know exactly what you mean, but I'm probably so inured to it by now that I've just come to accept it. Obviously not everyone feels this way!
- cshimmin 3y ago
  For most modern ML projects in python you can just do something like `conda env create -f environment.yaml` then straight to `./main.py`. This handles very complex dependencies in a single command. The example you gave works because llama.cpp specifically strives to have no dependencies. But this is not an intrinsically useful goal; there's a reason software libraries were invented. I always have fun when I find out that the thing I'm trying to compile needs -std=C++26 and glibc 3.0, and I'm running on a HPC cluster where I can't use a system level package manager, and I don't want to be arsed to dockerize every small thing I want to run. For scientific and ML uses, conda has basically solved the whole "python packaging is a mess" that people seem to still complain about, at least on the end-user side. Sure, conda is slow as hell but there's a drop in replacement (mamba) that solves that issue.
  - josephg 3y ago
    Conda? Mamba? Or should I use Venv? What are the commands to “activate” an environment? And why do I have to do that anyway, given thats not needed in any other programming language? Which of those systems support nested dependencies properly? Do any of them support the dependency graph containing multiple, mutually incompatible copies of the same package? Coming from rust (and nodejs before that), the package management situation in python feels like a mess. It’s barely better than C and C++ - both of which are also a disaster. (Make? Autotools? CMake? Use vendored dependencies? System dependencies? Which openblas package should I install from apt? Are any of them recent enough? Kill me.) Node: npm install. npm start. Rust: cargo run. Cargo run —-release. I don’t want to pick from 18 flavours of “virtual environments” that I have to remember how to to “activate”. And I don’t want to deal with transitive dependency conflicts, and I don’t want to be wading through my distro’s packages to figure out how to manually install dependencies. I just want to run the program. Python and C both make that much more difficult than it needs to be.
    - baq 3y ago
      > npm install. npm start. Yeah, no… my current $DAYJOB has a confusing mix of nx, pnpm, npm commands in multiple projects. Python is bad but node is absolutely not a good example.
      - josephg 3y ago
        Eh. Node / npm on its own is generally fine, especially if you use a private package repository for internally shared libraries. The problems show up when compiling javascript for the web as well as nodejs. If you stick to server side javascript using node and npm, it all works pretty well. It’s much nicer than venv / conda. And it handles transitive dependency conflicts and all sorts of wacky problems. It’s just that almost nobody does that. What we want instead is to combine typescript, js bundlers, wasm, es modules and node packages all together to make web apps. And that’s more than enough to bring seasoned engineers to tears. Let alone adding in svelte / solidjs compilation on top of all that. I have sweats just thinking about it.
- vachina 3y ago
  You can “cd llama.cpp && make && ./main" because the developers made that happen. A self contained zero dependency Python project is also literally “python app.py”
  - peatmoss 3y ago
    This is true, though in practice I so often find large, non-trivial examples of the former, but very seldom find examples of the latter. The barriers to entry are lower with python, and so I think there tends to be a lower standard for deployment.
- ojosilva 3y ago
  Python sits on the C-glue segment of programming languages (where Perl, PHP, Ruby and Node are also notable members). Being a glue language means having APIs to a lot of external toolchains written in not only C/C++ but many other compiled languages, APIs and system resources. Conda, virtualenv, etc. are godsend modules for making it all work, or even better, to freeze things once they all work, without resourcing to Docker, VMs or shell scripts. It's meant for application and DevOps people who need to slap together, ie, ML, Numpy, Elasticsearch, AWS APIs and REST endpoints and Get $hit Done. It's annoying to see them "glueys" compared to the binary compiled segment where the heavy lifting is done. Python and others exist to latch on and assimilate. Resistance is futile: https://pypi.org/project/pyllamacpp/ https://pypi.org/project/pyllamacpp/ https://www.npmjs.com/package/llama-node https://www.npmjs.com/package/llama-node https://packagist.org/packages/kambo/llama-cpp-php https://packagist.org/packages/kambo/llama-cpp-php https://github.com/yoshoku/llama_cpp.rb https://github.com/yoshoku/llama_cpp.rb
deleted 3y ago
[deleted]
superkuh 3y ago
Pytorch (+GPU) dependency and python container type diversity are particularly bad. Programmers may not perceive this since they're already managing their python environment keeping all the OS/libs/containers/applications in the alignment required for things to work but it's quite complex. I couldn't do it. In comparison I could just type git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and make . And it worked. And since then I've managed to get llama.cpp clBLAS partial GPU acceleration working with my AMD RX 580 8GB. Plus with the llama.cpp CPU mmap stuff I can run multiple LLM IRC bot processes using the same model all sharing the RAM representation for free. Are there even ways to run 2 or 3 bit models in pytorch implementations like llama.cpp can do? It's pretty rad I could run a 65B llama in 27 GB of RAM on my 32GB RAM system (and still get better perplexity than 30B 8 bit).
- manfre 3y ago
  > In comparison I could just type git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and make . And it worked. You're comparing a single, well managed project that had put effort into user onboarding against all projects of a different language and proclaiming that an entire language/ecosystem is crap. The only real take away is that many projects, independent of language, put way too little effort towards onboarding users.
  - superkuh 3y ago
    That's exactly it. Who wants to try pulling in an entire language with wide depenencies and it's ecosystem of envs/containers/etc when a single program will do it? Not people who just want to run inference on pre-made models.
slashtom 3y ago
llama cpp is just cpp inference on the llama model. PyTorch is a library to train neural networks. I'm not sure why people are conflating these two totally different projects..
whywhywhywhy 3y ago
>Easy deployment on iOS I guess? Deploying a PyTorch running Python app to anything in a way a user can just run it is a struggle. Even iOS aside that’s not a small use case that’s all local and offline ML potential. Really makes me wonder about the whole language, did they ever expect the code to have to run elsewhere than the machines the writer controls.

civilitty 3y ago

Easy deployment anywhere, not just iOS. I haven't used Python in years so I have no idea what package manager is the best now, completely forgot how to use virtualenv, and it only took a few weeks to completely fuck up my local Python install ("Your version of CUDA doesn't match the one used to blah blah") Python is a mess. llama.cpp was literally a git clone followed by "cd llama.cpp && make && ./main" - I can recite the commands from memory and I haven't done any C/C++ development in a long time.

nmfisher 3y ago
Thanks for the response. Interesting to see a lot of other people echoing the same comments - that dependency management in Python is an absolute PITA. I know exactly what you mean, but I'm probably so inured to it by now that I've just come to accept it. Obviously not everyone feels this way!
cshimmin 3y ago
For most modern ML projects in python you can just do something like `conda env create -f environment.yaml` then straight to `./main.py`. This handles very complex dependencies in a single command. The example you gave works because llama.cpp specifically strives to have no dependencies. But this is not an intrinsically useful goal; there's a reason software libraries were invented. I always have fun when I find out that the thing I'm trying to compile needs -std=C++26 and glibc 3.0, and I'm running on a HPC cluster where I can't use a system level package manager, and I don't want to be arsed to dockerize every small thing I want to run. For scientific and ML uses, conda has basically solved the whole "python packaging is a mess" that people seem to still complain about, at least on the end-user side. Sure, conda is slow as hell but there's a drop in replacement (mamba) that solves that issue.
- josephg 3y ago
  Conda? Mamba? Or should I use Venv? What are the commands to “activate” an environment? And why do I have to do that anyway, given thats not needed in any other programming language? Which of those systems support nested dependencies properly? Do any of them support the dependency graph containing multiple, mutually incompatible copies of the same package? Coming from rust (and nodejs before that), the package management situation in python feels like a mess. It’s barely better than C and C++ - both of which are also a disaster. (Make? Autotools? CMake? Use vendored dependencies? System dependencies? Which openblas package should I install from apt? Are any of them recent enough? Kill me.) Node: npm install. npm start. Rust: cargo run. Cargo run —-release. I don’t want to pick from 18 flavours of “virtual environments” that I have to remember how to to “activate”. And I don’t want to deal with transitive dependency conflicts, and I don’t want to be wading through my distro’s packages to figure out how to manually install dependencies. I just want to run the program. Python and C both make that much more difficult than it needs to be.
  - baq 3y ago
    > npm install. npm start. Yeah, no… my current $DAYJOB has a confusing mix of nx, pnpm, npm commands in multiple projects. Python is bad but node is absolutely not a good example.
    - josephg 3y ago
      Eh. Node / npm on its own is generally fine, especially if you use a private package repository for internally shared libraries. The problems show up when compiling javascript for the web as well as nodejs. If you stick to server side javascript using node and npm, it all works pretty well. It’s much nicer than venv / conda. And it handles transitive dependency conflicts and all sorts of wacky problems. It’s just that almost nobody does that. What we want instead is to combine typescript, js bundlers, wasm, es modules and node packages all together to make web apps. And that’s more than enough to bring seasoned engineers to tears. Let alone adding in svelte / solidjs compilation on top of all that. I have sweats just thinking about it.
vachina 3y ago
You can “cd llama.cpp && make && ./main" because the developers made that happen. A self contained zero dependency Python project is also literally “python app.py”
- peatmoss 3y ago
  This is true, though in practice I so often find large, non-trivial examples of the former, but very seldom find examples of the latter. The barriers to entry are lower with python, and so I think there tends to be a lower standard for deployment.
ojosilva 3y ago
Python sits on the C-glue segment of programming languages (where Perl, PHP, Ruby and Node are also notable members). Being a glue language means having APIs to a lot of external toolchains written in not only C/C++ but many other compiled languages, APIs and system resources. Conda, virtualenv, etc. are godsend modules for making it all work, or even better, to freeze things once they all work, without resourcing to Docker, VMs or shell scripts. It's meant for application and DevOps people who need to slap together, ie, ML, Numpy, Elasticsearch, AWS APIs and REST endpoints and Get $hit Done. It's annoying to see them "glueys" compared to the binary compiled segment where the heavy lifting is done. Python and others exist to latch on and assimilate. Resistance is futile: https://pypi.org/project/pyllamacpp/ https://pypi.org/project/pyllamacpp/ https://www.npmjs.com/package/llama-node https://www.npmjs.com/package/llama-node https://packagist.org/packages/kambo/llama-cpp-php https://packagist.org/packages/kambo/llama-cpp-php https://github.com/yoshoku/llama_cpp.rb https://github.com/yoshoku/llama_cpp.rb

deleted 3y ago

[deleted]

superkuh 3y ago

Pytorch (+GPU) dependency and python container type diversity are particularly bad. Programmers may not perceive this since they're already managing their python environment keeping all the OS/libs/containers/applications in the alignment required for things to work but it's quite complex. I couldn't do it. In comparison I could just type git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and make . And it worked. And since then I've managed to get llama.cpp clBLAS partial GPU acceleration working with my AMD RX 580 8GB. Plus with the llama.cpp CPU mmap stuff I can run multiple LLM IRC bot processes using the same model all sharing the RAM representation for free. Are there even ways to run 2 or 3 bit models in pytorch implementations like llama.cpp can do? It's pretty rad I could run a 65B llama in 27 GB of RAM on my 32GB RAM system (and still get better perplexity than 30B 8 bit).

manfre 3y ago
> In comparison I could just type git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and make . And it worked. You're comparing a single, well managed project that had put effort into user onboarding against all projects of a different language and proclaiming that an entire language/ecosystem is crap. The only real take away is that many projects, independent of language, put way too little effort towards onboarding users.
- superkuh 3y ago
  That's exactly it. Who wants to try pulling in an entire language with wide depenencies and it's ecosystem of envs/containers/etc when a single program will do it? Not people who just want to run inference on pre-made models.

slashtom 3y ago

llama cpp is just cpp inference on the llama model. PyTorch is a library to train neural networks. I'm not sure why people are conflating these two totally different projects..

whywhywhywhy 3y ago

>Easy deployment on iOS I guess? Deploying a PyTorch running Python app to anything in a way a user can just run it is a struggle. Even iOS aside that’s not a small use case that’s all local and offline ML potential. Really makes me wonder about the whole language, did they ever expect the code to have to run elsewhere than the machines the writer controls.