7 ms·

I’m in the same boat. With text-to-image models I pick out software of my choice and drop a downloaded model into an input directory, then launch a GUI. What

I’m in the same boat. With text-to-image models I pick out software of my choice and drop a downloaded model into an input directory, then launch a GUI. What is it about LLMs that makes actually being able to run them locally so complex in comparison? I’m genuinely curious as a layman.

gcr 3y ago
Nothing in particular. Setting up a scientific python+pytorch stack is difficult if you're unfamiliar with the python packaging ecosystem. If you're not on a "happy path" of "ubuntu, nvidia, and anaconda," then lots of things could go wrong if you're configuring from scratch. If you want to run these models efficiently, hardware acceleration is a must, but managing the intersection between {GPU, operating system, architecture, Python version, Python virtualenv location} is tricky. That's even before you deal with hardware-specific implementation details! - Running NVidia? Double-check your CUDA library version, kernel module version, (optional) CuDNN, and pytorch version - Running AMD? Double-check your ROCm library version, AMD drivers, and make sure that you use Pytorch provided by AMD with ROCm support - On Apple machines? Double-check that your M1 hardware actually has proper hardware support, then download and install a custom Pytorch distribution linked with M1 support, and make sure that the numpy library version has been properly linked with Accelerate.framework or else your BLAS calls run on the CPU rather than the undocumented AMX coprocessor. If you want to run on the ANE, you'll additionally need a working xcode toolchain and a version of the CoreML model compiler that can read your serialized pytorch model files properly. I think the pain of getting things working makes it easier to just throw up one's hands and pay someone else to run your model for you.
- simonw 3y ago
  It is wild to me how hard it is to run these things GPU-accelerated on an Apple M1/M2. The hardware support should be amazing for this, given that the CPU and GPU share the same RAM. I mostly end up running the CPU versions and grumbling about how slow they are.
  - bugglebeetle 3y ago
    GPU acceleration is pretty easy with llama.cpp. You just run make with an extra flag and then an argument or two at runtime.
    - simonw 3y ago
      Therein lies the problem. I want to write instructions for people to use my software that don't expect them to know how to run make, or even to have a C compiler installed.
      - bugglebeetle 3y ago
        Then you more or less need a GUI like OpenAI built with ChatGPT so you control the whole environment. Even setting up LLM in Homebrew required me to do the whole install twice because of some arcane error.
        simonw 3y ago
        I think I can fix that by shipping a bottle release - "brew install simonw/llm/llm" currently attempts to compile Pydantic's Rust extensions, which means it runs incredibly slowly. I built a bottle for it which installs much faster, but you currently have to download the file from https://static.simonwillison.net/static/2023/llm--0.5.arm64_ventura.bottle.1.tar.gz https://static.simonwillison.net/static/2023/llm--0.5.arm64_... - I've not yet figured out how to get that to install when the user runs "brew install simonw/llm/llm" - issue here: https://github.com/simonw/llm/issues/102 https://github.com/simonw/llm/issues/102
    - simonw 3y ago
      Adrien Brault on Twitter gave me this recipe, which worked perfectly: https://gist.github.com/adrienbrault/b76631c56c736def9bc1bc2167b5d129 https://gist.github.com/adrienbrault/b76631c56c736def9bc1bc2...
  - smoldesu 3y ago
    > The hardware support should be amazing for this I mean, caveat emptor; CoreML has been barebones for years, and Apple isn't exactly notorious for huge commitment to third party APIs. The writing was on the wall with how Metal was rolled out and how fast OpenCL got dropped, it honestly doesn't surprise me at all at this point. Even the current Apple Silicon support in llama.cpp is fudged with NEON and Metal Shaders instead of Apple's "Neural Engine".
fomine3 3y ago
We know that GPT-4 is the king at this time, so only researchers, developers, and enthusiasts setup local LLMs. They don't need easier way to setup. For image generation, Local Stable Diffusion is the king for certain big genre of image generation. Many people including PC newbie try it, so there are many guides.
nl 3y ago
In general the default LLM models are too big to run on a consumer GPU without quantization. That means additional steps or alternative downloads are needed. Also generative graphics models are about 6 months further advanced than generative text models because Stable Diffusion came out before Llama (Llama being the first "useful" smaller-large-language-model)

gcr 3y ago

Nothing in particular. Setting up a scientific python+pytorch stack is difficult if you're unfamiliar with the python packaging ecosystem. If you're not on a "happy path" of "ubuntu, nvidia, and anaconda," then lots of things could go wrong if you're configuring from scratch. If you want to run these models efficiently, hardware acceleration is a must, but managing the intersection between {GPU, operating system, architecture, Python version, Python virtualenv location} is tricky. That's even before you deal with hardware-specific implementation details! - Running NVidia? Double-check your CUDA library version, kernel module version, (optional) CuDNN, and pytorch version - Running AMD? Double-check your ROCm library version, AMD drivers, and make sure that you use Pytorch provided by AMD with ROCm support - On Apple machines? Double-check that your M1 hardware actually has proper hardware support, then download and install a custom Pytorch distribution linked with M1 support, and make sure that the numpy library version has been properly linked with Accelerate.framework or else your BLAS calls run on the CPU rather than the undocumented AMX coprocessor. If you want to run on the ANE, you'll additionally need a working xcode toolchain and a version of the CoreML model compiler that can read your serialized pytorch model files properly. I think the pain of getting things working makes it easier to just throw up one's hands and pay someone else to run your model for you.

simonw 3y ago
It is wild to me how hard it is to run these things GPU-accelerated on an Apple M1/M2. The hardware support should be amazing for this, given that the CPU and GPU share the same RAM. I mostly end up running the CPU versions and grumbling about how slow they are.
- bugglebeetle 3y ago
  GPU acceleration is pretty easy with llama.cpp. You just run make with an extra flag and then an argument or two at runtime.
  - simonw 3y ago
    Therein lies the problem. I want to write instructions for people to use my software that don't expect them to know how to run make, or even to have a C compiler installed.
    - bugglebeetle 3y ago
      Then you more or less need a GUI like OpenAI built with ChatGPT so you control the whole environment. Even setting up LLM in Homebrew required me to do the whole install twice because of some arcane error.
      - simonw 3y ago
        I think I can fix that by shipping a bottle release - "brew install simonw/llm/llm" currently attempts to compile Pydantic's Rust extensions, which means it runs incredibly slowly. I built a bottle for it which installs much faster, but you currently have to download the file from https://static.simonwillison.net/static/2023/llm--0.5.arm64_ventura.bottle.1.tar.gz https://static.simonwillison.net/static/2023/llm--0.5.arm64_... - I've not yet figured out how to get that to install when the user runs "brew install simonw/llm/llm" - issue here: https://github.com/simonw/llm/issues/102 https://github.com/simonw/llm/issues/102
  - simonw 3y ago
    Adrien Brault on Twitter gave me this recipe, which worked perfectly: https://gist.github.com/adrienbrault/b76631c56c736def9bc1bc2167b5d129 https://gist.github.com/adrienbrault/b76631c56c736def9bc1bc2...
- smoldesu 3y ago
  > The hardware support should be amazing for this I mean, caveat emptor; CoreML has been barebones for years, and Apple isn't exactly notorious for huge commitment to third party APIs. The writing was on the wall with how Metal was rolled out and how fast OpenCL got dropped, it honestly doesn't surprise me at all at this point. Even the current Apple Silicon support in llama.cpp is fudged with NEON and Metal Shaders instead of Apple's "Neural Engine".

fomine3 3y ago

We know that GPT-4 is the king at this time, so only researchers, developers, and enthusiasts setup local LLMs. They don't need easier way to setup. For image generation, Local Stable Diffusion is the king for certain big genre of image generation. Many people including PC newbie try it, so there are many guides.

nl 3y ago

In general the default LLM models are too big to run on a consumer GPU without quantization. That means additional steps or alternative downloads are needed. Also generative graphics models are about 6 months further advanced than generative text models because Stable Diffusion came out before Llama (Llama being the first "useful" smaller-large-language-model)