4 ms·

LMStudio? No, it's the easiest way to run am LLM locally that I've seen to the point where I've stopped looking at other alternatives. It's cross-platform (Wi

LMStudio? No, it's the easiest way to run am LLM locally that I've seen to the point where I've stopped looking at other alternatives. It's cross-platform (Win/Mac/Linux), detects the most appropriate GPU in your system and tells you whether the model you want to download will run within it's RAM footprint. It lets you set up a local server that you can access through API calls as if you were remotely connected to an online service.

vunderba 9mo ago
FWIW, Ollama already does most of this: - Cross-platform - Sets up a local API server The tradeoff is a somewhat higher learning curve, since you need to manually browse the model library and choose the model/quantization that best fit your workflow and hardware. OTOH, it's also open-source unlike LMStudio which is proprietary.
- randallsquared 9mo ago
  I assumed from the name that it only ran llama-derived models, rather than whatever is available at huggingface. Is that not the case?
  - fenykep 9mo ago
    No, they have quite a broad list of models: https://ollama.com/search https://ollama.com/search [edit] Oh and apparently you can also directly run some models directly from HuggingFace: https://huggingface.co/docs/hub/ollama https://huggingface.co/docs/hub/ollama
    - ashirviskas 9mo ago
      Just use llama.cpp. Ollama tried to force their custom API (not the openai standard), they obscure the downloaded models making them a pain to use with other implementations, blatantly used llama.cpp as a thin wrapper without communicating it properly and now has to differentiate somehow to start making money. If you've ever used a terminal, use llama.cpp. You can also directly run models from llama.cpp afaik.
      - fenykep 9mo ago
        Yes, I wanted to try it already but setting up an environment with an MI50 was a bit tricky so I wanted to try something I knew first. Now that I have ollama running I will give llama.cpp a shot.
        ashirviskas 9mo ago
        Ooh, I have experience with it. If you're on linux, just use Vulkan. If you face any other issues, just google my username + "MI50 32GB vbios reddit". It depends on which vBIOS you have, but that post on reddit has most of the info you may need. Good luck!

vunderba 9mo ago

FWIW, Ollama already does most of this: - Cross-platform - Sets up a local API server The tradeoff is a somewhat higher learning curve, since you need to manually browse the model library and choose the model/quantization that best fit your workflow and hardware. OTOH, it's also open-source unlike LMStudio which is proprietary.

randallsquared 9mo ago
I assumed from the name that it only ran llama-derived models, rather than whatever is available at huggingface. Is that not the case?
- fenykep 9mo ago
  No, they have quite a broad list of models: https://ollama.com/search https://ollama.com/search [edit] Oh and apparently you can also directly run some models directly from HuggingFace: https://huggingface.co/docs/hub/ollama https://huggingface.co/docs/hub/ollama
  - ashirviskas 9mo ago
    Just use llama.cpp. Ollama tried to force their custom API (not the openai standard), they obscure the downloaded models making them a pain to use with other implementations, blatantly used llama.cpp as a thin wrapper without communicating it properly and now has to differentiate somehow to start making money. If you've ever used a terminal, use llama.cpp. You can also directly run models from llama.cpp afaik.
    - fenykep 9mo ago
      Yes, I wanted to try it already but setting up an environment with an MI50 was a bit tricky so I wanted to try something I knew first. Now that I have ollama running I will give llama.cpp a shot.
      - ashirviskas 9mo ago
        Ooh, I have experience with it. If you're on linux, just use Vulkan. If you face any other issues, just google my username + "MI50 32GB vbios reddit". It depends on which vBIOS you have, but that post on reddit has most of the info you may need. Good luck!