3 ms·

[flagged]

llmtosser 1y ago
This is not true. No inference engine does all of: - Model switching - Unload after idle - Dynamic layer offload to CPU to avoid OOM
- ekianjo 1y ago
  this can be added to llama.cpp with llama.swap currently so even without Ollama you are not far off
mchiang 1y ago
sorry that you feel the way you feel. :( I'm not sure which package we use that is triggering this. My guess is llama.cpp based on what I see on social? Ollama has long shifted to using our own engine. We do use llama.cpp for legacy and backwards compatibility. I want to be clear it's not a knock on the llama.cpp project either. There are certain features we want to build into Ollama, and we want to be opinionated on the experience we want to build. Have you supported our past gigs before? Why not be more happy and optimistic in seeing everyone build their dreams (success or not). If you go build a project of your dreams, I'd be supportive of it too.
- Maxious 1y ago
  > Have you supported our past gigs before? Docker Desktop? One of the most memorable private equity rugpulls in developer tooling? Fool me once shame on you, fool me twice shame on me
dangoodmanUT 1y ago
Yes everyone should just write cpp to call local LLMs obviously
- otabdeveloper4 1y ago
  Yes, but llama.cpp already comes with a ready-made OpenAI-compatible inference server.
  - reverius42 1y ago
    I think people are getting hung up on the "llama.cpp" name and thinking they need to write C++ code to use it. llama.cpp isn't (just) a C++ library/codebase -- it's a CLI application, server application (llama-server), etc.
api 1y ago
> Repackaging existing software while literally adding no useful functionality was always their gig. Developers continue to be blind to usability and UI/UX. Ollama lets you just install it, just install models, and go. The only other thing really like that is LM-Studio. It's not surprising that the people behind it are Docker people. Yes you can do everything Docker does with Linux kernel and shell commands, but do you want to? Making software usable is often many orders of magnitude more work than making software work.
- otabdeveloper4 1y ago
  > Ollama lets you just install it, just install models, and go. So does the original llama.cpp. And you won't have to deal with mislabeled models and insane defaults out of the box.
  - lxgr 1y ago
    Can it easily run as a server process in the background? To me, not having to load the LLM into memory for every single interaction is a big win of Ollama.
    - otabdeveloper4 1y ago
      Yes, of course it can.
      - lxgr 1y ago
        I wouldn't consider that a given at all, but apparently there's indeed `llama-server` which looks promising! Then the only thing that's missing seems to be a canonical way for clients to instantiate that, ideally in some OS-native way (systemd, launchcd etc.), and a canonical port that they can connect to.
dang 1y ago
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- deleted 1y ago
  [deleted]

[flagged]

This is not true. No inference engine does all of: - Model switching - Unload after idle - Dynamic layer offload to CPU to avoid OOM

ekianjo 1y ago
this can be added to llama.cpp with llama.swap currently so even without Ollama you are not far off

mchiang 1y ago

sorry that you feel the way you feel. :( I'm not sure which package we use that is triggering this. My guess is llama.cpp based on what I see on social? Ollama has long shifted to using our own engine. We do use llama.cpp for legacy and backwards compatibility. I want to be clear it's not a knock on the llama.cpp project either. There are certain features we want to build into Ollama, and we want to be opinionated on the experience we want to build. Have you supported our past gigs before? Why not be more happy and optimistic in seeing everyone build their dreams (success or not). If you go build a project of your dreams, I'd be supportive of it too.

Maxious 1y ago
> Have you supported our past gigs before? Docker Desktop? One of the most memorable private equity rugpulls in developer tooling? Fool me once shame on you, fool me twice shame on me

dangoodmanUT 1y ago

Yes everyone should just write cpp to call local LLMs obviously

otabdeveloper4 1y ago
Yes, but llama.cpp already comes with a ready-made OpenAI-compatible inference server.
- reverius42 1y ago
  I think people are getting hung up on the "llama.cpp" name and thinking they need to write C++ code to use it. llama.cpp isn't (just) a C++ library/codebase -- it's a CLI application, server application (llama-server), etc.

api 1y ago

> Repackaging existing software while literally adding no useful functionality was always their gig. Developers continue to be blind to usability and UI/UX. Ollama lets you just install it, just install models, and go. The only other thing really like that is LM-Studio. It's not surprising that the people behind it are Docker people. Yes you can do everything Docker does with Linux kernel and shell commands, but do you want to? Making software usable is often many orders of magnitude more work than making software work.

otabdeveloper4 1y ago
> Ollama lets you just install it, just install models, and go. So does the original llama.cpp. And you won't have to deal with mislabeled models and insane defaults out of the box.
- lxgr 1y ago
  Can it easily run as a server process in the background? To me, not having to load the LLM into memory for every single interaction is a big win of Ollama.
  - otabdeveloper4 1y ago
    Yes, of course it can.
    - lxgr 1y ago
      I wouldn't consider that a given at all, but apparently there's indeed `llama-server` which looks promising! Then the only thing that's missing seems to be a canonical way for clients to instantiate that, ideally in some OS-native way (systemd, launchcd etc.), and a canonical port that they can connect to.

dang 1y ago

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

deleted 1y ago
[deleted]