9 ms·

That is fantastic. I'm building a small macOS SwiftUI client with llama cpp built in, no server-client model, and it's already so useful with models like openhe

by keyle 3y ago

That is fantastic. I'm building a small macOS SwiftUI client with llama cpp built in, no server-client model, and it's already so useful with models like openhermes chat 7B, and fast. If this opens it to smaller laptops, wow! We truly live in crazy time. The rate of improvement in this field is off the walls.

turnsout 3y ago
That sounds awesome! Can you share any details about how you're working with llama cpp? Is it just via the Swift <> C bridge? I've toyed with the idea of doing this, and wonder if you have any pointers before I get started.
joshmarlow 3y ago
Not sure if this is where your head is, but I think there's a lot of value in integrating LLMs directly into complex software. Jira, Salesforce, maybe K8s - should all have an integrated LLMs that can walk you through how to perform a nuanced task in the software.
- dpacmittal 3y ago
  Why would the LLM walk you through and not just do the nuanced task on its own?
  - pennomi 3y ago
    I assume the human maintains some of the necessary context in their meat memory.
  - joshmarlow 3y ago
    IMO, for many real business use cases, the hallucinations are still a big deal. Once we have models that are more reliable, I think it makes sense to go down that path - the AI is the interface to the software. But until we're there, a system that just provides guidance that the user can validate is a good stepping stone - and one I suspect is immediately feasible!
- debarshri 3y ago
  Walkthrough is generally performed once or not so frequently. It would be a bad investment if you just use it for just this use case
  - alpaca128 3y ago
    A beginner tutorial is also not used frequently by users, but that doesn't make it a bad investment. I an LLM can help a lot with getting familiar with the tool it could be pretty valuable, especially after a UI rework etc.
- manmal 3y ago
  Imagine good error messages, with hints for mitigation and maybe smart retry w/ mitigations applied.
emadm 3y ago
3b is good for 8gb MacBook Air etc. 7b is slightly too big. Sure these will continue to improve, phi2 is a good base as well

by keyle 3y ago

turnsout 3y ago

That sounds awesome! Can you share any details about how you're working with llama cpp? Is it just via the Swift <> C bridge? I've toyed with the idea of doing this, and wonder if you have any pointers before I get started.

joshmarlow 3y ago

Not sure if this is where your head is, but I think there's a lot of value in integrating LLMs directly into complex software. Jira, Salesforce, maybe K8s - should all have an integrated LLMs that can walk you through how to perform a nuanced task in the software.

dpacmittal 3y ago
Why would the LLM walk you through and not just do the nuanced task on its own?
- pennomi 3y ago
  I assume the human maintains some of the necessary context in their meat memory.
- joshmarlow 3y ago
  IMO, for many real business use cases, the hallucinations are still a big deal. Once we have models that are more reliable, I think it makes sense to go down that path - the AI is the interface to the software. But until we're there, a system that just provides guidance that the user can validate is a good stepping stone - and one I suspect is immediately feasible!
debarshri 3y ago
Walkthrough is generally performed once or not so frequently. It would be a bad investment if you just use it for just this use case
- alpaca128 3y ago
  A beginner tutorial is also not used frequently by users, but that doesn't make it a bad investment. I an LLM can help a lot with getting familiar with the tool it could be pretty valuable, especially after a UI rework etc.
manmal 3y ago
Imagine good error messages, with hints for mitigation and maybe smart retry w/ mitigations applied.

emadm 3y ago

3b is good for 8gb MacBook Air etc. 7b is slightly too big. Sure these will continue to improve, phi2 is a good base as well