5 ms·

I love the concept, but I'd really prefer being able to use it against local llms (localai, ollama, etc).

• edunteman 3y ago
  Seems like a great feature (and honestly allows us to do smarter things for strictly structured generation). I'm curious, what's your main motivation for local llms vs hosted APIs?
  • imtringued 3y ago
    I'm pretty sure the primary reason is that you don't want an update of the hosted LLM to suddenly break your application without warning.
  • ramses0 3y ago
    Not wanting to get dinged for $20/mo? Ability to use offline for local home automation (eg: "given a verbal input request, determine the home devices in scope and their on/off state" => "given the current devices state and the verbal request, generate a list of home assistant actions to perform"), using a custom model for the above, etc.
    • yunohn 3y ago
      > Not wanting to get dinged for $20/mo? OpenAI (& peer) API pricing is honestly quite cheap - I spend way less than 20$ a month for my needs. On the other hand, ChatGPT offers a different experience, so I pay for that too. https://help.openai.com/en/articles/7127956-how-much-does-gpt-4-cost https://help.openai.com/en/articles/7127956-how-much-does-gp...
• nate_nowack 3y ago
  as with marvin, you can just swap the base url and use any of the oss proxy libs that clone the openai api (but since they don't do function calling [except for mistral i think], its not as good afaik)