3 ms·

Why is it necessary to sign up for two external services if this is running the LLM locally?

• mchiang 3y ago
  No need to sign up for any service to use Continue with Ollama. This will do the inference all locally. I believe if you want to do the inference using Replicate or Together, you’ll have to sign up for their services.
  • Xiol32 3y ago
    I don't have a Mac. So it's either have a Mac or don't run locally if you want to use this service?
    • mchiang 3y ago
      Sorry about that. We are working on building the Windows and Linux versions. May I ask which specific OS you are on?
      • hawk_ 3y ago
        Not OP, but would be great to have it working on Linux. How can one help?
  • sestinj 3y ago
    ^exactly, Replicate or Together are only options that are potentially more convenient
• deleted 3y ago
  [deleted]
• deleted 3y ago
  [deleted]
• daemonologist 3y ago
  You can cobble together an OpenAI-esque server locally with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/2766 https://github.com/ggerganov/llama.cpp/issues/2766 and the script here: https://www.reddit.com/r/LocalLLaMA/comments/15ak5k4/short_guide_to_hosting_your_own_llamacpp_openai/ https://www.reddit.com/r/LocalLLaMA/comments/15ak5k4/short_g... and use Continue's GGML Model option to query it: https://continue.dev/docs/customization#local-models-with-ggml https://continue.dev/docs/customization#local-models-with-gg... (I've gotten this working, more or less, but don't have the hardware to make it practical so I can't give any feedback about Continue or the model.)
  • Havoc 3y ago
    Amazing. Was looking for instructions to do precisely that