3 ms·

This question keeps popping up but I don't get it. Everyone and their dog has an OpenAI-compatible API. Why not just serve a local LLM and put api.openai.com 12

• tbocek 2y ago
  There is a difference between chat and code completion. While with chat, you can use localhost with llama.cpp, but code completion you cannot do that: https://github.com/zed-industries/zed/issues/12519 https://github.com/zed-industries/zed/issues/12519. The config for chat, you can do: "language_models": { "openai": { "version": "1", "api_url": "http://localhost:8080", "low_speed_timeout_in_seconds": 120, "available_models": [ { "provider": "openai", "name": "Qwen2.5-Coder-7B-Instruct-Q8_0.gguf", "display_name": "llama.cpp", "max_tokens": 131072 } ] } }, While for code completion, you have two choices atm: supermaven and copilot: https://zed.dev/docs/completions https://zed.dev/docs/completions.
  • aargh_aargh 2y ago
    Thank you, makes sense. I haven't used code completion yet.