3 ms·
FWIW: you can use any other proxy server for this to any openai compatible api server. e.g. with mitmproxy and llama-cpp-python server python -m llama_cpp.s
by SushiHippie 3y ago
FWIW: you can use any other proxy server for this to any openai compatible api server.
e.g. with mitmproxy and llama-cpp-python server
python -m llama_cpp.server --n_ctx 4096 --n_gpu_layers 1 --model ./path/to/..gguf
and then with mitmproxy in another terminal
mitmproxy -p 5001 --mode reverse:http://127.0.0.1:8000
and then set this in your vscode settings.json (the same as for localpilot):
"github.copilot.advanced": {
"debug.testOverrideProxyUrl": "http://localhost:5001",
"debug.overrideProxyUrl": "http://localhost:5001"
}
works way better for me than localpilot
- cgrs 3y agoJust out of curiosity, why do you need a reverse proxy? Couldn't you connect to that openai-compatible server directly?
- SushiHippie 3y agoThanks, I don't know why I didn't think of that. Just tried it and it works at least with the llama-cpp-python server, but I suppose some other servers and very likely hosted services (SNI, certificate subjects, ...) would have problems with that.
- newman314 3y agoI'm assuming that this means that I could also potentially connect to a local more powerful box?
- SushiHippie 3y agoYup
- hoschicz 3y agoHow is context-from-following-tokens implemented in the pure OAI API? I assumed there must be specialized models that have two separate context windows.