3 ms·
I do the same thing with cloudflare tunnels and managing the cloudflare tunnel process and the llama.cpp server with systemd on my home internet. Have a 13B ru
by czhu12 2y ago
I do the same thing with cloudflare tunnels and managing the cloudflare tunnel process and the llama.cpp server with systemd on my home internet.
Have a 13B running on a 3070 with 16 gpu layers and the rest running off CPU.
Performs okay, but way cheaper than renting a GPU on the cloud.
- Melatonic 2y agoRead the other comment and also immediately thought of Cloudflare tunnel instead - is there a reason you chose that? Wondering if I should do the same with my old Titan XP (probably slower than your 3070 but it does have 12gb of vram)
- czhu12 2y agoNo reason other than that cloudflare is quite a nice reverse proxy and I already use it for managing my DNS's. Thus far, I've not noticed any issues with it, other than when my home wifi goes down.