3 ms·
I'd love a way to do this locally -- pool all the PCs in our own office for in-office pools of compute. Any suggestions from anyone? We currently run ollama but
by TuringNYC 6mo ago
I'd love a way to do this locally -- pool all the PCs in our own office for in-office pools of compute. Any suggestions from anyone? We currently run ollama but manually manage the pools
- damezumari 6mo agohttps://github.com/exo-explore/exo https://github.com/exo-explore/exo
- utopiah 6mo agoSeems like so much more work than "just" paying for https://huggingface.co https://huggingface.co or whichever other neocloud who already did all the setup for you and just waits for your credit card per minute/seconds/token.
- TuringNYC 6mo agoIt is much more work because for many workloads you have geographic ringfencing and cannot send it out to the cloud
- utopiah 6mo agoDoubt this kind of workloads would agree to send data then to a cloud of randos devices, precisely when cloud providers to certify they aren't looking at clients data (Customer-managed encryption keys, CMEK).
- TuringNYC 6mo ago>> Doubt this kind of workloads would agree to send data then to a cloud of randos devices, Totally agree, which is why i said "I'd love a way to do this locally -- pool all the PCs in our own office for in-office pools of compute."
- zozbot234 6mo agoIf you set CPUSchedulingPolicy=idle Nice=19 IOSchedulingClass=idle in the ollama server configuration it should run in the background with lowest priority.