4 ms·
It's a little bit ironic that the package is called "onprem" but the second line imports an external model from huggingface... from onprem import LLM u
by gitgud 3y ago
It's a little bit ironic that the package is called "onprem" but the second line imports an external model from huggingface...
from onprem import LLM
url = 'https://huggingface.co/TheBloke/CodeUp-Llama.....'
llm = LLM(url, n_gpu_layers=43) # see below for GPU information
Anyway looks like a great little project, nice work!
- keyle 3y agoI'm not the OP but I believe it is correct. It says 'RUN' ... LLM. Meaning you have to get it from somewhere. Training your own from nothing is a monumental task, I don't think many of us can realistically do it from scratch.
- scubbo 3y agoIt's still running on-prem if the execution happens on compute hardware that you control. If importing code from somewhere else disqualified something from being on-prem, then no application would ever be on-prem - when's the last time a meaningful application had no dependencies!?
- paxys 3y agoCode that runs on prem doesn't magically show up on your server, it still has to be fetched from somewhere.