4 ms·

Nice, I wish it was a little easier to integrate these models into Chat UIs like the one from Vercel or even a simple Gradio app. Does anyone have any Spaces/C

Nice, I wish it was a little easier to integrate these models into Chat UIs like the one from Vercel or even a simple Gradio app. Does anyone have any Spaces/Colab notebooks/etc to try this out on? Thanks!

brucethemoose2 3y ago
There are many UIs for running locally, but the easiest is koboldcpp: https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp Its a llama.cpp wrapper descended from the roleplaying community, but works fine (and performantly) for questioning and such. You will need to download the model from HF quantize it yourself: https://github.com/ggerganov/llama.cpp#prepare-data--run https://github.com/ggerganov/llama.cpp#prepare-data--run
- brianjking 3y ago
  Plenty of ways to run locally - I'm looking for ways for inference via Colab or Huggingface.
  - brucethemoose2 3y ago
    Oh, well you cab login to huggingface and deploy it in a space with a button above the model. Not sure about colab at the moment.
    - brianjking 3y ago
      That can give you an inference endpoint for the API, I'm talking about a full chat UI where you can set the temperature, etc.
      - spmurrayzzz 3y ago
        This isnt specifically Colab or HF, but have you checked out any of the community Runpod templates? Theres a few out there that give you a mostly turnkey way to deploy models and test via Oobabooga, koboldai, or similar. I use the one-click UI from TheBloke pretty frequently for inference testing, and I know theres some newer ones that also give you fine-tuning capabilities as well.
        brianjking 3y ago
        Link to the runpod templates? These run on your local machine or can I use Colab/Huggingface GPUs?
        spmurrayzzz 3y ago
        https://www.runpod.io/console/templates https://www.runpod.io/console/templates This is the readme for the one I mentioned: https://github.com/TheBlokeAI/dockerLLM/blob/main/README_Runpod_LocalLLMsUIandAPI.md https://github.com/TheBlokeAI/dockerLLM/blob/main/README_Run... > can I use Colab/Huggingface GPUs? You use these templates on the runpod platform itself. Theres no free tier equivalent like you have with Colab/HF, but currently you can rent an RTX 4090 for $0.69/hr so its pretty affordable.
        brianjking 3y ago
        Yeah, but I'd like to try deploying this on Huggingface or Lambda Labs. Do you know if that is possible?
        spmurrayzzz 3y ago
        In theory yes, since its fundamentally just a Dockerfile, any provider that supports docker is possible. I know HF Spaces have a "docker" sdk option[1], but I personally haven't used them myself. [1] https://huggingface.co/docs/hub/spaces-sdks-docker-first-demo https://huggingface.co/docs/hub/spaces-sdks-docker-first-dem...
ccooffee 3y ago
I've found https://chat.lmsys.org/ https://chat.lmsys.org/ to be a useful multi-LLM chat app without scary ToS data-mining clauses.

brucethemoose2 3y ago

There are many UIs for running locally, but the easiest is koboldcpp: https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp Its a llama.cpp wrapper descended from the roleplaying community, but works fine (and performantly) for questioning and such. You will need to download the model from HF quantize it yourself: https://github.com/ggerganov/llama.cpp#prepare-data--run https://github.com/ggerganov/llama.cpp#prepare-data--run

brianjking 3y ago
Plenty of ways to run locally - I'm looking for ways for inference via Colab or Huggingface.
- brucethemoose2 3y ago
  Oh, well you cab login to huggingface and deploy it in a space with a button above the model. Not sure about colab at the moment.
  - brianjking 3y ago
    That can give you an inference endpoint for the API, I'm talking about a full chat UI where you can set the temperature, etc.
    - spmurrayzzz 3y ago
      This isnt specifically Colab or HF, but have you checked out any of the community Runpod templates? Theres a few out there that give you a mostly turnkey way to deploy models and test via Oobabooga, koboldai, or similar. I use the one-click UI from TheBloke pretty frequently for inference testing, and I know theres some newer ones that also give you fine-tuning capabilities as well.
      - brianjking 3y ago
        Link to the runpod templates? These run on your local machine or can I use Colab/Huggingface GPUs?
        spmurrayzzz 3y ago
        https://www.runpod.io/console/templates https://www.runpod.io/console/templates This is the readme for the one I mentioned: https://github.com/TheBlokeAI/dockerLLM/blob/main/README_Runpod_LocalLLMsUIandAPI.md https://github.com/TheBlokeAI/dockerLLM/blob/main/README_Run... > can I use Colab/Huggingface GPUs? You use these templates on the runpod platform itself. Theres no free tier equivalent like you have with Colab/HF, but currently you can rent an RTX 4090 for $0.69/hr so its pretty affordable.
        brianjking 3y ago
        Yeah, but I'd like to try deploying this on Huggingface or Lambda Labs. Do you know if that is possible?
        spmurrayzzz 3y ago
        In theory yes, since its fundamentally just a Dockerfile, any provider that supports docker is possible. I know HF Spaces have a "docker" sdk option[1], but I personally haven't used them myself. [1] https://huggingface.co/docs/hub/spaces-sdks-docker-first-demo https://huggingface.co/docs/hub/spaces-sdks-docker-first-dem...

ccooffee 3y ago

I've found https://chat.lmsys.org/ https://chat.lmsys.org/ to be a useful multi-LLM chat app without scary ToS data-mining clauses.