3 ms·

Ah. Well you can build a dockerfile (with the 4 bit triton repo) locally here, then deploy it: https://github.com/oobabooga/text-generation-webui#alternative-do

Ah. Well you can build a dockerfile (with the 4 bit triton repo) locally here, then deploy it: https://github.com/oobabooga/text-generation-webui#alternative-docker https://github.com/oobabooga/text-generation-webui#alternati... I havent explored beyond that. Actually there are tons of quantizatons on hugging face, just search for "4bit". 128g usually (but not always) means a Triton variant, ggml means a cpu variant. I know that vicuna 7/13B and the "uncensored" vicuna 7B are quantized on sf, but there are already some newer, supposedly better instruction following mixes.

jiggawatts 4y ago
This line jumps out at me right away: RUN git clone https://github.com/oobabooga/GPTQ-for-LLaMa /build This makes this Dockerfile completely non-reproducible. I know how to work around it, but seriously, is this how Python users in the AI/ML space do things regularly!? Just make something work "once" on "my machine" with the "current commit" and then call it a day? It's a really shocking experience coming form the Rust and C# ecosystems where everything is strongly versioned, packages have "lock" files, and there's even SHA hashes to really pin things down. Meanwhile in this world it's just: "Get the latest version of this thing. It won't work, of course, because it only worked for a week in 2023 when I did it, but good luck!"
- brucethemoose2 4y ago
  > Seriously, is this how Python users in the AI/ML space do things regularly!? Just make something work "once" on "my machine" with the "current commit" and then call it a day? Bingo! Welcome to ML land, you are truly starting to understand. Honestly the oobabooga repo is 100x better than average. Most ML repos are one off research papers or demonstrations that barely (or don't) work on precise python environments and then are completely abandoned. Others live in dependency hell, with CUDA hell in particular being very common.

jiggawatts 4y ago

This line jumps out at me right away: RUN git clone https://github.com/oobabooga/GPTQ-for-LLaMa /build This makes this Dockerfile completely non-reproducible. I know how to work around it, but seriously, is this how Python users in the AI/ML space do things regularly!? Just make something work "once" on "my machine" with the "current commit" and then call it a day? It's a really shocking experience coming form the Rust and C# ecosystems where everything is strongly versioned, packages have "lock" files, and there's even SHA hashes to really pin things down. Meanwhile in this world it's just: "Get the latest version of this thing. It won't work, of course, because it only worked for a week in 2023 when I did it, but good luck!"

brucethemoose2 4y ago
> Seriously, is this how Python users in the AI/ML space do things regularly!? Just make something work "once" on "my machine" with the "current commit" and then call it a day? Bingo! Welcome to ML land, you are truly starting to understand. Honestly the oobabooga repo is 100x better than average. Most ML repos are one off research papers or demonstrations that barely (or don't) work on precise python environments and then are completely abandoned. Others live in dependency hell, with CUDA hell in particular being very common.