3 ms·

Thanks for the insightful response. This is exactly the type of answer I was looking for. What's the best way to educate myself on the end-to-end process of dep

Thanks for the insightful response. This is exactly the type of answer I was looking for. What's the best way to educate myself on the end-to-end process of deploying a production grade model smartly in a cost efficient manner?

FezzikTheGiant 2y ago
This might be asking for too much but is there a guide that explains each part of this process? Your comment made the higher level way clearer for me and I'd like to go into the weeds a bit on each of these
- fzzzy 2y ago
  download llama.cpp convert the fine tuned model into gguf format. choose a number of quantization bits such that the final gguf will fit in your free ram + vram run the llama.cpp server binary. choose the -ngl number of graphics layers which is the max number that will not overflow your vram (i just determine it experimentally, i start with the full number of layers, divide by two if it runs out of vram, multiply by 1.5 if there is enough vram, etc) make sure to set the temperature to 0 if you are doing facts based language conversion and not creative tasks if it's too slow, get more vram ollama, kobold.cpp, and just running the model yourself with a python script as described by the original commenter are also options, but the above is what i have been enjoying lately. everyone else in this thread is saying you need gpus but this really isn't true. what you need is ram. if you are trying to get a model that can reason you really want the biggest model possible. the more ram you have the less quantized you have to make your production model. if you can batch your requests and get the result a day later, you just need as much ram as you can get and it doesn't matter how many tokens per second you get. if you are doing creative generation then this doesn't matter nearly as much. if you need realtime then it gets extremely expensive fast to get enough vram to host your whole model (assuming you want as large a model as possible for better reasoning capability)
  - FezzikTheGiant 2y ago
    Interesting. Thanks for the response. Do you have any resources where I can educate myself about this? How did you learn what you know about LLMs?
    - elliotto 2y ago
      Hi, I work at a startup where we train / fine tune / inference models on a gcp kubernetes cluster on some a100s. There isn't really that much information about how to do this properly because everyone is working it out and it changes month by month. It requires a bunch of DevOps an infrastructure knowledge above and beyond the raw ml knowledge. Your best bet is probably just to tool around and see what you can do.
    - fzzzy 2y ago
      Well, when Llama 1 came out I signed up and downloaded it, and that led me to llama.cpp. I followed the instructions to quantize the model to fit in my graphics card. Then later when more models like llama2 and mixtral came out I would download and evaluate them. I kept up on hacker news posts and any comments about things I didn't understand. I've also found the localllama subreddit to be a great way to learn. Any time I saw a comment on anything I would try it, like ollama, kobold.cpp, sillytavern, textgen-webui, and more. I also have a friend who has been into ai for many years and we always exchange links to new things. I developed a retrieval augmented generation (rag) app with him and a "transformation engine" pipeline. So following ai stories on hn and reddit, learning through doing, and applying what I learned to real projects.
      - FezzikTheGiant 2y ago
        Thanks. Very cool. Have you ever tried to implement a transformer from scratch? Like in the Attention is all you need paper? Can a first/second year college student do it
        xyc 2y ago
        Andrej Karpathy's course is a good resource: https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs...
d13 2y ago
Try this: https://github.com/mlabonne/llm-course https://github.com/mlabonne/llm-course
- FezzikTheGiant 2y ago
  Thanks!! This is really cool

FezzikTheGiant 2y ago

This might be asking for too much but is there a guide that explains each part of this process? Your comment made the higher level way clearer for me and I'd like to go into the weeds a bit on each of these

fzzzy 2y ago
download llama.cpp convert the fine tuned model into gguf format. choose a number of quantization bits such that the final gguf will fit in your free ram + vram run the llama.cpp server binary. choose the -ngl number of graphics layers which is the max number that will not overflow your vram (i just determine it experimentally, i start with the full number of layers, divide by two if it runs out of vram, multiply by 1.5 if there is enough vram, etc) make sure to set the temperature to 0 if you are doing facts based language conversion and not creative tasks if it's too slow, get more vram ollama, kobold.cpp, and just running the model yourself with a python script as described by the original commenter are also options, but the above is what i have been enjoying lately. everyone else in this thread is saying you need gpus but this really isn't true. what you need is ram. if you are trying to get a model that can reason you really want the biggest model possible. the more ram you have the less quantized you have to make your production model. if you can batch your requests and get the result a day later, you just need as much ram as you can get and it doesn't matter how many tokens per second you get. if you are doing creative generation then this doesn't matter nearly as much. if you need realtime then it gets extremely expensive fast to get enough vram to host your whole model (assuming you want as large a model as possible for better reasoning capability)
- FezzikTheGiant 2y ago
  Interesting. Thanks for the response. Do you have any resources where I can educate myself about this? How did you learn what you know about LLMs?
  - elliotto 2y ago
    Hi, I work at a startup where we train / fine tune / inference models on a gcp kubernetes cluster on some a100s. There isn't really that much information about how to do this properly because everyone is working it out and it changes month by month. It requires a bunch of DevOps an infrastructure knowledge above and beyond the raw ml knowledge. Your best bet is probably just to tool around and see what you can do.
  - fzzzy 2y ago
    Well, when Llama 1 came out I signed up and downloaded it, and that led me to llama.cpp. I followed the instructions to quantize the model to fit in my graphics card. Then later when more models like llama2 and mixtral came out I would download and evaluate them. I kept up on hacker news posts and any comments about things I didn't understand. I've also found the localllama subreddit to be a great way to learn. Any time I saw a comment on anything I would try it, like ollama, kobold.cpp, sillytavern, textgen-webui, and more. I also have a friend who has been into ai for many years and we always exchange links to new things. I developed a retrieval augmented generation (rag) app with him and a "transformation engine" pipeline. So following ai stories on hn and reddit, learning through doing, and applying what I learned to real projects.
    - FezzikTheGiant 2y ago
      Thanks. Very cool. Have you ever tried to implement a transformer from scratch? Like in the Attention is all you need paper? Can a first/second year college student do it
      - xyc 2y ago
        Andrej Karpathy's course is a good resource: https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs...

d13 2y ago

Try this: https://github.com/mlabonne/llm-course https://github.com/mlabonne/llm-course

FezzikTheGiant 2y ago
Thanks!! This is really cool