3 ms·
GCP system to run llama2 Chat model 7B 13B and 70B
which system is best to run llama2 Chat 7B 13B and 70B model on GCP for production? So it can handle multiple requests and mention multiple user chat sessions at the same time. And do load balancing effectively
- tikkun 3y agoLook into vllm and text-generation-inference Why not use an API though? Some to check out are: Replicate, Together AI, Fireworks AI, Deepinfra, Anyscale Endpoints
- extasia 3y agoI can personally recommend deepinfra. Been using it the past few days and it's served me very well.
- yeesian 3y agohttps://cloud.google.com/vertex-ai/docs/start/explore-models#list_of_models_with_open-source_tuning_or_serving_recipes_on https://cloud.google.com/vertex-ai/docs/start/explore-models...
- gongy 3y agoYou can have load balanced inference and also fine tune models on Modal (disclaimer: I wrote this guide) You can get a load balanced endpoint serving hundreds of tok/s per GPU using vLLM or Text Generation Inference in a few lines of code: https://modal.com/docs/guide/ex/text_generation_inference https://modal.com/docs/guide/ex/text_generation_inference