Yes, it runs totally fine. I ran it in Oobabooga/text generation web ui. Nice thing about it is that it autodownloads all necessary gpu binaries on it's own and

Yes, it runs totally fine. I ran it in Oobabooga/text generation web ui. Nice thing about it is that it autodownloads all necessary gpu binaries on it's own and creates a isolated conda env. I asked same questions on the official 70b demo and got same answers. I even got better answers with ooba, since the demo cuts text early Ooobabooga: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui Model: TheBloke_Llama-2-70B-chat-GPTQ from https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ https://huggingface.co/TheBloke/Llama-2-70B-chat-GPTQ ExLlama_HF loader gpu split 20,22, context size 2048 on the Chat Settings tab, choose Instruction template tab and pick Llama-v2 from the instruction template dropdown Demo: https://huggingface.co/blog/llama2#demo https://huggingface.co/blog/llama2#demo