4 ms·

Running llama-2-7b-chat at 8 bit quantization, and completions are essentially at GPT-3.5 levels (and instant) on a single RTX4090 using 15gb VRAM. I don't thin

Running llama-2-7b-chat at 8 bit quantization, and completions are essentially at GPT-3.5 levels (and instant) on a single RTX4090 using 15gb VRAM. I don't think most people realize just how small and efficient these models are going to become.

jeanloolz 3y ago
7B or 70B?
- ramesh31 3y ago
  >7B or 70B? 7B 8bit GGML running on a single 4090 with llama.cpp. It's hard to overstate the massive jump in capability between llama 1 and 2.
  - rimeice 3y ago
    Are you hosting that somewhere? If so, how much does that cost and do you have concurrent users?
    - ramesh31 3y ago
      >Are you hosting that somewhere? Tensordock. RTX4090 instances are ~$0.50/hr and can handle 3/4 concurrent users each.