4 ms·

You wouldn’t be able to fit the whole model into 8GB VRAM. It’s faster than not using a GPU at all, but most of it would still be computed on the CPU.

• baq 3y ago
  IME ollama ran mixtral on a 1070 fast enough.
  • dimask 3y ago
    Though it most probably does not run in on the 1070 but rather on the cpu. It cannot fit on a 1070, it is not about speed, a 1070 cannot run it period.
    • Dkuku 3y ago
      In llama.cpp You can offload some of the layers to gpu with -ngl X. Where x is the number of layers