4 ms·

How are you getting this to run fast? I'm on a top of the line M1 MBP and getting 1 token every 8 minutes.

• markasoftware 4y ago
  probably pytorch is very optimized to x86. It's likely using lots of SIMD and whatnot. I'm sure it's possible to get similar performance on m1 macs, but not with the current version of pytorch. Do you have enough ram? (not swapping to disk)?
• ingenieroariel 4y ago
  Try switching all the .cuda() to .mps() I got a 100x speedup on a different language model on a Macbook M1 Air. https://pytorch.org/docs/stable/notes/mps.html https://pytorch.org/docs/stable/notes/mps.html
  • singularity2001 4y ago
    dedicated fork: https://github.com/remixer-dec/llama-mps https://github.com/remixer-dec/llama-mps
• jwitthuhn 4y ago
  Same experience for me, looks like it is only using one cpu core instead of all of them.