2 ms·

Local is the way. Any benchmarks on latency it has on CPU?

• unusual_typo 4mo ago
  I just ran the benchmark on my macbook. 582 ms for 1k tokens and 4.64 s for 8k