2 ms·

Could you try with ./target/release/chat --model llama3.2-1b-it-q80.lmrs --show-metrics To know how many token/s you get?

• simonw 2y ago
  Nice, just tried that with "tell me a long tall tale" as the prompt and got: Speed: 26.41 tok/s Full output: https://gist.github.com/simonw/6f25fca5c664b84fdd4b72b0918546e0 https://gist.github.com/simonw/6f25fca5c664b84fdd4b72b091854...
  • jodleif 2y ago
    How much with llama.cpp? A 1b model should be a lot faster on a m2
    • hobofan 2y ago
      Given the fact that this at the core relies on the `rayon` and `wide` libraries, which are decently baseline optimized but quite a bit away from what llama.cpp can do when being specialized on such a specific use-case, I think the speed is about what I would expect. So yeah, I think there is a lot of room for optimization, and the only reason one would use this today is if they want to have a "simple" implementation that doesn't have any C/C++ dependencies for build tooling reasons.
      • littlestymaar 2y ago
        Your point is valid when it comes to rayon (I don't know much about wide) being inherently slower than custom optimization, but from what I've seen I suspect rayon isn't even the bottleneck in terms of performance, there's some decent margin of improvement (I'd expect at least double the throughput) without even doing arcane stuff.