4 ms·

This is so awesome and exciting. I have an M1 iMac and it was trivially easy to get this working and generating text. And the performance is VERY impressive, es

This is so awesome and exciting. I have an M1 iMac and it was trivially easy to get this working and generating text. And the performance is VERY impressive, especially considering that it's not even using any of the built in "neural compute" stuff. Also, the model seems like it doesn't have any political correctness conditioning based on some of the completions it has given me on controversial prompts. I can't wait until someone gets the 13b model working (sounds like this should happen in the next day or so) and gets the repetition penalty working.

lynguist 4y ago
It is using the built-in neural accelerators, that’s why it’s fast, that’s why it’s only supported on Macs so far. The code makes use of official Apple APIs which delegate the necessary BLAS calls to the available hardware.
- kamranjon 4y ago
  Confusingly there are 2 mechanisms to do matrix operations on the new apple hardware - AMX (https://github.com/corsix/amx https://github.com/corsix/amx) - and the ANE (apple neural engine) - which is enabled by CoreML. This code does not run on the neural engine but the author has a branch for his whisper.cpp project which uses it here: https://github.com/ggerganov/whisper.cpp/pull/566 https://github.com/ggerganov/whisper.cpp/pull/566 - so it may not be long before we see it applied here as well. All of this is to say that it actually could get significantly faster if some of this work was able to be handed to the ANE with CoreML.
  - lynguist 4y ago
    You’re right! I wrote it too fast without thinking!
  - raphlinus 4y ago
    Three. You can also do it in Metal, which as of recently has cooperative matrix multiplication in the form of the simd_matrix type (this is similar functionality as "tensor cores" in the Nvidia world). I have no idea what the software support is, but I have seen analysis suggesting that the raw tensor multiplication throughput is larger than ANE for the high-end GPUs.
sillysaurusx 4y ago
Repetition penalty is a matter of, generate a token, then multiply that logit by the penalty. (If the logit is negative, divide instead of multiply.) https://github.com/shawwn/llama https://github.com/shawwn/llama has an implementation (check the commit history).

lynguist 4y ago

It is using the built-in neural accelerators, that’s why it’s fast, that’s why it’s only supported on Macs so far. The code makes use of official Apple APIs which delegate the necessary BLAS calls to the available hardware.

kamranjon 4y ago
Confusingly there are 2 mechanisms to do matrix operations on the new apple hardware - AMX (https://github.com/corsix/amx https://github.com/corsix/amx) - and the ANE (apple neural engine) - which is enabled by CoreML. This code does not run on the neural engine but the author has a branch for his whisper.cpp project which uses it here: https://github.com/ggerganov/whisper.cpp/pull/566 https://github.com/ggerganov/whisper.cpp/pull/566 - so it may not be long before we see it applied here as well. All of this is to say that it actually could get significantly faster if some of this work was able to be handed to the ANE with CoreML.
- lynguist 4y ago
  You’re right! I wrote it too fast without thinking!
- raphlinus 4y ago
  Three. You can also do it in Metal, which as of recently has cooperative matrix multiplication in the form of the simd_matrix type (this is similar functionality as "tensor cores" in the Nvidia world). I have no idea what the software support is, but I have seen analysis suggesting that the raw tensor multiplication throughput is larger than ANE for the high-end GPUs.

sillysaurusx 4y ago

Repetition penalty is a matter of, generate a token, then multiply that logit by the penalty. (If the logit is negative, divide instead of multiply.) https://github.com/shawwn/llama https://github.com/shawwn/llama has an implementation (check the commit history).