3 ms·
but why would you do C++ when its quite clear ML load is highly parallel. the page says vectorized by NEON but no mention whether its autovectorized by gcc or h
by DesiLurker 4y ago
but why would you do C++ when its quite clear ML load is highly parallel. the page says vectorized by NEON but no mention whether its autovectorized by gcc or hand optimized. That will have a pretty significant performance impact.
- tempaccount420 4y agoYou can read the code. It's open source.
- eigenvalue 4y agoSeems like it's extremely high performance for me on an M1 running the 7B model. Totally usable.
- DesiLurker 4y agojust because something is high performance does not means it cannot be improved by say another 2X. My point was not whether its usable or not, it is that if you are going to run on CPU vectorization is rather important part and its odd that the landing page has no remark about it.