4 ms·
I thought this too, but it sounds like the performance of the GPU ultimately holds it back. Maybe it is the software, but I have yet to see a test on Mac silico
by _ea1k 3y ago
I thought this too, but it sounds like the performance of the GPU ultimately holds it back. Maybe it is the software, but I have yet to see a test on Mac silicon that performed well.
- schappim 3y agoIt is not too bad, 7B is doing 432 tokens per second on a Macbook[1]. [1] Video: https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/147_falcon/falcon-7b.mp4 https://huggingface.co/datasets/huggingface/documentation-im...