3 ms·
I ran Llama 2 70B in my browser (Chrome Canary on a 64GB MacBook M2) using this. It took a long time to start running, but then: prefill: 0.9654 tokens/sec, de
by simonw 3y ago
I ran Llama 2 70B in my browser (Chrome Canary on a 64GB MacBook M2) using this. It took a long time to start running, but then:
prefill: 0.9654 tokens/sec, decoding: 3.2589 tokens/sec
Honestly amazed that this is even possible. I haven't even run Llama 2 70B on my laptop NOT using a web browser yet.
- muskmusk 3y agoThat is impressive. Interesting that the prefill (i am guessing this is prompt processing) is so much slower than decoding. Its my understanding that under normal circumstances decoding is memory bandwidth bound which prompt processing isn't due to batching. Is there some quirk in your setup?
- larrysalibra 3y agoStrange. I'm running Llama2 70b on Chrome Canary on a 64GB MacBook M1 Max...~1.5 older...and seeing better performance. It's slow but usable! prefill: 2.1963 tokens/sec, decoding: 3.4708 tokens/sec
- 3abiton 3y agoThis is a crazy good performance!
- HPsquared 3y agoHow's the quantization?
- TradingPlaces 3y agoMac Studio 2022 64 GB RAM M1 Max. Lots of other stuff running. prefill: 1.1620 tokens/sec, decoding: 2.4105 tokens/sec
- LegitShady 3y agoRTX 4080 Llama-2-77b-chat-hf-q4f32_1 210.6742 tokens/sec, decoding: 20.7758 tokens/sec Can't tell if its really taking advantage of the all the power of the card, though.