2 ms·

These new models get released in such an extreme cadence now it's almost impossible to stay ahead of the curve in production deployments.. Absolutely can't wai

by 19h 3y ago

These new models get released in such an extreme cadence now it's almost impossible to stay ahead of the curve in production deployments.. Absolutely can't wait to test drive this one -- although I'm pretty sure my 96GB M2 MacBook is unable to run it.. time for M2 Ultra? :-)) Edit: > You will need at least 400GB of memory to swiftly run inference with Falcon-180B. What the ...

ronsor 3y ago
I think this is before quantization (fp16). If quantized to 3 or 4 bit precision, you should be able to run it.
api 3y ago
You should be able to run it 4-bit quantized... barely.
bee_rider 3y ago
It is inference, so maybe it is just CPU memory?