3 ms·
3 billion parameters. Does that mean I will be able to run on a 8gb consumer GPU?
by tarruda 3y ago
3 billion parameters. Does that mean I will be able to run on a 8gb consumer GPU?
- generalizations 3y agoMeans that once it's incorporated into llama.cpp, you can run it on your laptop.
- amasad 3y agoHopefully on phones too
- RHab 3y agoNo, I could only get 2.7B to run on 8GB VRam unfortunatly.
- pera 3y agotheir pytorch_model.bin is 10.4GB
- tarruda 3y agoI just loaded this on my laptop's RTX 3070 GPU by following the instructions here: https://huggingface.co/replit/replit-code-v1-3b https://huggingface.co/replit/replit-code-v1-3b I don't know how I can test the model, but it seem loading worked. When I run `nvidia-smi` on another terminal, I see `5188MiB / 8192MiB` in the memory-usage column.
- swyx 3y agoyou can load it but you cant run inference? whats the issue?
- tarruda 3y agoNo issue, I'm simply unfamiliar with python machine learning APIs. I managed to run inference locally by installing the requirements and running app.py from the demo: https://huggingface.co/spaces/replit/replit-code-v1-3b-demo/tree/main https://huggingface.co/spaces/replit/replit-code-v1-3b-demo/... It is very fast on my RTX 3070, VRAM usage goes to ~= 6.3GB during inference.
- dontwearitout 3y agoProbably not out of the box but if some of the local deep learning wizards get a quantized version working well and optimize it a bit, definitely.