4 ms·
I don't have the hardware to run the 60B model to test this at the moment - How does it perform with programming, for example making a basic python script to s
by thewataccount 4y ago
I don't have the hardware to run the 60B model to test this at the moment -
How does it perform with programming, for example making a basic python script to scrape a website, or a bash script, etc?
I've managed to run the 13B* at 8bit with decent performance on a 4090 - but it's only 24GB of VMRAM so I've been struggling to run the 30B at anything more then a snails pace.
- arthurcolle 4y agoyou mean the 13B ?
- thewataccount 4y agoYeah my bad, everyone is a bit all over the place with the numbers in this thread. I'm not exactly sure how these numbers were chosen, they seem a bit odd?
- 0xbadc0de5 4y agoThe 13b and 30b run quite well on a 4090 at 4-bit quantization.
- thewataccount 4y agoAh dang I missed that I was still using the 8bit mode, I'll look into that thanks!