3 ms·
Is not ego-driven, the requirements for running any model dropped by more than half, you can know run the largest model, 30b, on domestic over the shelf compute
by augusto-moura 4y ago
Is not ego-driven, the requirements for running any model dropped by more than half, you can know run the largest model, 30b, on domestic over the shelf computers. The change is very welcome in the community
- tempaccount420 4y agoThere were no improvements to memory use, the earlier GBs were a measurement error. If you couldn't run a model before (or were swapping, so running very slow), then you still have the same problem. You will actually "swap" a lot more than before if you have barely enough memory, but this is fixable with the --mlock flag. Edit: For everyone downvoting, please tell me what is wrong with my comment. I don't have a bone in this fight.
- xiphias2 4y ago4chan came here to chime in a bit. You were downvoted by 10 year old children, good luck explaining to them how memory locking works :)