3 ms·
https://github.com/mit-han-lab/TinyChatEngine https://github.com/mit-han-lab/TinyChatEngine Turns out the original source is actually somewhat informative. Inc
by lrem 3y ago
https://github.com/mit-han-lab/TinyChatEngine https://github.com/mit-han-lab/TinyChatEngine
Turns out the original source is actually somewhat informative. Including telling you how much hardware do you need. This blog post looks like your typical note you leave for yourself to annotate a bit of your shell history.
- pmontra 3y agoI wish that all these repos were more clear about the hardware requirements. Seeing that it runs on a 8 GB Raspberry, probably with abysmal performance, I'd say that it will run on my 32 GB Intel laptop on the CPU. Will it run on its Nvidia card? I remember that the rule of thumb was one GB of GPU RAM per G parameters, so I'd say that it won't run. However this has 4 bit quantization so it could have lower requirements. Of course the main problem is that I don't know enough about the subject to reason on it on my own.
- dkjaudyeqooe 3y agoRoughly speaking I believe it's the number of parameters times the size of the parameters. So in the 4 bit case it's half a gigabyte per billion parameters. From a performance point of view (quantized) integer parameters are going to run better on CPUs than floating point parameters.
- physicsgraph 3y agoYour assessment is exactly correct -- the blog post is my note-to-self about getting the repo to work. My "added value" in the post is a Dockerfile for ease of installation.