3 ms·
Their 7b model performs much better than the 13b model I have running locally with alpaca.cpp. I think this is because (looking at their code) they stick with t
by UncleOxidant 4y ago
Their 7b model performs much better than the 13b model I have running locally with alpaca.cpp. I think this is because (looking at their code) they stick with the float16 parameters while alpaca.cpp is using the quantizized 4bit parameters. The difference is quite noticeable.
- int_19h 4y agoWhy do you believe the difference is due to quantization rather than all the fine-tuning that they did? Given the difference between GPT-3 and GPT-3.5 - pretty much all of which is fine-tuning and RLHF - I find it much easier to believe in the latter.
- starik36 4y ago> float16 parameters vs 4bit parameters What does that mean?
- arbol 4y agoIt's referring to the precision with which the weights are stored in the models. Models are basically giant matrices of decimals in the range 0-1 and they represent the strengths of the paths between the neurons. Greater precision (float 16) will give better predictions. You lose accuracy in predictions when you reduce the precision (float 4) but it makes the model much easier to run on lower spec hardware, as it requires less RAM to store lower precision floats. https://en.m.wikipedia.org/wiki/Half-precision_floating-point_format https://en.m.wikipedia.org/wiki/Half-precision_floating-poin...