12 ms·
At first glance i would be surprised if this works at all. The readme says it loads a significant amount of data the first time, 250mb. None of the LLM weights
by xrd 3y ago
At first glance i would be surprised if this works at all.
The readme says it loads a significant amount of data the first time, 250mb. None of the LLM weights I know of are less than several gigabytes in size.
It says it only requires 512mb of RAM. None of the interesting LLMs I know of run in less than 6gb of VRAM.
It says it uses no API keys, which is great, but that means inference is local, which I can't imagine works with the above constraints.
- nattaylor 3y agoAt 512gb RAM, it uses https://huggingface.co/jncraton/LaMini-Flan-T5-248M-ct2-int8 https://huggingface.co/jncraton/LaMini-Flan-T5-248M-ct2-int8 which says "This model is one of our LaMini-LM model series in paper "LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions". This model is a fine-tuned version of google/flan-t5-base on LaMini-instruction dataset that contains 2.58M samples for instruction fine-tuning"
- Translationaut 3y agoThose minified models are still equal or bigger compared to the initial "attention is all you need" transformer.