4 ms·
BigDL-LLM: running LLM on your laptop using INT4
- colorant 3y agoBigDL-LLM is a library for running LLM (language language model) on your local laptop using INT4 with very low latency on CPU. (It is built on top of the excellent work of llama.cpp, gptq, bitsandbytes, etc., and supports any Hugging Face Transformers model).