4 ms·
PostgresML Adds GPTQ and GGML Quantized LLM Support for HuggingFace Transformers
- montanalow 3y agoQuantization allows PostgresML to fit larger models in less RAM. These algorithms perform inference significantly faster on NVIDIA, Apple and Intel hardware. Half-precision floating point and quantized optimizations are now available for your favorite LLMs downloaded from Huggingface.