4 ms·

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

by qeternity 3y ago

stri8ed 3y ago
How does the quantization happen? Are the weights preprocessed before loading the model?
- sebzim4500 3y ago
  Yes, but to my knowledge it doesn't do any of the complicated optimization stuff that SOTA quantisation methods use. It basically is just doing a bunch of rounding. There are advantages to simplicity, after all.
  - brucethemoose2 3y ago
    Its not so simple anymore, see https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684
- ggerganov 3y ago
  The weights are preprocessed into integer quants combined with scaling factors in various configurations (4, 5, 8-bits and recently more exotic 2, 3 and 6-bit quants). At runtime, we use efficient SIMD implementations to perform the matrix multiplication at integer level, carefully optimizing for both compute and memory bandwidth. Similar strategies are applied when running GPU inference - using custom kernels for fast Matrix x Vector multiplications
ggerganov 3y ago
ggml started with focus on CPU inference, but lately we have been augmenting it with GPU support. Although still in development, it already has partial CUDA, OpenCL and Metal backend support
- qeternity 3y ago
  Hi Georgi - thanks for all the work, have been following and using since the availability of Llama base layers! Wasn’t implying it’s CPU only, just that it started as a CPU optimized library.
- freedomben 3y ago
  As a person burned by nvidia, I can't thank you enough for the OpenCL support
- ignoramous 3y ago
  (a novice here who knows a couple of fancy terms) > ...lately we have been augmenting it with GPU support. Would you say you'd then be building an equivalent to Google's JAX? Someone even asked if anyone would build a C++ to JAX transpiler [0]... I am wondering if that's something you may implement? Thanks. [0] https://news.ycombinator.com/item?id=35475675 https://news.ycombinator.com/item?id=35475675

No, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp

stri8ed 3y ago

How does the quantization happen? Are the weights preprocessed before loading the model?

sebzim4500 3y ago
Yes, but to my knowledge it doesn't do any of the complicated optimization stuff that SOTA quantisation methods use. It basically is just doing a bunch of rounding. There are advantages to simplicity, after all.
- brucethemoose2 3y ago
  Its not so simple anymore, see https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684
ggerganov 3y ago
The weights are preprocessed into integer quants combined with scaling factors in various configurations (4, 5, 8-bits and recently more exotic 2, 3 and 6-bit quants). At runtime, we use efficient SIMD implementations to perform the matrix multiplication at integer level, carefully optimizing for both compute and memory bandwidth. Similar strategies are applied when running GPU inference - using custom kernels for fast Matrix x Vector multiplications

ggerganov 3y ago

ggml started with focus on CPU inference, but lately we have been augmenting it with GPU support. Although still in development, it already has partial CUDA, OpenCL and Metal backend support

qeternity 3y ago
Hi Georgi - thanks for all the work, have been following and using since the availability of Llama base layers! Wasn’t implying it’s CPU only, just that it started as a CPU optimized library.
freedomben 3y ago
As a person burned by nvidia, I can't thank you enough for the OpenCL support
ignoramous 3y ago
(a novice here who knows a couple of fancy terms) > ...lately we have been augmenting it with GPU support. Would you say you'd then be building an equivalent to Google's JAX? Someone even asked if anyone would build a C++ to JAX transpiler [0]... I am wondering if that's something you may implement? Thanks. [0] https://news.ycombinator.com/item?id=35475675 https://news.ycombinator.com/item?id=35475675