3 ms·

Llama.cpp I think has a ton of clone-and-own boilerplate, presumably from having grown so quickly (I think one of their .cu files is over 10k lines or so, rough

Llama.cpp I think has a ton of clone-and-own boilerplate, presumably from having grown so quickly (I think one of their .cu files is over 10k lines or so, roughly, ATM). While I haven't seen the model storage and distribution format, the rewrite to GGUF for file storage seems to have been a big boon/boost to the project. Thanks Phil! Cool stuff. Also, he's a really nice guy to boot. Please say hi from Fern to him if you ever run into him. I mean it literally, make his life a hellish barrage of nonstop greetings from Fern.

liuliu 3y ago
I honestly think have a way to just use json (a.k.a. safetensors) / msgpack or some lightweight metadata serializer is a better route than coming up with a new file format. That's also why I just use SQLite to serialize the metadata (and tensor weights, this part is an oversight).
- andy99 3y ago
  Gguf is cleaner to read in languages that don't have a json parsing library, and works with memory mapping in C. It's very appealing for minimal inference frameworks vs other options.
  - liuliu 3y ago
    safetensors can mmap too because the tensor data are just offsets and you are free to align to whatever you want. It is hard to keep metadata minimal, and before long, you will start to have many different "atom"s and end-up with things that mov supports but mp4 doesn't etc etc. (mov format is generally well-defined and easy-to-parse, but being a binary format, you have to write your parser etc is not a pleasant experience). If you just want minimal dependency, flatbuffers, capnproto, json are all well-supported on many platforms.
    - jart 3y ago
      mmap() requires that you map at page aligned intervals which must be congruent with the file offset. You can't just round down because some gpus like metal require that the data pointers themselves be page aligned too.
      - liuliu 3y ago
        Yeah, safetensors separates metadata and tensor data. The metadata is an offset reference to the tensor data that you are free to define yourselves. In that way, you can create files in safetensors format but the tensor data itself is paged aligned offsets.
  - 3abiton 3y ago
    But usually AWQ get recommended for GPU inference over GGUF
    - Mathnerd314 3y ago
      By who? Only comparison I have seen is that it sucks vs. EXL2 https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/ https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacp...
      - 3abiton 3y ago
        The issue is benchmarks for LLMs or models formats are tough to compare, as there are many factors at play. But beyond ooba's comparison, many other sources recommend GPTQ or AWQ for GPU inference as it gives better quality for the same quant level (AWQ apparently takes more VRAM though, but better quality). Given how many models are available I would take these tests with a grain of salts.
    - Eisenstein 3y ago
      Haven't heard this. Was this a few months ago? A lot happens in this space over that time span.
- fzzzy 3y ago
  I think a binary format is obviously the right answer here.
qrios 3y ago
Thank you for the reference to the CUDA file [1]. It's always nice to see how complex data structures are handled in GPUs. Does anyone have any idea what the bit patterns are for (starting at line 1529)? [1] https://github.com/ggerganov/llama.cpp/blob/master/ggml-cuda.cu https://github.com/ggerganov/llama.cpp/blob/master/ggml-cuda...
- thrtythreeforty 3y ago
  Those have to do with dequantization. It involves table lookups and some adjusting math.

liuliu 3y ago

I honestly think have a way to just use json (a.k.a. safetensors) / msgpack or some lightweight metadata serializer is a better route than coming up with a new file format. That's also why I just use SQLite to serialize the metadata (and tensor weights, this part is an oversight).

andy99 3y ago
Gguf is cleaner to read in languages that don't have a json parsing library, and works with memory mapping in C. It's very appealing for minimal inference frameworks vs other options.
- liuliu 3y ago
  safetensors can mmap too because the tensor data are just offsets and you are free to align to whatever you want. It is hard to keep metadata minimal, and before long, you will start to have many different "atom"s and end-up with things that mov supports but mp4 doesn't etc etc. (mov format is generally well-defined and easy-to-parse, but being a binary format, you have to write your parser etc is not a pleasant experience). If you just want minimal dependency, flatbuffers, capnproto, json are all well-supported on many platforms.
  - jart 3y ago
    mmap() requires that you map at page aligned intervals which must be congruent with the file offset. You can't just round down because some gpus like metal require that the data pointers themselves be page aligned too.
    - liuliu 3y ago
      Yeah, safetensors separates metadata and tensor data. The metadata is an offset reference to the tensor data that you are free to define yourselves. In that way, you can create files in safetensors format but the tensor data itself is paged aligned offsets.
- 3abiton 3y ago
  But usually AWQ get recommended for GPU inference over GGUF
  - Mathnerd314 3y ago
    By who? Only comparison I have seen is that it sucks vs. EXL2 https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/ https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacp...
    - 3abiton 3y ago
      The issue is benchmarks for LLMs or models formats are tough to compare, as there are many factors at play. But beyond ooba's comparison, many other sources recommend GPTQ or AWQ for GPU inference as it gives better quality for the same quant level (AWQ apparently takes more VRAM though, but better quality). Given how many models are available I would take these tests with a grain of salts.
  - Eisenstein 3y ago
    Haven't heard this. Was this a few months ago? A lot happens in this space over that time span.
fzzzy 3y ago
I think a binary format is obviously the right answer here.

qrios 3y ago

Thank you for the reference to the CUDA file [1]. It's always nice to see how complex data structures are handled in GPUs. Does anyone have any idea what the bit patterns are for (starting at line 1529)? [1] https://github.com/ggerganov/llama.cpp/blob/master/ggml-cuda.cu https://github.com/ggerganov/llama.cpp/blob/master/ggml-cuda...

thrtythreeforty 3y ago
Those have to do with dequantization. It involves table lookups and some adjusting math.