3 ms·
I honestly think have a way to just use json (a.k.a. safetensors) / msgpack or some lightweight metadata serializer is a better route than coming up with a new
by liuliu 3y ago
I honestly think have a way to just use json (a.k.a. safetensors) / msgpack or some lightweight metadata serializer is a better route than coming up with a new file format. That's also why I just use SQLite to serialize the metadata (and tensor weights, this part is an oversight).
- andy99 3y agoGguf is cleaner to read in languages that don't have a json parsing library, and works with memory mapping in C. It's very appealing for minimal inference frameworks vs other options.
- liuliu 3y agosafetensors can mmap too because the tensor data are just offsets and you are free to align to whatever you want. It is hard to keep metadata minimal, and before long, you will start to have many different "atom"s and end-up with things that mov supports but mp4 doesn't etc etc. (mov format is generally well-defined and easy-to-parse, but being a binary format, you have to write your parser etc is not a pleasant experience). If you just want minimal dependency, flatbuffers, capnproto, json are all well-supported on many platforms.
- jart 3y agommap() requires that you map at page aligned intervals which must be congruent with the file offset. You can't just round down because some gpus like metal require that the data pointers themselves be page aligned too.
- liuliu 3y agoYeah, safetensors separates metadata and tensor data. The metadata is an offset reference to the tensor data that you are free to define yourselves. In that way, you can create files in safetensors format but the tensor data itself is paged aligned offsets.
- 3abiton 3y agoBut usually AWQ get recommended for GPU inference over GGUF
- Mathnerd314 3y agoBy who? Only comparison I have seen is that it sucks vs. EXL2 https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp/ https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacp...
- 3abiton 3y agoThe issue is benchmarks for LLMs or models formats are tough to compare, as there are many factors at play. But beyond ooba's comparison, many other sources recommend GPTQ or AWQ for GPU inference as it gives better quality for the same quant level (AWQ apparently takes more VRAM though, but better quality). Given how many models are available I would take these tests with a grain of salts.
- Eisenstein 3y agoHaven't heard this. Was this a few months ago? A lot happens in this space over that time span.
- fzzzy 3y agoI think a binary format is obviously the right answer here.