4 ms·
As LLMs have quite minor changes between architectures, would it make sense to just embed the model compiled to some sort of simple bytecode right in the GGUF f
by RicoElectrico 3y ago
As LLMs have quite minor changes between architectures, would it make sense to just embed the model compiled to some sort of simple bytecode right in the GGUF file? Then, only implement specific new operations when researchers come up with a new model that gains enough traction to be of interest.
- sroussey 3y agoYeah, but you want to avoid remote code execution: https://www.bleepingcomputer.com/news/security/malicious-ai-models-on-hugging-face-backdoor-users-machines/ https://www.bleepingcomputer.com/news/security/malicious-ai-...
- RicoElectrico 3y agoThe bytecode would not even need to be Turing-complete. Or maybe it could take inspiration from eBPF which gives some guarantees. What you posted is related to the design oversight of Python's pickle format.
- sroussey 3y agoI think ONNX does what you say.
- liuliu 3y agoNot really. We've been on that road before. Embedding computation graph into the file makes changes to the computation graph harder (you need to make sure it is backward compatible). This is OK in general (as we have onnx already), but then if you have dynamic shape and the fact that different optimizations we implemented are actually tied to the computation graph, this is simply not optimal. (BTW, this is why PyTorch just embed the code into the pth file, much easier and backward compatible than a static computation graph).
- taminka 3y agowait, why is embedding the graph into the file bad? it enables really clean separation of the core autodiff library and whatever backend you want to use to accelerate the graph computations, which can simply read the file and be completely independent of the core implementation but also, if you just store the tensors in some arbitrary order and then store the indices of the order in which they have to read and traversed, you can easily adjust the graph to add stuff like layer fusion or smth similar (i'm not really familiar w/ comp graph optimisations tbh) what would an alternative look like anyway?
- rahimnathwani 3y agoIt seems like a lot of innovation is around training, no? GGML (the library that reads GGUF format) supports these values for the required 'general.architecture': llama mpt gptneox gptj gpt2 bloom falcon rwkv
- throwawaybbq1 3y agoI've also been trying to figure out GGUF and the other model formats going around. I'm horrified to see there is no model architecture details in the file! As you say, it seems they are hard-coding the above architectures as constants. If a new hot model comes out, one would need to update the reader code (which has the new model arch implemented). Am I understanding this right? I'm also a bit confused by the quantization aspect. This is a pretty complex topic. GGML seems to use 16bit as per the article. If was pushing it to 8bit, I reckin I'd see no size improvement the GGML file? The article says they encode quantization versions in that file. Where are they defined?
- rahimnathwani 3y agoWhy are you horrified? In designing software, there's often a trade off between (i) generality / configurability, and (ii) performance. llama.cpp is built for inference, not for training or model architecture research. It seems reasonable to optimize for performance, which is what ~100% of llama.cpp users care about.
- throwawaybbq1 3y agoGGUF files seems to be proliferating. I think some folks (like myself) make an incorrect assumption that the format has more portability/generalizability than it appears to have. Hence, the horror!