Technically, this library is for loading a checkpoint one layer at a time and would work for any model you can load from a state_dict. To support the inference

Technically, this library is for loading a checkpoint one layer at a time and would work for any model you can load from a state_dict. To support the inference for the model of your choice, someone should write code that uses zipslicer to instantiate layers of the model while threading activations through active layers - and it will make economic sense if you use this code with large batch size in long-term computation. There is no magic trick yet to compute interactive sampling (think ChatGPT) without fitting the whole model in, at least, RAM+VRAM, better in VRAM. You could go slightly different route and use this non-interactive mode just one time to compress the weights down to int8, and this can, in turn, make the model runnable on your hardware in interactive mode without layer streaming from disk. Say, your model is 12B params and you have 16G RAM - it can work perfectly well if each weight takes 1 byte, but the weights are distributed in bf16/fp16 and your PC doesn't even support these so it instantiates the weights in fp32 - and you are out of RAM. Zipslicer can help here by allowing you to execute weight compression. While this "offline execution engine" or "parameter compression engine" is slightly beyond the scope of this library, I can write it as an example or as a separate library, and describe this in a blogpost. For the purpose of making an example I would prefer to work with permissively licensed models such as this one https://huggingface.co/Salesforce/codegen-6B-nl https://huggingface.co/Salesforce/codegen-6B-nl - but someone could easily fork the repo and adapt the code later to whatever model they need. It would be nice if people proposed desirable features in project's Github issues and I could see which features get votes.