3 ms·
"From scratch" seems to be a matter of opinion. "Pure pytorch" maybe, except it uses HF transformers. So it's LoRA on top of common frameworks...
by andy99 3y ago
"From scratch" seems to be a matter of opinion. "Pure pytorch" maybe, except it uses HF transformers. So it's LoRA on top of common frameworks...
- rasbt 3y agoYeah, the LoRA part is from scratch. The LLM backbone in this example is not, this is to provide a concrete example. But you could apply the exact same LoRA from scratch code to a pure PyTorch model if you wanted to: E.g. class MultilayerPerceptron(nn.Module): def __init__(self, num_features, num_hidden_1, num_hidden_2, num_classes): super().__init__() self.layers = nn.Sequential( nn.Linear(num_features, num_hidden_1), nn.ReLU(), nn.Linear(num_hidden_1, num_hidden_2), nn.ReLU(), nn.Linear(num_hidden_2, num_classes) ) def forward(self, x): x = self.layers(x) return x model = MultilayerPerceptron( num_features=num_features, num_hidden_1=num_hidden_1, num_hidden_2=num_hidden_2, num_classes=num_classes ) model.layers[0] = LinearWithLoRA(model.layers[0], rank=4, alpha=1) model.layers[2] = LinearWithLoRA(model.layers[2], rank=4, alpha=1) model.layers[4] = LinearWithLoRA(model.layers[4], rank=4, alpha=1)
- 2024throwaway 3y agoThis apple pie recipe claims to be from scratch, but they cooked it in an off the shelf oven. So it's from scratch on top of the universe...
- deleted 3y ago[deleted]
- _bifc 3y agoIf anyone is interested in a more 'pure' or 'scratch' implementation, check out https://github.com/michaelnny/QLoRA-LLM https://github.com/michaelnny/QLoRA-LLM. (author here) It also supports 4-bit quantized LoRA, using only PyTorch and bitsandbytes, without any other tools.