4 ms·
Surprised to see 172 items in training set. Is that sufficient scale for QLoRA? I had kinda assumed one needed 1000s
by Havoc 3y ago
Surprised to see 172 items in training set. Is that sufficient scale for QLoRA? I had kinda assumed one needed 1000s
- mattew 3y agoThat blew my mind as well. I would have thought a much larger dataset would have been required.
- antupis 3y agoIf they are very good examples then less is more https://arxiv.org/abs/2305.11206 https://arxiv.org/abs/2305.11206
- swyx 3y agoarguably it takes more effort/expertise to curate those examples than just pull in a bunch of data that roughly loosely covers the intended surface area. bitter lesson redux?
- sp332 3y agoMost QLoRAs only target attention layers. This gives pretty good performance for minimal processing time. But this one adds adapters to all linear layers. Also I'm not sure what a "step" is but they ran the whole dataset through at least twice.