3 ms·
The 7B model just barely fits in a 3090, but I've yet to get useful output; it usually gets stuck in a repetition loop. Spent some time playing with sampling p
by Filligree 4y ago
The 7B model just barely fits in a 3090, but I've yet to get useful output; it usually gets stuck in a repetition loop.
Spent some time playing with sampling parameters, to no effect. Has anyone else had a go?
- nlightcho 4y agoMake sure you are using the instruction tuned model. The base model will be difficult to prompt. It works in 8-bit with about 12GB of VRAM usage. Here's sample code: https://gist.github.com/AlexanderDzhoganov/a1d1ebdb018e2e573a54a5796700b4ff https://gist.github.com/AlexanderDzhoganov/a1d1ebdb018e2e573...
- cheald 4y agoIf you're getting stuck in a sampling loop, try raising the temperature (0.6+) and/or increasing the repetition penalty (1.1-1.2 often work well).