3 ms·
Would it speed up prompt processing if you increased the -ub (and -b) parameters.
by aktenlage 14d ago
Would it speed up prompt processing if you increased the -ub (and -b) parameters.
- wombat23 14d agoI don't see any significant speed up. also with defaults according to --help -b, --batch-size N logical maximum batch size (default: 2048) -ub, --ubatch-size N physical maximum batch size (default: 512) I also tried double the default. that also means that they can be left to default settings, apparently.