3 ms·
If you’re running just one GPU your context is limited to 1024 tokens, as far as I could tell. I couldn’t see what the context size is for more cards though.
by hmottestad 2y ago
If you’re running just one GPU your context is limited to 1024 tokens, as far as I could tell. I couldn’t see what the context size is for more cards though.