4 ms·
I think we're using different meaning of "behaves like". I meant "has tokens/sec performance comparable to".
by yencabulator 7mo ago
I think we're using different meaning of "behaves like". I meant "has tokens/sec performance comparable to".
- BoorishBears 7mo agoI'm using model performance because inference is definitely not comparable to a 17B model when you're streaming model weights on and off disk storage.