4 ms·
I don't quite understand, what would 100K buy you? AFAIK you would get about ~5 concurrent users, with a max context window of ~128K tokens on the larger model
by stopachka 4mo ago
I don't quite understand, what would 100K buy you?
AFAIK you would get about ~5 concurrent users, with a max context window of ~128K tokens on the larger models.
This wouldn't be good enough for coding -- are you guys thinking of using it for something else?
- bendews 4mo agoBy my calculations 100k could get you 18 5090's + compute to host them, or 18 96gb Mac mini's. You can get a lot of context window and users out of that setup.
- FuriouslyAdrift 4mo agoGigabyte 4x AMD Instinct MI300A rack server (512GB GPU RAM total) Roughly equivalent to 4x H200's for less than half the price. Vaguely around 60k tokens per second...