3 ms·
We're running Kimi 2.8 on a $107k server and getting around 50k tokens per second or better on most things. We've already saved money compared to last years to
by FuriouslyAdrift 3mo ago
We're running Kimi 2.8 on a $107k server and getting around 50k tokens per second or better on most things.
We've already saved money compared to last years token cost on Claude/Gemini
- mdp2021 3mo ago> We're running Kimi 2.8 on a $107k server Equipped with what? Is it CPU based inference, a mix...?
- wongarsu 3mo agoNot GP, but my educated guess is that they are running a system with between 4 and 6 MI325 or MI355x or similar AMD GPUs. With the 50k tps as the total figure for all parallel requests. Those cards have a lot of memory for their price, allowing you to push to really high batch sizes while still having a large context size for each request
- FuriouslyAdrift 3mo ago4x MI300A in a Gigabyte server
- logicallee 3mo agoAre you developing software? Is most of it used on a coding agent? (Like Claude Code or ChatGPT Codex?) If so, what coding agent do you use? If you're not developing software what do you use it for (roughly)?
- FuriouslyAdrift 3mo agoOur use cases are data analysis, software development, general AI chat with RAG so far.
- freediddy 3mo agohow many requests per second can the server take?
- FuriouslyAdrift 3mo agoThat's about it across the whole server. Kimi 2.8 is about as big as we can go. Kimi 3 is massive.
- ajju 3mo agoWhat is your config?
- retinaros 3mo agoon 1 paralel request maybe
- FuriouslyAdrift 3mo agoCorrect but even with 30 simultaneous users, we're not seeing any noticable lag compared to AIaaS even with agentic workloads.