3 ms·
> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. I don't understand. How does that make it unusable? Is the limit sha
by Aurornis 2mo ago
> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks.
I don't understand. How does that make it unusable? Is the limit shared by an entire team at once?
150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find.
I think the 128K context limit is the real ceiling. These models aren't amazing at long context, but once you account for a short input prompt, the input files, and headroom for a compaction summary, there isn't a lot left for the problem.
- conception 2mo ago150k by account. At 1.5k a second you hit it very quickly.
- devy 2mo agoExactly, it burns the tokens 3000x faster, which means the budget ($$$$$$) runs out so faster it will stop super quick, not able to perform long-duration work. At 27B parameter size, the intelligence is not able to accomplish work within a short amount time. Consequently, it become not usable.
- gerdesj 2mo agoI (we) run Qwen3.8-27B-FP8 on a DGX Spark box - that's roughly £4000 of hardware. I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve. To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.
- jacquesm 2mo agoHow fast is it?
- kristjansson 2mo agoWith MTP and FP4 I max out at 30ish t/s on mine. Without MTP or in regimes where the drafter performs poorly it’s about 10 t/s. FP8 is about half that
- jacquesm 2mo agoThank you, always nice to see real world performance figures. We run a pretty large rig, 10 GPUs right now (this goes up and down with various experiments, getting this many GPUs to play nice at x16 GEN4 with any motherboard is a challenge), 240G VRAM in total. 256G RAM and a TR PRO. For small models the comms overhead is larger than the gains so there I have to reduce the number of active GPUs. On this machine I'm getting between 150 and 200 tg/s with FP8, but it took a lot of time and tweaking to get to that, and not all of the improvements held up when combined with other improvements. I've been playing with this stuff for a while now and it is interesting how fast the frontier is moving and how much you can now do on your own hardware. For larger models the communications overhead is low enough that we can run them on bigger groups of GPUs, and using hacked drivers to give us p2p capabilities on some of our GPUs also boosts performance considerably once you start to hit communications limits. Typically we get 50G/second in p2p mode (full duplex, half that one way). From a cost perspective running locally is not interesting, but it allows us to do experiments that model providers would likely balk at, gives us censorship free access and allows us to work with data that we would not want to share with model providers (or can't share due to NDAs). I will look into running ninfer, I was aware of them but had not yet gotten around to using it.
- wincy 2mo agoParent already responded but just for reference an RTX 5090 with Ninfer hits 160 tokens/second with qwen 3.8 27B which is very usable.
- conception 2mo agoThe problem is most providers hit tok/sec limits really fast. 1m/min is the default and the only place I can get 10m+ is from first party providers without a lot of upfront cash.
- a012 2mo agoUnusable is too stretch IMO, you can still use it in tiny tasks that’ll respond almost instantly
- datadrivenangel 2mo ago150k tokens per minute at 1.5k tokens per second means you can have like 3 users concurrently and that's not a lot.
- wild_egg 2mo agoIt's a limit on input tokens. So that's 3 50k requests per minute. At Cerebras speeds, that's about 5 seconds of usage per minute. I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time. Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was so much waiting time.
- amelius 2mo agoCan't you do something with multiple accounts?
- sandworm101 2mo agoOr just buy a 5060. This will run on most any 16gb card. Slower for sure but far cheaper than another subscription.
- embedding-shape 2mo agoOr buy a raspberry pi with a SSD, about the same difference, if you're giving up on the 1500 tokens/s anyways.
- ma2kx 2mo agoThats not the point if you choose Cerebras as provider.
- interactivecode 2mo agoThe whole point is the speed
- nateb2022 2mo ago[dead]
- jychang 2mo agoYou would lose caching (if they cache)
- gerdesj 2mo ago128k context is not a limit of the model, that's a limit of implementation: "Context Length: 262,144 natively and extensible up to 1,000,000 tokens." https://huggingface.co/Qwen/Qwen3.8-27B https://huggingface.co/Qwen/Qwen3.8-27B
- selcuka 2mo agoTPM means Tokens per Minute.
- zxexz 2mo agoGP is referring to GGP’s last paragraph. 150k t/m, yes, and 128k context.
- Aurornis 2mo agoWe're talking about the Cerebras implementation, which is limited to 128K. It's in the link.
- gerdesj 2mo agoYes, and I said its a limit of the implementation and not the model.
- deleted 2mo ago[deleted]