3 ms·
These people have zero idea what they're doing. Not a single mention of pipeline parallelism that would actually make the setup useful to run a big model.
by jimmoores 1mo ago
These people have zero idea what they're doing. Not a single mention of pipeline parallelism that would actually make the setup useful to run a big model.
- kelmoran 1mo agoI feel like you would want to run 8 smaller models separately for quantity of raw output. 1 big model is slow and isnt guaranteed to make no mistakes.
- monster_truck 1mo agoThat's not quite how it works. Throwing Deepseek V4 Flash on 4 of these would net you something like >200tk/s for 16 concurrent requests, that's 600 million _output_ tokens a month. Guess what happens when you use 8
- teaearlgraycold 1mo agoDoesn't really mean anything without a specific use case to guide model selection. The thing is Qwen 3.8 27B can be ran on far far cheaper hardware. If you're spending the big bucks on these rigs you probably made the wrong choice if you aren't using models that require all of that VRAM.
- schaefer 1mo agocan you point to a write up that discusses what you're talking about? because I would read it.
- itkovian_ 1mo agoI can’t stand it. Very engineering-y over specified formal language around a complete lack of core understanding. Is damaging other people read this and try to learn things from it.