3 ms·
DeepSeek-v3 Technical Report [pdf]
- deleted 2y ago[deleted]
- pongogogo 2y agoThe big news here is the training costs, $5.576m total cost, equivalent to 2788k training hours on H800 GPU at $2 per hour. This for a model that is (according to DeepSeek's own benchmarks) SOTA for open source.
- boroboro4 2y agoCrazy amount of innovations in one technical report: - successful fp8 quantized training for SOTA model - multi token prediction, mostly to improve training results, but also to enable speculative decoding - very high sparsity per request (37B activated params per 671B total params) - using reasoning data (from DeepSeek R1) to fine-tune and improve results on math & coding - manual balancing of compute / communication in their infrastructure, up to SM level