2 ms·

In addition Deepseek just showed that they are serving all that demand (admittedly inadequately) from a cluster of only 2000 H800 GPUs.

yorwba 2y ago
Are you sure? They stated that 2048 H800s participated in the V3 training run, but it's unlikely they didn't have additional clusters in use for other experiments at the same time, and they don't absolutely have to use H800s for inference, and they don't all have to be colocated in a single cluster like for distributed training, so it would make sense for them to rent additional inference capacity piecemeal on the open market as long as the unit economics work out. So barring an explicit statement from DeepSeek to that effect, it sounds like the kind of misunderstanding that would result from a game of telephone.
- fspeech 2y ago
  https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md https://github.com/deepseek-ai/open-infra-index/blob/main/20... Statistics of DeepSeek's Online Service All DeepSeek-V3/R1 inference services are served on H800 GPUs with precision consistent with training. Specifically, matrix multiplications and dispatch transmissions adopt the FP8 format aligned with training, while core MLA computations and combine transmissions use the BF16 format, ensuring optimal service performance. Additionally, due to high service load during the day and low load at night, we implemented a mechanism to deploy inference services across all nodes during peak daytime hours. During low-load nighttime periods, we reduce inference nodes and allocate resources to research and training. Over the past 24 hours (UTC+8 02/27/2025 12:00 PM to 02/28/2025 12:00 PM), the combined peak node occupancy for V3 and R1 inference services reached 278, with an average occupancy of 226.75 nodes (each node contains 8 H800 GPUs). Assuming the leasing cost of one H800 GPU is $2 per hour, the total daily cost amounts to $87,072.

yorwba 2y ago

Are you sure? They stated that 2048 H800s participated in the V3 training run, but it's unlikely they didn't have additional clusters in use for other experiments at the same time, and they don't absolutely have to use H800s for inference, and they don't all have to be colocated in a single cluster like for distributed training, so it would make sense for them to rent additional inference capacity piecemeal on the open market as long as the unit economics work out. So barring an explicit statement from DeepSeek to that effect, it sounds like the kind of misunderstanding that would result from a game of telephone.

fspeech 2y ago
https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md https://github.com/deepseek-ai/open-infra-index/blob/main/20... Statistics of DeepSeek's Online Service All DeepSeek-V3/R1 inference services are served on H800 GPUs with precision consistent with training. Specifically, matrix multiplications and dispatch transmissions adopt the FP8 format aligned with training, while core MLA computations and combine transmissions use the BF16 format, ensuring optimal service performance. Additionally, due to high service load during the day and low load at night, we implemented a mechanism to deploy inference services across all nodes during peak daytime hours. During low-load nighttime periods, we reduce inference nodes and allocate resources to research and training. Over the past 24 hours (UTC+8 02/27/2025 12:00 PM to 02/28/2025 12:00 PM), the combined peak node occupancy for V3 and R1 inference services reached 278, with an average occupancy of 226.75 nodes (each node contains 8 H800 GPUs). Assuming the leasing cost of one H800 GPU is $2 per hour, the total daily cost amounts to $87,072.