3 ms·
They're all LLMs, so no, not tiny, but not exactly huge either: > Our current deployment runs in a cross-region cluster comprising 213 H20 GPUs, serving twenty
by CaptainOfCoit 1y ago
They're all LLMs, so no, not tiny, but not exactly huge either:
> Our current deployment runs in a cross-region cluster comprising 213 H20 GPUs, serving twenty-eight 1.8–7B models (TP=1) and nineteen 32–72B models (TP=4).