3 ms·
They are working with tiny models. Not sure how well it'd scale to bigger models (if at all).
by checker659 1y ago
They are working with tiny models. Not sure how well it'd scale to bigger models (if at all).
- CaptainOfCoit 1y agoThey're all LLMs, so no, not tiny, but not exactly huge either: > Our current deployment runs in a cross-region cluster comprising 213 H20 GPUs, serving twenty-eight 1.8–7B models (TP=1) and nineteen 32–72B models (TP=4).