3 ms·
The plan seems to be for lots and lots of smaller satellites. For inferencing it can work well. One satellite could contain a handful of CPUs and do batch infe
by mike_hearn 9mo ago
The plan seems to be for lots and lots of smaller satellites.
For inferencing it can work well. One satellite could contain a handful of CPUs and do batch inferencing of even very large models, perhaps in the beginning at low speeds. Currently most AI workloads are interactive but I can't see that staying true for long, as things improve and they can be trusted to work independently for longer it makes more sense to just queue stuff up and not worry about exactly how high your TTFT is.
For training I don't see it today. In future maybe. But then, most AI workloads in future should be inferencing not training anyway.
- KoolKat23 9mo agoLatency means this still makes no sense to me. Perhaps some batch background processing job such as research or something but that's stretching.
- mike_hearn 9mo agoI think the most providers all give high latency batch APIs significant discounts. A lot of AI workloads feel batch-oriented to me, or could be once they move beyond the prototype and testing phases. Chat will end up being a small fraction of load in the long term.
- KoolKat23 9mo agoThat would imply there's still capacity here on earth for this type of traffic.