3 ms·

Th author is suggesting we will start to see specially built servers that are optimized for AI inference. There's no reason these can't use special CPUs that ut

Th author is suggesting we will start to see specially built servers that are optimized for AI inference. There's no reason these can't use special CPUs that utilize odd instructions. If inference does turn out to be optimized differently from training I think it's unlikely that we won't see a whole ecosystem surrounding it with specially built "inference" cpus and the like.

brucethemoose2 3y ago
> Th author is suggesting we will start to see specially built servers that are optimized for AI inference. So far, cool genAI projects are rarely ported to anything outside of Nvidia or ROCM. Hence I am skeptical of this accelerator ecosystem. There is a good chance AWS, Microsoft, Google and such invest heavily in their own inference chips for internal use, but these will all be limited to those respective ecosystems.