2 ms·

Well, if CPUs are equipped with extra instructions like Intel AMX then why not? https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions https://en.wikipedia.or

• brucethemoose2 3y ago
  Only newer Intel server CPUs have this, and even then its a really odd instruction to "activate" and use. Even without AMX, llama.cpp is already fairly bandwidth bound for short responses, and the cost/response on Sapphire Rapids is not great. I bet performance is much better on Xeon Max (Sapphire Rapids with HBM), but those SKUs are very expensive and rare.
  • HDThoreaun 3y ago
    Th author is suggesting we will start to see specially built servers that are optimized for AI inference. There's no reason these can't use special CPUs that utilize odd instructions. If inference does turn out to be optimized differently from training I think it's unlikely that we won't see a whole ecosystem surrounding it with specially built "inference" cpus and the like.
    • brucethemoose2 3y ago
      > Th author is suggesting we will start to see specially built servers that are optimized for AI inference. So far, cool genAI projects are rarely ported to anything outside of Nvidia or ROCM. Hence I am skeptical of this accelerator ecosystem. There is a good chance AWS, Microsoft, Google and such invest heavily in their own inference chips for internal use, but these will all be limited to those respective ecosystems.