https://github.com/ggerganov/ggml https://github.com/ggerganov/ggml TinyGrad is also targeting CPU inference, and IIRC it works ok in Apache TVM. One note is

https://github.com/ggerganov/ggml https://github.com/ggerganov/ggml TinyGrad is also targeting CPU inference, and IIRC it works ok in Apache TVM. One note is that prompt ingestion is extremely slow on CPU compared to GPU. So short prompts are fine (and tokens can be streamed once the prompt is ingested), but long prompts feel extremely sluggish. Another is that CPUs with more than 128-bit DDR5 memory busses are very expensive, and CPU token generation is basically RAM bandwidth bound.