I have been working on my own local inference software: https://github.com/ryao/llama3.c/blob/master/run.c https://github.com/ryao/llama3.c/blob/master/run.c

I have been working on my own local inference software: https://github.com/ryao/llama3.c/blob/master/run.c https://github.com/ryao/llama3.c/blob/master/run.c First, CXL is useless as far as I am concerned. The smaller LLM stuff in 1 and 2 is overrated. LLMs get plenty of things wrong and while the capabilities of small LLMs is increasing, they just are never as good as the larger LLMs in my testing. To give an example, between a small LLM that gets things right 20% of the time and a large one that gets things right 40% of the time, you are never going to want to deal the small one. Even when they improvement you will just find new things that they are not able to do well. At least, that is my experience. Finally, the 75% savings figure in 3 is misleading. It applies to the context, not the LLMs themselves. It is very likely that nobody will use it, since it is a form of lossy compression that will ruin the ability of the LLM to repeat what is in its memory.