The idea of making it easier to download modes from HuggingFace, and serve them via llama-cpp-python could be a good contribution, but unfortunately there are t

The idea of making it easier to download modes from HuggingFace, and serve them via llama-cpp-python could be a good contribution, but unfortunately there are two different names that the "Localllm" project introduces for itself - Localllm and the 'llm' command, both of them problematic for quite a few of reasons, perhaps they could just drop naming it anything to do with local? E.g: Simon Willison's well known `llm` package on PyPI already uses the `llm` command. A name for the command that differs from the project name creates extra work for no good reason. Look at this positioning Google use in the blog post: "localllm combined with Cloud Workstations revolutionizes AI-driven application development by letting you use LLMs locally on CPU and memory within the Google Cloud environment." - implying a local setup that's actually reliant on cloud resources is a contradiction. Maybe they could rename it with a single name that has no conflicts or misleading terms that makes it clear it's to help setup and run inference without confusing the matter about how local things might sometimes be. It won't be nice to read the scathing critiques in some of the other comments! Maybe they could fix up the mess in querylocal.py! Being able to run the same code locally as in a cloud platform is a nice development pattern but optimal inference setup for many ML situations is a tricky problem, are you really making good use of the CPU / GPU / TPU? The underlying server tech: llama-cpp-python recently added support for draft models that could make inference much faster. How easy would using that from Localllm be? I'd be concerned `Localllm > llama-cpp-python > llama.cpp` might be too many layers of systems to optimise or code with. Milliseconds often matter. Having a fast, well priced solution is not easy, people want as many tokens per second as they can get, ideally with no warm-up. How would they solve that? People looking at fast inference options might like to see Banana's parting words, as they wind down their serverless GPU cloud, in their swansong, they suggest a few hosted inference options people might want to consider: https://news.ycombinator.com/item?id=39288915 https://news.ycombinator.com/item?id=39288915