7 ms·
Localllm lets you develop gen AI apps on local CPUs
- brucethemoose2 3y agoEven Google is making a llama.cpp wrapper. Technically a wrapper of a wrapper, though llama-cpp-python is quite excellent... TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary).
- andy99 3y agoOh it's a wrapper? I just skimmed and thought they reinvented llama.cpp for some reason, which would have been mildly interesting.
- williamstein 3y ago(edit: this is completely irrelevant remark. sorry) It's (1) written in Python, and (2) ollama appears nowhere in their Docker repo [1], and (3) it gets the models only from huggingface instead of the ollama repos. Thus maybe it is NOT a wrapper of ollama? [1] https://github.com/googlecloudplatform/localllm https://github.com/googlecloudplatform/localllm
- andy99 3y agohttps://github.com/GoogleCloudPlatform/localllm/blob/main/llm-tool/setup.py#L30 https://github.com/GoogleCloudPlatform/localllm/blob/main/ll... I see llama.cpp in there, but I didn't go deeper than that. Ollama is not the same thing.
- thund 3y agodoes Functionary work with generic models like llama.cpp? had the impression you could only use Functionary GGUFs
- gbickford 3y agoLlama.cpp is an inference engine. The author of llama.cpp designed gguf. Funcionary is a model that does function calling. You can download functionary weights in the gguf format and then run it using llama.cpp on low-end machines using CPU or GPU or a mix of both.
- thund 3y agook so the answer is no, thanks
- behnamoh 3y agoThe GP educated you and you still said the answer was "no"? It's clearly "yes".
- TeMPOraL 3y agoI read the original question as "is the 'Functionary' thing something that works with any LLM you can run in llama.cpp, not just only with the special 'Functionary' finetuned models?", to which the answer now is clearly no. Very interesting question, as I also read top-level post as saying it's a llama-cpp-python feature, and therefore I could shove some random dolphin-mixtral or whatever I have on my hard drive at it, and it will work.
- brucethemoose2 3y agoThis question is confusing to me. Functionary is a series of function calling llama architecture finetuned models, not a new base model, if thats what you mean. Hence, it can be turned into a gguf like any other llama finetune. It just has a very esoteric prompting format, but llama-cpp-python is specifically set up to leverage it.
- thund 3y agofrom the parent: > TBH I would just recommend using that and all its under-the-radar features (like grammar, and built in function calling support with Functionary). I never heard about Functionary, this comment seems to say “I would use llama.cpp plus Functionary”. Now, since llama.cpp allows to plug in several models, I was wondering if Functionary works with several models too, ie is Functionary a library that allows to add function calling to models? The following answer to my comment helped clarifying that Functionary is not a lib, but a specific model, or that is what I understand from the answer. So the answer is “no, you can’t use functionary with many models (eg it’s not like llama.cpp that works with multiple models) because Functionary is a model itself” :-)
- discardedrefuse 3y agoSlightly off topic, here is the best local llama.cpp wrapper I've run into: https://github.com/Mozilla-Ocho/llamafile https://github.com/Mozilla-Ocho/llamafile You can download any .gguf model (not just the ones in their examples) and run it locally (as long as you have the ram for it). I was running 7B models with ease on an old FX8350 and now 13B models on a 5600X (32GB RAM on both machines). This wrapper spins up a local web server that runs a simple web frontend to use immediately with no code, but also exposes an OpenAI compatible API for dev work and alt frontends (like SillyTavern).
- 3abiton 3y agoWhat's the speed? I heard of llamafile before.
- discardedrefuse 3y agoI wasn't really timing it, but it felt fine, especially since it prints out while generating. The 13B model on the 5600X is about 10 seconds. Tho once the conversation gets too long (about 10 replies) it will go sideways. I gotta play with the default settings. IIRC the 7B on the Fx8350 was closer to 25 seconds per response.
- m00x 3y agoI wonder what need they're fulfilling here. It's not really more useful than using llamacpp-python. This is not something old Google would even release.
- bigiain 3y ago> I wonder what need they're fulfilling here. The need to fill more seats on the AI hype train. (Someone's probably got "release AI software as Google open source" as an OKR)
- pmontra 3y ago> We suggest using a machine type of e2-standard-32 (32 vCPU, 16 core and 128 GB memory), an admittedly beefy machine. Average price per month $966.56 https://gcloud-compute.com/e2-standard-32.html https://gcloud-compute.com/e2-standard-32.html
- williamstein 3y agoYou can also get exactly that same VM as a spot instance with a 10GB disk and external ip address in the us-west4 region for $142.04/month. (Spot instance prices vary dramatically from one region to another, but that one is very discounted in us-west4.) Also, is this localllm kind of like a Google competitor to some aspects of ollama?
- api 3y agoYou could buy 4-6 machines like that per year at that price. Gives you a taste of cloud mark-up. Bandwidth markup is far more insane.
- eyegor 3y agoFor $1000 you could basically build one today. 5950x (16c cpu) is $400, 128gb kit of ddr4 is $250-300, maybe enough budget left for mobo, ssd, psu, case, cooler if you bought used bits (may also need cheapest gpu on Craigslist depending on your needs). All new ~$1200. Aws/gcp prices are nuts compared to on prem, although hetzner and co would be much cheaper.
- omgwtfbyobbq 3y agoYeah, Cloud is definitely a money maker. If someone has a microcenter nearby, they could build an 8 core ddr4 system with 128gb of ram for maybe $800. $3200+ could build a 64 core/128gb ddr5 rig. https://www.newegg.com/tyan-s8050gm2ne-9534-amd-epyc-9534-2-45ghz-installed/p/N82E16813151352 https://www.newegg.com/tyan-s8050gm2ne-9534-amd-epyc-9534-2-...
- RecycledEle 3y agoPrice a HP z440 with those specs on eBay or Amazon.
- andy99 3y agoOne thing I'm happy about is that the gguf file format is getting widely used, here endorsed by Google. I vastly prefer working with gguf and have adopted it for my project, hopefully it gets the critical mass to become a "native" format and we can stop with pytorch checkpoints.
- brucethemoose2 3y agoOn the contrary, it's not one-size-fits-all. It doesn't support everything other backends need (like certain quantization schemes). If it somehow does, then its not universally compatible and that's just going to create more confusion.
- behnamoh 3y agoIt supports grammar and that's more than enough in practice.
- brucethemoose2 3y agoGrammar has nothing to do with gguf, as it's logit processing (and that is being integrated into other backends as well).
- UncleOxidant 3y ago> pip3 install openai Why is that required?
- deleted 3y ago[deleted]
- brucethemoose2 3y agoEveryone uses that for openai compatible api requests. Its actually great. Most backends and frontends "just work" with each other because they all talk with openai (albeit with some caveats).
- smcleod 3y agoI think you might be thinking of litellm? OpenAI is for OpenAI.com (although you can override the base URL it’s far from ideal).
- Tostino 3y agoWhat do you mean "far from ideal"? It's literally just passing in your services URL rather than using the default when instantiating the client. Something you will need to do with any client you use.
- spacecadet 3y agoI wondered that too. I see a comment, but I still wonder why its "required" here.
- deleted 3y ago[deleted]
- simonw 3y agoI'm not thrilled about https://github.com/GoogleCloudPlatform/localllm/blob/main/llm-tool/setup.py https://github.com/GoogleCloudPlatform/localllm/blob/main/ll... calling their Python package "llm" and installing "llm" as a CLI command, when my similar https://llm.datasette.io/ https://llm.datasette.io/ project has that namespace reserved on PyPI already: https://pypi.org/project/llm/ https://pypi.org/project/llm/
- rmorey 3y agoI thought exactly the same - it actually drives me nuts that Python packages have this confusing dual-name nature. You were there first!
- triyambakam 3y agoI actually wish that no one took the llm namespace. It's annoyingly vague.
- simonw 3y agoI think it's a pretty great name for my tool, given that it lets you interact with dozens of different LLMs via plugins. Plus I like that it's three letters, since it's designed for CLI usage.
- triyambakam 3y agoBut your tool isn't an llm. llmt for e.g. llm tool could be better
- thih9 3y agoYes - at the same time it would similarly fit other tools where some form of working with LLMs is the main feature. And with time we might get more of these. At least it’s not called “ai”, we have that already, e.g. https://github.com/yufeikang/ai-cli https://github.com/yufeikang/ai-cli
- wokwokwok 3y agoSure, but it’s not like you own the name LLM. I mean, let’s say you picked say, “ghost pony”; ok, sure, it’s obviously a name. But LLM? I mean, it’s not like you can get a lot more generic than that. If your tools name is “tool” and some else makes a tool called “tool”, you can’t really complain that much…
- smcleod 3y agoHow can you call something “local”LLM and have it rely on Google cloud?! > “…within the Google Cloud Workstation”
- seu 3y agoExactly my thought.
- popcorncowboy 3y agoLocal from _their_ perspective, deep inside the beast.
- tracerbulletx 3y agoThey've really jumped the shark. Creating a wrapper for llama.cpp (a project they don't even credit here) and acting like that's even interesting, then somehow implying this is a feature or benefit of using Google Cloud Workstations? This is the most technically bankrupt product page I've ever seen in my life and for a company that used to mean something to people to do it is pathetic.
- segmondy 3y agoit's actually a wrapper around llama-cpp-python. they ought to be embarrassed.
- m00x 3y agoGoogle really has fallen off. This is intern level stuff.
- vladsanchez 3y agohttps://www.sfgate.com/tech/article/google-workers-company-culture-pichai-18653877.php https://www.sfgate.com/tech/article/google-workers-company-c...
- mcapodici 3y agoHey that's my title ... almost :-) https://martincapodici.com/2023/07/15/no-local-gpu-no-problem-running-andrej-karpathys-nanogpt-on-modal-com/ https://martincapodici.com/2023/07/15/no-local-gpu-no-proble...
- hongweiyap5 3y agohvjvhcgxgxggcgcgcgcgcywww.@aliyun.com
- reissbaker 3y agoThis blog post has a distinct "written by an LLM" feel to it that I can't quite put my finger on.
- mirekrusin 3y agoBullet points starting with bolded text, longer, objective, soulless sentences devoided of "I".
- nicolezhu 3y agoAs a former Google employee working on their OSS stack, this feels a bit off brand. Here's a whole list of indie, OSS AI frameworks you can use: https://github.com/janhq/awesome-local-ai https://github.com/janhq/awesome-local-ai Disclaimer: I'm one of the core devs on Jan.
- iAkashPaul 3y agoThe amount of obfuscation & head-in the-ground was nauseating to read
- geuis 3y agoFor the interested, I've been having fun with llama.cpp. I've got it running well, if a bit slowly, on my 2019 intel MacBook Pro. There's configs you can tweak that get it to run faster, mainly dealing with memory mapping. I've been working on documenting my findings but haven't published yet. I say all that to encourage people to try it out with different models. It's not production quality but totally usable for testing things out even on older machines.
- airtonix 3y ago[dead]
- IanCal 3y agoA package called localllm (three l's in the middle, nice) which is focused on running models in GCP. > run LLMs locally on CPU and memory, right within the Google Cloud Workstation And in the code https://github.com/GoogleCloudPlatform/localllm/blob/main/querylocal.py https://github.com/GoogleCloudPlatform/localllm/blob/main/qu... there's just a shebang halfway down? import sys def main(argv): pass if __name__ == '__main__': main(sys.argv) #!/usr/bin/env python3 This is awful. Did bard write this?
- lynx23 3y agoLooks like the repo (localllm) only contains a smallish Python project to do the model downloading. When it comes to serving the model, it calls out a Python module llama_cpp.server, which is not part of the repo, likely something inside the Docker image. A bit disappointing given how the article sounds.
- Havoc 3y agoGoogle really doesn’t seem to have much of a coherent plan on AI by the looks of it
- luke-stanley 3y agoThe idea of making it easier to download modes from HuggingFace, and serve them via llama-cpp-python could be a good contribution, but unfortunately there are two different names that the "Localllm" project introduces for itself - Localllm and the 'llm' command, both of them problematic for quite a few of reasons, perhaps they could just drop naming it anything to do with local? E.g: Simon Willison's well known `llm` package on PyPI already uses the `llm` command. A name for the command that differs from the project name creates extra work for no good reason. Look at this positioning Google use in the blog post: "localllm combined with Cloud Workstations revolutionizes AI-driven application development by letting you use LLMs locally on CPU and memory within the Google Cloud environment." - implying a local setup that's actually reliant on cloud resources is a contradiction. Maybe they could rename it with a single name that has no conflicts or misleading terms that makes it clear it's to help setup and run inference without confusing the matter about how local things might sometimes be. It won't be nice to read the scathing critiques in some of the other comments! Maybe they could fix up the mess in querylocal.py! Being able to run the same code locally as in a cloud platform is a nice development pattern but optimal inference setup for many ML situations is a tricky problem, are you really making good use of the CPU / GPU / TPU? The underlying server tech: llama-cpp-python recently added support for draft models that could make inference much faster. How easy would using that from Localllm be? I'd be concerned `Localllm > llama-cpp-python > llama.cpp` might be too many layers of systems to optimise or code with. Milliseconds often matter. Having a fast, well priced solution is not easy, people want as many tokens per second as they can get, ideally with no warm-up. How would they solve that? People looking at fast inference options might like to see Banana's parting words, as they wind down their serverless GPU cloud, in their swansong, they suggest a few hosted inference options people might want to consider: https://news.ycombinator.com/item?id=39288915 https://news.ycombinator.com/item?id=39288915
- MrYellowP 3y agoWhat an awfully misleading name. This doesn't run locally, it runs on the cloud. What the fuck was that guy thinking?