4 ms·
Accessing Llama 2 from the command-line with the LLM-replicate plugin
- peatmoss 3y agoI feel like Simon's been on a tear with these LLM postings. Simon, I really enjoying you swashbuckling through this, and then documenting your travels.
- turnsout 3y agoDoes Simon sleep? He's unstoppable!
- MuffinFlavored 3y agoI kind of wonder the opposite. > created: October 29, 2007 > karma: 40256 Is he not potentially under the category of "potentially starved for online HackerNews attention"? Why does he feel the need to maintain a blog and libraries, etc. etc. and then submit it and let us all know about it?
- haswell 3y agoTechnology is fun! Sharing with others is fun! Why do any of us do anything?
- mdp2021 3y agoBecause we know we learn from others so we also offer. Constructive reciprocity. Not dissimilar from sharing code.
- IanCal 3y agoThis would be a solid point if he kept putting out nonsense or useless but catchy things, but he keeps putting out pretty concise useful things. Useful tech things are pretty popular here, for good reason.
- simonw 3y agoMore about my LLM tool (and Python library) here: https://llm.datasette.io/ https://llm.datasette.io/ Here's the full implementation of that llm-replicate plugin: https://github.com/simonw/llm-replicate/blob/0.2/llm_replicate/__init__.py https://github.com/simonw/llm-replicate/blob/0.2/llm_replica... If you want to write a plugin for some other LLM I have a detailed tutorial here: https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html https://llm.datasette.io/en/stable/plugins/tutorial-model-pl... - plus a bunch of examples linked from here: https://github.com/simonw/llm-plugins https://github.com/simonw/llm-plugins
- Anticlockwise 3y agoCan you or anyone else comment on how replicate's per-second pricing ends up comparing to OpenAI's per token pricing when using Llama2?
- simonw 3y agoMy hunch is that OpenAI is a lot cheaper. I've spent $0.26 on 115 seconds of compute with Llama 2 on Replicate so far, which is only a dozen test prompts.
- ta988 3y agoIt is insanely more expensive on replica and they don't have the 70b model yet which will make it even more prohibitive.
- richdougherty 3y agoLooks like it's here now: https://replicate.com/replicate/llama70b-v2-chat https://replicate.com/replicate/llama70b-v2-chat As for pricing, that model's pages says: "Predictions run on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 17 seconds." And the pricing page (https://replicate.com/pricing https://replicate.com/pricing) says Nvidia A100 (80GB) GPU hardware costs $0.0032 per second. So Llama 2 70B would "typically" cost under 17 x 0.0032 = $0.0544 per run.
- xrd 3y agoI'm so confused about running these models locally only. When reading about the llm tool, I thought, ok, this helps organize all the pieces on my machine. But then it uses a replicate API key, so clearly it requires a network connection. Is this just to download the models? I feel like we need a new license or packaging model that clearly states whether the computation is happening locally or remotely. It's very important to me and often hard to know until I'm a long way into the installation process.
- deleted 3y ago[deleted]
- ladberg 3y agoIdk, to me this reads pretty clearly remote (for this model in particularly): > My LLM tool provides command-line access to a wide variety of language models, both via web APIs and self-hosted on your own machine ... The brand new llm-replicate plugin provides CLI access to models hosted on Replicate, and this morning a16z-infra released a16z-infra/llama13b-v2-chat which provides Replicate API access to the new Llama 2 13B chat model. I'm also unsure what this would have to do with the license, when you install any other software do you expect the license to tell you whether processing happens locally or in the cloud?
- transcriptase 3y agoI’m in the same boat. With text-to-image models I pick out software of my choice and drop a downloaded model into an input directory, then launch a GUI. What is it about LLMs that makes actually being able to run them locally so complex in comparison? I’m genuinely curious as a layman.
- gcr 3y agoNothing in particular. Setting up a scientific python+pytorch stack is difficult if you're unfamiliar with the python packaging ecosystem. If you're not on a "happy path" of "ubuntu, nvidia, and anaconda," then lots of things could go wrong if you're configuring from scratch. If you want to run these models efficiently, hardware acceleration is a must, but managing the intersection between {GPU, operating system, architecture, Python version, Python virtualenv location} is tricky. That's even before you deal with hardware-specific implementation details! - Running NVidia? Double-check your CUDA library version, kernel module version, (optional) CuDNN, and pytorch version - Running AMD? Double-check your ROCm library version, AMD drivers, and make sure that you use Pytorch provided by AMD with ROCm support - On Apple machines? Double-check that your M1 hardware actually has proper hardware support, then download and install a custom Pytorch distribution linked with M1 support, and make sure that the numpy library version has been properly linked with Accelerate.framework or else your BLAS calls run on the CPU rather than the undocumented AMX coprocessor. If you want to run on the ANE, you'll additionally need a working xcode toolchain and a version of the CoreML model compiler that can read your serialized pytorch model files properly. I think the pain of getting things working makes it easier to just throw up one's hands and pay someone else to run your model for you.
- MrMan 3y agoI just downloaded llama2 - what more do I need to access it from the command line than the code and the weights?
- simonw 3y agoThis works too now for the 70b model: llm replicate add \ replicate/llama70b-v2-chat \ --chat --alias llama70b Then: llm -m llama70b "Invent an absurd ice cream sundae"