9 ms·
I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on
by simonw 3y ago
I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though):
1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/llamafile-server-0.1-llava-v1.5-7b-q4 https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/...:
wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llamafile-server-0.1-llava-v1.5-7b-q4
2. Make that binary executable, by running this in a terminal:
chmod 755 llamafile-server-0.1-llava-v1.5-7b-q4
3. Run your new executable, which will start a web server on port 8080:
./llamafile-server-0.1-llava-v1.5-7b-q4
4. Navigate to http://127.0.0.1:8080/ http://127.0.0.1:8080/ to upload an image and start chatting with the model about it in your browser.
Screenshot here: https://simonwillison.net/2023/Nov/29/llamafile/ https://simonwillison.net/2023/Nov/29/llamafile/
- mritchie712 3y agowoah, this is fast. On my M1 this feels about as fast as GPT-4.
- pmarreck 3y agoSame here on M1 Max Macbook Pro. This is great!
- pyinstallwoes 3y agoHow good is it in comparison
- int_19h 3y agoThe best models available to the public are only slightly better than the original (pre-turbo) GPT-3.5 on actual tasks. There's nothing even remotely close to GPT-4.
- pyinstallwoes 3y agoWhat’s the best in terms of coding assistance? What’s annoying about gpt 4 is that is seems badly nerfed in many ways. It is obviously being conditioned in its own political bias.
- dmd 3y ago... is the javascript it's writing for you too 'woke' or something?
- pyinstallwoes 3y agoLately it’s been refusing to write code at all. // implementation details here. I think it’s the opposite of woke, it’s slept!
- eyegor 3y agoIn my experience, the deepseek-coder-instruct family is at least as good as gpt-3.5-turbo on python. Even the 1.3b models are very good (and run okay on cpu), although you should use larger if you have the vram. There are even larger models (30b+) if you are drowning in vram, but I don't think they perform much better at coding than deepseek-coder-instruct 6.7b. 3-4gb vram or cpu (1.3b): https://huggingface.co/TheBloke/deepseek-coder-1.3b-instruct-GGUF https://huggingface.co/TheBloke/deepseek-coder-1.3b-instruct... Alternative for chat (1.3b): https://huggingface.co/TheBloke/evolvedSeeker_1_3-GGUF https://huggingface.co/TheBloke/evolvedSeeker_1_3-GGUF Alternative for chat (3b): https://huggingface.co/TheBloke/open-llama-3b-v2-wizard-evol-instuct-v2-196k-GGUF https://huggingface.co/TheBloke/open-llama-3b-v2-wizard-evol... 6-8gb vram (6.7b): https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct-GGUF https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...
- rightbyte 3y agoReally? How could this model not have had front page articles on HN? Is the self-hosted one as good as the one or their website?
- bilsbie 3y agoThanks for the tip! Any chance this would run on a 2011 MacBook?
- simonw 3y agoJustine says it needs MacOS 13.6+ - does that run on that machine?
- mcint 3y agoYes, with a patch https://en.wikipedia.org/wiki/MacBook_Pro#macOS https://en.wikipedia.org/wiki/MacBook_Pro#macOS from https://dortania.github.io/OpenCore-Legacy-Patcher/MODELS.html#macbook https://dortania.github.io/OpenCore-Legacy-Patcher/MODELS.ht... I thought my 2015 MBP wasn't able to upgrade. Good to know it's still supported.
- pmarreck 3y agodo you... have any plans to upgrade? A gen 2011 computer is going to get harder and harder to make work. even a used macbook from like 2019 would probably be a steal at this point, and that's 8 years further along All the new AI toys especially seem to love beefy newish hardware and especially GPU hardware if available
- getwiththeprog 3y agomakes one think that perhaps Apple isn't a good long term choice...
- pmarreck 3y agoApple is a fine long-term choice (speaking as a recent linux advocate actually, lol). But that computer is 12 years old
- interfixus 3y agoSo what? This crazy thing runs fine, albeit slowly, on my 12 year old ThinkPad. It's actually digesting an image of an anteater while I write this. Because of course it plays nicely and doesn't hog the hardware.
- callmeed 3y agowhen I try to do this (MBP M1 Max, Sonoma) I get 'killed' immediately
- carbocation 3y agoSame on an M1 Max 64G, Ventura. Xcode is installed[1]. 1 = ``` $ xcode-select --install xcode-select: error: command line tools are already installed, use "Software Update" in System Settings to install updates ```
- carbocation 3y agoFor whatever it's worth, the SHA sum is correct. The killed message is uninformative, looks like what happens when I'm OOM (but I have 64GB RAM of which only 24 is used for anything at the moment). $ sha256sum < llamafile-server-0.1-llava-v1.5-7b-q4 a138c5db9cff3b8905dd6e579c2ab6c098048526b53ae5ab433ff1d1edb9de24 - $ ./llamafile-server-0.1-llava-v1.5-7b-q4 Killed: 9
- carbocation 3y agoLooks like this may be due to crowdstrike, which I also have installed on this machine: https://github.com/Mozilla-Ocho/llamafile/issues/14#issuecomment-1834313659 https://github.com/Mozilla-Ocho/llamafile/issues/14#issuecom...
- raajg 3y agooh wow, I would have never imagined that 'that' would be preventing me from running llama + llava! Confirming that I have Crowdstrike running too.
- derwiki 3y agoOn a Macbook Pro M2, I get $ ./llamafile-server-0.1-llava-v1.5-7b-q4 [2] 25224 illegal hardware instruction ./llamafile-server-0.1-llava-v1.5-7b-q4
- lol768 3y ago$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4 $ ./llamafile-server-0.1-llava-v1.5-7b-q4 run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4 Hmm. Did I do something wrong? (Ubuntu 22.04 / ) Installing the portable binfmt_misc gets me further, but still: $ ./llamafile-server-0.1-llava-v1.5-7b-q4 zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4 $ sh -c ./llamafile-server-0.1-llava-v1.5-7b-q4 sh: 1: ./llamafile-server-0.1-llava-v1.5-7b-q4: Permission denied
- phh 3y agoLast thing you need is to chmod +x the interpreter: chmod +x /usr/bin/ape (it is indeed not in the README)
- lol768 3y agoThis was it, wget creates the file and it's non-executable (and I'd already double checked the actual llamafile was executable, but had missed this) - thanks!
- stavros 3y agoI get the same error, and there's no `ape` file to make excecutable, hm.
- jart 3y agoYou can manually download the `ape` command from https://cosmo.zip/pub/cosmos/bin/ https://cosmo.zip/pub/cosmos/bin/ Please see the Gotchas section of the README for the copy/pastable commands you can run: https://github.com/mozilla-Ocho/llamafile#gotchas https://github.com/mozilla-Ocho/llamafile#gotchas
- jart 3y agoYou can solve the run-detectors issue with: sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" You can solve the zsh permission denied issue by either (1) upgrade to zsh 5.9+ (I upstreamed a fix for this bug in zsh two years ago) or (2) use the sh -c workaround you discovered. If that one doesn't work, then it likely needs to be chmod +x. If the execute bit is set, and your sh still isn't working, then please let me know, because I'm not aware of any sh that still doesn't support APE. See the Gotchas section of the README https://github.com/mozilla-Ocho/llamafile#gotchas https://github.com/mozilla-Ocho/llamafile#gotchas
- bsenftner 3y agoSuper duper impressed. I've run llamafile-server-0.1-llava-v1.5-7b-q4 against the tests I need to pass for use in a project, and this passes them all, vision queries too. This is gonna change quite a bit, strategy-wise for quite a few people.
- StockHuman 3y agoPhenomenal quickstart, and thanks for the write-up. It’s so thrilling that we’re at this point in portability and ease relative performance.
- jxy 3y agoso next time llama.cpp releases an update, other people update their favorite backend, you redownload a 4.26 GB file. Epic. EDIT: oh, wait. Actually people usually have a handful to a few dozen of the these models lying around. When they update their backend, you just redownload every single model again. EDIT 2: right, you can release a program that automatically patches and updates the downloaded model+executables. Such an invention.
- Zuiii 3y agoThat's why I always download the original version and quantize myself. With enough swap, you can do it with a modest amount for ram. I never had to download a model twice. But yes, unless there is a way to patch it, bundling the model with the executable like this is going to be more wasteful.
- simonw 3y agoEach llamafile is a .zip, so if you want to extract the weights out of it you can extract the gguf file directly. unzip -l llamafile-server-0.1-llava-v1.5-7b-q4 | grep llava-v1 Archive: llamafile-server-0.1-llava-v1.5-7b-q4 4081004224 11-15-2023 22:13 llava-v1.5-7b-Q4_K.gguf 177415936 11-15-2023 22:13 llava-v1.5-7b-mmproj-Q4_0.gguf
- aphroz 3y agoCompared to modern bandwidth usage that's not such a big size anymore. Everyday millions of people download 100gb video games, watch 4k video podcasts, etc.
- simonw 3y agoYou can even run a full LLM in your browser these days - try https://webllm.mlc.ai/ https://webllm.mlc.ai/ in Chrome, it can load up a Llama-2-7b chat model (~4000MB, took my connection just under 3 minutes) and you can start chatting with it.
- column 3y agoThis is for convenience. You can also download a 4.45Mb executable (llamafile-server-0.1) and pass any GGUF model as an argument. > llamafile-server-0.1 -m llama-2-13b.Q8_0.gguf https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.1 https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.1
- thejosh 3y agoDamn this is fast and accurate! Crazy how far things are progressing.
- deleted 3y ago[deleted]
- sebmellen 3y agoWow, this is almost as good as chatgpt-web [0], and it works offline and is free. Amazing. In case anyone here hasn't used chatgpt-web, I recommend trying it out. With the new GPT-4 models you can chat for way cheaper than paying for ChatGPT Plus, and you can also switch back to the older (non-nerfed) GPT-4 models that can still actually code. [0]: https://github.com/Niek/chatgpt-web https://github.com/Niek/chatgpt-web
- oezi 3y agoWay cheaper? I thought that 1K Tokens (in+out) cost 0.04 USD in GPT-4 Turbo, which is roughly one larger chat response (2 screens). To reach parity with ChatGPT Plus pricing you need thus to use less than 500 such responses per month via API. For GPT-4 the pricing is double that (0.09 USD per 1K). So only 200 larger interactions to reach 20 USD cost. Or am I wrong?
- anonzzzies 3y agoIt depends on your usage; for me the plus sub is much cheaper than if I use the api directly, but I use it a lot for everything I do.
- topato 3y ago[flagged]
- sebmellen 3y agoIn my experience, each message with the 1106 preview model costs me about $0.006, which is acceptable. Most importantly, the API is higher availability (no "you have reached your message limit") and I feel more comfortable using proprietary data with it, as data sent through the API won't be used to train the model. Now, if the chat gets very long or is heavy on high-token strings (especially code), those costs can balloon up to the 9-12 cent region. I think this is because chatgpt-web loads all the prior messages in the chat into the context window, so if you create a new chat for each question you can lower costs substantially. Most often I don't need much prior context in my chat questions anyway, as I use ChatGPT more like StackOverflow than a conversation buddy. Also, it's a lot easier to run company subscriptions this way, as we don't have to provision a new card for each person to use the web version. I believe there is an Enterprise version of ChatGPT, but chatgpt-web is functionally equivalent and I'm sure it costs less.
- anonzzzies 3y agoVery nice; works perfect on Ubuntu 20.04. Doing 8 tokens/s on a pretty crappy server.
- belter 3y agoPerfectly on Fedora 39 on old ( and I mean old...) machines. This is actually shocking...shocking good...
- tluyben2 3y agoPopped it into a docker setup: https://github.com/tluyben/llamafile-docker https://github.com/tluyben/llamafile-docker to save even more keystrokes.
- luke-stanley 3y agoIt was already great, and this is more great for those who get Docker-Compose or are patient enough to figure out. But if you're gonna have Docker, you could also use bleeding edge llama.cpp with a few more lines too! What a time to be alive innit!
- finnjohnsen2 3y agoThank you kindly
- reisse 3y agoWhat is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.
- belter 3y agoDeploy your models into a Kubernetes cluster and let them fight for resources to death? A modern Roman Gladiators circus with Models?
- ukuina 3y agoGive them write access to your control-plane and the first one to write syntactically correct IaC wins!
- Bjartr 3y agoMore like a sequel to Core War... Which actually does sound pretty amusing now that I think about it. Call it Cloud War. https://en.m.wikipedia.org/wiki/Core_War https://en.m.wikipedia.org/wiki/Core_War
- finnjohnsen2 3y ago
- botanical 3y agoI just tried asking it a question: > User: What is the third planet from the sun? > Llama: The third planet from the sun is called Mars.
- creshal 3y agoSkynet is collaborating with the Martians already, I see.
- eek2121 3y agoLlama is just from the future. That is all…
- zoomablemind 3y ago> ...> Llama: The third planet from the sun is called Mars. Ask it if is there life on Mars in that parallel reality
- czbond 3y agoThe model is trained on large volume data, correct? Why would it get such a simple fact incorrect?
- JabavuAdams 3y agoLLMs are known to be bad at counting. It would be interesting to see the answer to "List the planets in our solar system, starting with the closest to the sun, and proceeding to farther and farther ones." Also, the knowledge can be kind of siloed. You often have to come at it in weird ways. Also, they are not fact-bases. They are next-token-predictors, with extra stuff on top. So if people on the internet often get the answer wrong, so will the model.
- mud_dauber 3y agoI just tried the same "third planet from the sun" question and got the correct response. No other training or tweaks. Can't wait to unleash Pluto questions.
- tmountain 3y agoThis is amazing. How does result quality compare to GPT4 for image analysis?
- simonw 3y agoIt's not remotely as good as GPT-4 Vision, which isn't a big surprise consisting it's running a 4GB, 7B parameter model on your laptop, trained by a small research team.
- m1thrandir 3y agoIs the process the same for running multi-part bins? Like the latest deekseek 67b model?
- pmarreck 3y agoAnyone have any tuning tips? I messed with some of the configs and now it's mostly hallucinating answers or going off the rails
- brianjking 3y agoThis can truly revolutionalize education and access, feels like what I hoped the One Laptop Per Child project would do, this could actually achieve. We just need someone with a heap of funds to package this up into a very inexpensive machine and distribute them.
- mark_l_watson 3y agoSimon, does this also work well on Apple Silicon? EDIT: never mind, other commenters here answered my question: yes it does work. I have a M2Pro with 32 G on chip memory - amazingly effective for experimenting with open LLMs.
- theptip 3y agoMy pride as a technologist tells me I should be able to get any Python package up and running, but man, AI dependency management is a dumpster fire right now; adding GPU driver versions into the mix seems to make everything really brittle. This seems like a great approach to compare multiple models, in particular.
- NoOn3 3y agoGot this: Terminating on uncaught SIGILL.
- jart 3y agoI'm assuming you're on Apple Silicon? Please follow https://github.com/Mozilla-Ocho/llamafile/issues/11 https://github.com/Mozilla-Ocho/llamafile/issues/11 which is tracking this. We've received multiple reports even though I personally haven't figured out how to reproduce it yet.
- NoOn3 3y agoNo just old i5-2500k 16GB Ram and vega 56 GPU 8GB Vram.
- NoOn3 3y agoI suspect it's lack of AVX2 support in my cpu.
- jart 3y agoSo you have a Sandybridge processor with AVX support (but not AVX2). Could you open llamafile in a debugger and find out which instructing is faulting and what its address in memory is? I haven't personally tested Sandybridge but I'm reasonably certain we designed the build to not insert any incompatible instructions in there. Our intent is to support you. I also might be able to fish my old ThinkPad out of the closet if you don't have time. In any case it'll happen soon.
- NoOn3 3y agoI tried but gdb showed me nothing, this screenshot(https://pxscdn.com/public/m/_v2/97422265439883264/bc40e5d2a-3c51e6/JF4cRHC2lMRA/mebGnttixotw07UTiDJJOLqIWIsrmLYMygtoIzrT.png https://pxscdn.com/public/m/_v2/97422265439883264/bc40e5d2a-...). gdb layout asm shows "No Assembly Available". Maybe I'm not so skilled in debbuging such programs it's running under wine it seemed to me.
- silentguy 3y agowget url in step 1 seems to be wrong. It didn't work. This url `https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llava-v1.5-7b-q4-server.llamafile https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/ma... ` seems to be working. It's from the link you posted.
- jart 3y agoIt's back. Sorry about that.