5 ms·
I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version). I tried running a full codebase
by simonw 2y ago
I'm absolutely amazed at how capable the new 1B model is, considering it's just a 1.3GB download (for the Ollama GGUF version).
I tried running a full codebase through it (since it can handle 128,000 tokens) and asking it to summarize the code - it did a surprisingly decent job, incomplete but still unbelievable for a model that tiny: https://gist.github.com/simonw/64c5f5b111fe473999144932bef4218b https://gist.github.com/simonw/64c5f5b111fe473999144932bef42...
More of my notes here: https://simonwillison.net/2024/Sep/25/llama-32/ https://simonwillison.net/2024/Sep/25/llama-32/
I've been trying out the larger image models to using the versions hosted on https://lmarena.ai/ https://lmarena.ai/ - navigate to "Direct Chat" and you can select them from the dropdown and upload images to run prompts.
- GaggiX 2y agoLlama 3.2 vision models don't seem that great if they have to compare them to Claude 3 Haiku or GPT4o-mini. For an open alternative I would use Qwen-2-72B model, it's smaller than the 90B and seems to perform quite better. Also Qwen2-VL-7B as an alternative to Llama-3.2-11B, smaller, better in visual benchmarks and also Apache 2.0. Molmo models: https://huggingface.co/collections/allenai/molmo-66f379e6fe3b8ef090a8ca19 https://huggingface.co/collections/allenai/molmo-66f379e6fe3..., also seem to perform better than Llama-3.2 models while being smaller and Apache 2.0.
- dannyobrien 2y agoWhat interface do you use for a locally-run Qwen2-VL-7B? Inspired by Simon Willison's research[1], I have tried it out on Hugging Face[2]. Its handwriting recognition seems fantastic, but I haven't figured out how to run it locally yet. [1] https://simonwillison.net/2024/Sep/4/qwen2-vl/ https://simonwillison.net/2024/Sep/4/qwen2-vl/ [2] https://huggingface.co/spaces/GanymedeNil/Qwen2-VL-7B https://huggingface.co/spaces/GanymedeNil/Qwen2-VL-7B
- Eisenstein 2y agoMiniCPM-V 2.6 is based on Qwen 2 and is also great at handwriting. It works locally with KoboldCPP. Here are the results I got with a test I just did. Image: * https://imgur.com/wg0kdQK https://imgur.com/wg0kdQK Output: * https://pastebin.com/RKvYQasi https://pastebin.com/RKvYQasi OCR script used: * https://github.com/jabberjabberjabber/LLMOCR/blob/main/llmocr.py https://github.com/jabberjabberjabber/LLMOCR/blob/main/llmoc... Model weights: MiniCPM-V-2_6-Q6_K_L.gguf, mmproj-MiniCPM-V-2_6-f16.gguf Inference: * https://github.com/LostRuins/koboldcpp/releases/tag/v1.75.2 https://github.com/LostRuins/koboldcpp/releases/tag/v1.75.2
- hansoolo 2y agoThanks for the hint. Will try the out!
- jona-f 2y agoShould the line "p.o. 5rd w/ new W5 533" say "p.o. 3rd w/ new WW 5W .533R"? What does p.o. stand for? I can't make out the first letter. It looks more like the f, but the nodge on the upper left only fits the p. All the other p's look very different though.
- Eisenstein 2y ago'Replaced R436, R430 emitter resistors on right-channel power output board with new wire-wound 5watt .33ohm 5% with ceramic lead insulators'
- jona-f 2y agoThx :). I thought the 3 looked like a b but didn't think brd would make any sense. My reasoning has led me astray.
- Eisenstein 2y agoYeah. If you realize that a large part of the llm's 'ocr' is guessing due to context (token prediction) and not actually recognizing the characters exactly, you can see that it is indeed pretty impressive because the log it is reading uses pretty unique terminology that it couldn't know from training.
- f38zf5vdt 2y ago1. Ignore the benchmarks. I've been A/Bing 11B today with Molmo 72B [1], which itself has an ELO neck-and-neck with GPT4o, and it's even. Because everyone in open source tends to train on validation benchmarks, you really can not trust them. 2. The method of tokenization/adapter is novel and uses many fewer tokens than all comparable CLIP/SigLIP-adapter models, making it _much_ faster. Attention is O(n^2) on memory/compute per sequence length. [1] https://molmo.allenai.org/blog https://molmo.allenai.org/blog
- GaggiX 2y agoHow about its performance compare to Qwen-2-72B tho?
- f38zf5vdt 2y agoRefer to the blog post I linked. Molmo is ahead of Qwen2 72b.
- benreesman 2y agoIt’s not just open source that trains on the validation set. The big labs have already forgotten more about gaming MMLU down to the decimal than the open source community ever knew. Every once in a while they get sloppy and Claude does a faux pas with a BIGBENCH canary string or some other embarrassing little admission of dishonesty like that. A big lab gets exactly the score on any public eval that they want to. They have their own holdouts for actual ML work, and they’re some of the most closely guarded IP artifacts, far more valuable than a snapshot of weights.
- sumedh 2y agoI tried some OCR use cases, Claude Sonnet just blows Molmo.
- knicholes 2y agoWhen you say "blows," do you mean in a subservient sense or more like, "it blows it out of the water?"
- foxhop 2y agoThe llama 3.0, 3.1, & 3.2 all use the TikToken tokenizer which is the open source openai tokenizer.
- littlestymaar 2y agoGP is talking about context windows, not the number of token used by the tokenizer.
- sva_ 2y agoSomewhat confusingly, it appears the tokenizer vocabulary as well as the context length are both 128k tokens!
- littlestymaar 2y agoYup, that's why I wanted to clarify things.
- jackbravo 2y agoI saw that you mention https://github.com/simonw/llm/ https://github.com/simonw/llm/. Hadn't seen this before. What is its purpose? And why not use ollama instead?
- dannyobrien 2y agollm is Simon's command line front-end to a lot of the llm apis, local and cloud-based. Along with aider-chat, it's my main interface to any LLM work -- it works well with a chat model, one-off queries, and piping text or output into a llm chain. For people who live on the command line, or are just put-off by web interfaces, it's a godsend. About the only thing I need to look further abroad for is when I'm working multi-modally -- I know Simon and the community are mainly noodling over the best command line UX for that: https://github.com/simonw/llm/issues/331 https://github.com/simonw/llm/issues/331
- n8henrie 2y agoI've only used ollama over cli. As per the parent poster -- do you know if there are advantages over ollama for CLI use? Have you used both?
- dannyobrien 2y agoI haven't used Ollama, but from what I've seen, it seems to operate at a different level of abstraction compared to `llm`. I use `llm` to access both remote and local models through its plugin ecosystem[1]. One of the plugins allows you to use Ollama-served local models. This means you can use the same CLI interface with Ollama[2], as well as with OpenAI, Gemini, Anthropic, llamafile, llamacpp, mlc, and others. I select different models for different purposes. Recently, I've switched my default from OpenAI to Anthropic quite seamlessly. [1] - https://llm.datasette.io/en/stable/plugins/directory.html#plugin-directory https://llm.datasette.io/en/stable/plugins/directory.html#pl... [2] - https://github.com/taketwo/llm-ollama https://github.com/taketwo/llm-ollama
- awwaiid 2y agoThe llm CLI is much more unixy, letting you pipe data in and out easily. It can use hosted and local models, including ollama.
- forgingahead 2y agoWhat are people using to check token length of code bases? I'd like to point certain app folders to a local LLM, but no idea how that stuff is calculated? Seems like some strategic prompting (eg: this is a rails app, here is the folder structure with file names, and btw here are the actual files to parse) would be more efficient than just giving it the full app folder? No point giving it stuff from /lib and /vendor for the most part I reckon.
- simonw 2y agoI use my https://github.com/simonw/ttok https://github.com/simonw/ttok command for that - you can pipe stuff into it for a token count. Unfortunately it only uses the OpenAI tokenizers at the moment (via tiktoken), so counts for other models may be inaccurate. I find they tend to be close enough though.
- sumedh 2y agoYou can try Gemini Token count. https://ai.google.dev/api/tokens https://ai.google.dev/api/tokens
- xyc 2y agoYou can use llama.cpp server's tokenize endpoint to tokenize and count the tokens: https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md#post-tokenize-tokenize-a-given-text https://github.com/ggerganov/llama.cpp/blob/master/examples/...
- lowyek 2y agoHi simon, is there a way to run the vision model easily on my mac locally?
- simonw 2y agoNot that I’ve seen so far, but Ollama are pending a solution for that “soon”.
- v3ss0n 2y agoI doubt ollama team can do much about it. Ollama are just wrapper on top of heavy lifter
- deleted 2y ago[deleted]
- Patrick_Devine 2y agoThe draft PRs are already up in the repo.
- theaniketmaurya 2y agoYou can run it with LitServe (MPS GPU), here is the code - https://lightning.ai/lightning-ai/studios/deploy-llama-3-2-vision-with-litserve https://lightning.ai/lightning-ai/studios/deploy-llama-3-2-v...
- faangguyindia 2y agoIf you are in US, you get 1 billion tokens a DAY with Gemini (Google) completely free of cost. Gemini Flash is fast with upto 4 million token context. Gemini Flash 002 improved in math and logical abilities surpassing Claude and Gpt 4o You can simply use Gemini Flash for Code Completion, git review tool and many more.
- nycdatasci 2y agoThis is great for experimentation, but as others have pointed out recently there are persistent issues with Gemini that prevent use in actual products. The recitation/self-sensoring issue results in random failures: https://github.com/google/generative-ai-docs/issues/257 https://github.com/google/generative-ai-docs/issues/257
- faangguyindia 2y agoI had this problem too but 002 solves this I think (not tested exhaustively), but I've not run into any problems since 002 and vertex + block all on all safety is now working fine, earlier I had problems with "block all" in safety settings and api throwing errors. I am using it in https://github.com/zerocorebeta/Option-K https://github.com/zerocorebeta/Option-K (currently it doesn't have lowest safety settings because api wouldn't allow it, but now I am going to push new update with safety disabled) Why? I've another application which is working since yesterday after 002 launch, I've safety settings to none and it will not answer certain questions but since yesterday it answers everything.
- o11c 2y agoAnd yet - if Gemini actually bothers to tell you when it detects verbatim copying of copyrighted content, how often must that occur on other AIs without notice?
- a2128 2y agoIs this sustainable though, or are they just trying really hard to attract users? If I build all of my tooling on it, will they start charging me thousands of dollars next year once the subsidies dry up? With a local model running with open source software, at least I can know that as long as my computer can still compute, the model will still run just as well and just as fast as it did on day 1, and cost the same amount of electricity
- TZubiri 2y agoThis obsession with using AI to help with programming is short sighted. We discover gold and you think of gold pickaxes.
- magicalhippo 2y agoI'm new to this game. I played with Gemma 2 9B in an agent-like role before and was pleasantly surprised. I just tried some of the same prompts with Llama 3.2 3B and found it doesn't stick to my instructions very well. Since I'm a n00b, does this just mean Llama 3.2 3B instruct was "tuned more softly" than Gemma 2 instruct? That is, could one expect to be able to further fine-tune it to more closely follow instructions?