6 ms·
Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
- thedatamonger 2y agothis looks very awesome. can someone tell me why there is no chatter about this? is there something else out there that blows this out of the water in terms of ease of use and access to sample many LLM's ?
- perrygeo 2y agoOpposite reaction here. I was just thinking, man I hear about Ollama every single day on HN. Not sure a point release is news :-)
- throw03172019 2y agoLola a has been brought up many times on HN. It’s a great tool!
- chadsix 2y agoOllama is really organized - it relies on llama but the UX and organization it provides makes it legit. We recently made a one-click wizard to run Open WebUI and Ollama together, self hosted and remotely accessible but locally hosted [1] [1] https://github.com/ipv6rslimited/cloudseeder https://github.com/ipv6rslimited/cloudseeder
- FieryTransition 2y agoI use a mix of using llamacpp directly via my own python bindings and using it via llamacpp-python for function calling and full control over parameters and loading, but otherwise ollama is just great for ease of use. There's really not a reason not to use it, if just want to load gguf models and don't have any intricate requirements.
- brrrrrm 2y agoHN isnt really the best space for LLM news - r/LocalLlama and twitter are much better. I think HN has some cultural issues with “AI” news
- wkat4242 2y agoHmm I don't think so. Most comments are pretty positive. I think the articles are just not really upvoted unless it's really big news, makes sense because HN is for more than just AI. But I don't think it's anti-AI like most people here would be pretty anti-cryptocurrency (and for good reason IMO)
- p1esk 2y agoI didn’t upvote it because I don’t use Ollama. To experiment with LLMs I use Huggingface. Does Ollama provide something I cannot get with Huggingface?
- gertop 2y agoHugging face is a model repository. Ollama allows you to run those models. Different things.
- p1esk 2y agoI run models using HF just fine. I mean I’m using HF transformers repo, which gets models from HF hub. Or do you mean commercial deployment of models for inference?
- simonw 2y agoAre you talking about the Hugging Face Python libraries, the Hugging Face hosted inference APIs, the Hugging Face web interfaces, the Hugging Face iPhone app, Hugging Face Spaces (hosted Docker environments with GPU access) or something else?
- p1esk 2y agoI updated my comment above: I’m using HF transformers repo, which gets models from HF hub.
- CharlesW 2y agoI can recommend LM Studio and Msty if you're looking for something with an integrated UX.
- gertop 2y agoLM Studio is a lot more user friendly, probably the easiest UI to use out there. No terminal nonsense, no manual to read. Just double click and chat. It even explains to you what the model names mean (eg diff between Q4_1 Q4_K Q4_K_M... For whatever reason all the other tools assume you know what it means). Built-in model recommendations are also handy. Very friendly tool! However it's not open-source.
- Cheer2171 2y agoWhy do you think there is no chatter about this? There have been hundreds of posts about ollama on HN. This is a point release of an already well known project.
- yjftsjthsd-h 2y ago[Why] do models require a new version? It can already take arbitrary gguf; I assumed they just had a registry online
- ynniv 2y agoThey do, and I was using the "new" models before the update. Perhaps there is tuning or bug fixes for them? Or they just want to confirm that these are supported. There are some new models that do have different architectures, so sometimes an update is necessary.
- refulgentis 2y agoPhi 3 has a unique architecture that needed some additions to llama.cpp's conversion script. Also Phi 3 is an absolute mess, there's no reliable way to latch on to when it's done writing a message and no one wants to admit it, people are patching around it instead. ex. I could condition on "\n\n<|assistant|>||<|system|>||<|user>", but it'd still be wrong. Pretty much everything Phi 3 feels like it needed to all come out within 48 hours a month too early. The ONNX genai library doesn't work on Mac, at all, the mobile SDKs don't support it...sigh
- FieryTransition 2y agoBecause the way they are quantized takes time to get bug-free when new architectures are released. If a model was quantized with a known bug in the quantizer, then it effectively makes those quantized versions buggy and they need to be requantized with a new version of llamacpp which has this fixed.
- jerrygenser 2y agoI wonder if Ollama will or plans to have other "Supported backends" than llama.cpp. It's listed on the very last line of their readme as if the llama.cpp dependency is just incidental and a very minor detail rather than Ollama as a deployment mechanism for llama.cpp and gguf based models.
- sdesol 2y agoI don't think they will move away from llama.cpp until they are forced to. The number of people contributing to llama.cpp is quite significant [1] and it wouldn't make sense to use another backend given how quickly llama.cpp is iterating and growing. [1] https://devboard.gitsense.com/ggerganov?r=ggerganov%2Fllama.cpp&nb=true&comments=ignore&issues=ignore https://devboard.gitsense.com/ggerganov?r=ggerganov%2Fllama.... Full disclosure: This is my tool
- refulgentis 2y agoghost of christmas future The chance onnx becomes significantly relevant here went from 1% to 15% this week. They're demo'ing ~2x faster inference with Phi-3. There's been fits and starts on LLMs in ONNX for a year, but, with Wintel's AI PC™ push, and all the constituent parts in place (4 bit quants! adaptive quants!), I'd put very good money on it.
- sdesol 2y agoSo you are saying Ollama is a strong MS acquisition in the future if onnx works out.
- refulgentis 2y agono, ONNX is a Microsoft project, I don't know why people know what Ollama is and I don't think they will in a year
- sdesol 2y ago
- anotherpaulg 2y agoI actually just benchmarked Llama3 70B coding with aider, and it did quite well. It scored similar to GPT 3.5. You can use Llama3 70B with aider via Ollama [0]. It's also available for free via Groq [1] (with rate limits). And OpenRouter has it available [2] for low cost on their paid api. [0] https://aider.chat/docs/llms.html#ollama https://aider.chat/docs/llms.html#ollama [1] https://aider.chat/docs/llms.html#groq https://aider.chat/docs/llms.html#groq [2] https://aider.chat/docs/llms.html#openrouter https://aider.chat/docs/llms.html#openrouter
- typpo 2y agoPaul's benchmarks are excellent and they're the first thing I look for to get a sense of a new model performance :) For those looking to create their own benchmarks, promptfoo[0] is one way to do this locally: prompts: - "Write this in Python 3: {{ask}}" providers: - ollama:chat:llama3:8b - ollama:chat:phi3 - ollama:chat:qwen:7b tests: - vars: ask: a function to determine if a number is prime - vars: ask: a function to split a restaurant bill given individual contributions and shared items Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases. [0] https://github.com/typpo/promptfoo https://github.com/typpo/promptfoo
- jkh1 2y agoAnother benchmark: https://www.biorxiv.org/content/10.1101/2024.04.19.590278v2.full.pdf https://www.biorxiv.org/content/10.1101/2024.04.19.590278v2....
- wiktor-k 2y agoOllama is simply great! I was quite surprised how easy it is to integrate through their API. A simple chat using Ollama + llama3 is less than 40 lines of TypeScript: https://github.com/wiktor-k/llama-chat https://github.com/wiktor-k/llama-chat
- oulipo 2y agoNice! Would there be a way to do that streaming, with streaming voice input too?
- wiktor-k 2y agoStreaming is not a problem (it's just a simple flag: https://github.com/wiktor-k/llama-chat/blob/main/index.ts#L26 https://github.com/wiktor-k/llama-chat/blob/main/index.ts#L2...) but I've never used voice input. The examples show image input though: https://github.com/ollama/ollama/blob/main/docs/api.md#request-with-images https://github.com/ollama/ollama/blob/main/docs/api.md#reque... Maybe you can file an issue here: https://github.com/ollama/ollama/issues https://github.com/ollama/ollama/issues
- stephen37 2y agoI love working with Ollama, I was really surprised at how easy it is to build a simple RAG system with it. For example: https://github.com/stephen37/ollama_local_rag https://github.com/stephen37/ollama_local_rag
- addandsubtract 2y agoNice, I've been looking out for something like this! What's Jina AI and how is it local if I need an API key for it? Also, this is the first time I'm hearing about poetry. Might be worth including in the prerequisites (unless I can just stick with pip?)
- sumedh 2y ago> What's Jina AI and how is it local if I need an API key for it? Looks like its a vector DB used for creating and looking up embeddings (vectors). LLM is the second part of RAG, the first part is having a good embedding model.
- stephen37 2y agoOh yes good point, it's because at first I was using a local embedding model and then decided to use JinaAI as they have good embeddings. Jina AI is doing different things but one of them is having embeddings and I use their English/ German embeddings as in one demo I am working with German data. You can use pip as well but yes, let me add something about Poetry in case people don't know about it :)
- Sammi 2y agoIs there a copilot-like autocomplete vscode plugin that uses Ollama?
- arvinsim 2y agoDownload deepsync coder model via Ollama and connect to the model using CodeGPT plugin?
- prometheon1 2y agoYes, the continue.dev plugin can use ollama as backend