6 ms·
Ollama v0.1.45
- behnamoh 2y agoso basically the llama.cpp wrapper got updated because llama.cpp got updated…
- IAmNotACellist 2y ago[flagged]
- spzb 2y agoHard to see how it's a "ripoff" given that llama.cpp is MIT licensed. It's pretty much encouraged to re-use MIT-licensed code.
- zamalek 2y agoI mean, docker is just a ripoff of Linux namespaces with an API on top.
- IAmNotACellist 2y agoThe gulf between Docker and ollama is like the gulf between a calculator and counting on your fingers.
- wslh 2y agoYou said it first! I remember when Docker appeared and I was trying to understand what Docker offered beyond configuring the operating system until I realized there was nothing to understand! I told this because one of my companies do Microsoft Windows ad-hoc virtualization at a user space level and tried to figure out how to benchmark against Docker.
- brewtide 2y agoI've installed ollama and the open chat (?) software on a local machine. It was pretty simple to get working. Is it 'just as simple' to install llama.cpp ? If not, I would consider the ollama wrapper to have an actual use-case. You can make a working wifi temp/humidity sensor with an esp32, dht22, esphome, and home assistant. It's not hard but many many people wouldn't have a clue where to start on this. They buy the more expensive packed items. Also, a useful use case. Opening up any sort of demographic, may make it worth it. (But attributions should surely exist)
- hehdhdjehehegwv 2y agoYeah, I’m busy doing actual work - Ollama acting as a wrapper is exactly what I want.
- rvnx 2y agoBasically someone created a rather well thought .sh (and even with a nice GUI) that is super helpful. It's free to use, and free to re-use, as it is under MIT licence. Well, just thanks for their work.
- hehdhdjehehegwv 2y agoYeah and the GitHub for .cpp links directly to ollama - it’s literally advertised by the core maintainer as a good project.
- yjftsjthsd-h 2y agoWhat, if anything, is the difference between release and pre-release in this context?
- jmorgan 2y agoPre-release versions are created to test new updates on bunch of different hardware setups (OS/GPUs) before releasing more broadly (and making new versions the default for the Linux/macOS/Windows installers – those pull from the 'latest' release). There are a good number of folks that test the pre-releases as well (thank you!) especially if there's a bug fix or new feature they are waiting for. "Watch"ing the repo on GitHub will send emails/notifications of new pre-release versions
- doctorpangloss 2y agoWhy is there so much investment in ecosystems outside of Hugging Face? And why do they keep reinventing PyPi?
- cosmez 2y agoI don't think diversity in a software ecosystem is a bad idea. Nothing is stopping Hugging Face from changing their terms and screwing us over
- nickpsecurity 2y agoGoogle is great. Their slogan is “Don’t Be Evil.” We don’t need to invest in anything else. (Time passes.) “This ad-heavy, search engine is horrible. We need more solid competition!” Supplier diversity prevents or reduces situations like the above.
- ms-menardi 2y agoHuggingface isn't local, while Ollama is.
- Carrok 2y agoThis is more important than people seem to realize. Local means almost no data security issues. Not local means, well, all the data security issues.
- Der_Einzige 2y agoWhat? That's not true at all. Citation needed!
- neals 2y ago"Huggingface isn't local, while Ollama is." - ms-menardi
- Der_Einzige 2y agoIt's just wrong and a lie. Huggingface is local.
- okwhateverdude 2y agoI think this is a neat project, and use it a lot. My only complaint is the lack of grammar support. llama.cpp that they wrap will take a grammar. The dumbest patch to enable this is like two lines. And they seem to be willfully ignoring the (pretty trivial) feature for some reason. I'd rather not maintain a -but-with-grammars fork, so here we are. https://github.com/ollama/ollama/pull/4525#issuecomment-2157586947 https://github.com/ollama/ollama/pull/4525#issuecomment-2157...
- richardanaya 2y agoHey, that's my PR yah ... it's strange.
- jmorgan 2y agoSorry it's taking so long to review and for the radio silence on the PR. We have been trying to figure out how to support more structured output formats without some of the side effects of grammars. With JSON mode (which uses grammars under the hood) there were originally quite a few issue reports namely around lower performance and cases where the model would infinitely generate whitespace causing requests to hang. This is an issue with OpenAI's JSON mode as well which requires the caller to "instruct the model to produce JSON" [1]. While it's possible to handle edge cases for a single grammar such as JSON (i.e. check for 'JSON' in the prompt), it's hard to generalize this to any format. Supporting more structured output formats is definitely important. Fine-tuning for output formats is promising, and this thread [2] also has some great ideas and links. [1] https://platform.openai.com/docs/guides/text-generation/json-mode https://platform.openai.com/docs/guides/text-generation/json... [2] https://github.com/ggerganov/llama.cpp/issues/4218 https://github.com/ggerganov/llama.cpp/issues/4218
- regen7253 2y agoThank you! I've been using llama.cpp for about a year now, mostly implementing some RAG and React related papers to stay up to date. I mostly used llama.cpp, but since a few months, I started to use both Ollama and Llama.cpp. If you added grammars I wouldn't have to be running the two servers, I think you're doing an excellent job out of maintaining Ollama. Every update is like Christmas. They also don't seem to have the server as a priority (it's still literally just an example of how you'd use their C api). So, I understand your position, since their server API has been quite unstable, and the grammar validation didn't work at all until February. I also still can't get their multiple model loading to work reliably right now. Having said that, GBNF is a godsend for my daily use cases. I even prefer using phi3b with a grammar than deal with the hallucinations of a 70b without it. Fine tuning helps a lot, but can't solve the problem fully (you still need to validate the generation), and it's a lot less agile when implementing ideas. Crating some synthetic data sets is easier if you have support for grammars. I think many like me are in the same spot. Thank you for being considerate about the stability and support that it would require. But please, take a look at the current state of their grammar validation, which is pretty good right now.
- deleted 2y ago[deleted]
- v3ss0n 2y agoReminder: You don't need ollama, running llamacpp is as easy as ollama. Ollama is just a wrapper over llamacpp.
- BaculumMeumEst 2y agoI spent less than 5 seconds learning how to use ollama: I just entered "ollama run llama3" and it worked flawlessly. I spent HOURS setting up llama.cpp from reading the docs and then following this guide (after trying and failing with other guides which turned out to be obsolete): https://voorloopnul.com/blog/quantize-and-run-the-original-llama3-8b-with-llama-cpp/ https://voorloopnul.com/blog/quantize-and-run-the-original-l... Using llama.cpp, I asked the resulting model "what is 1+1", and got a neverending stream of garbage. See below. So no, it is not anywhere near as easy to get started with llama.cpp. -------------------------------- what is 1+1?") and then the next line would be ("What is 2+2?"), and so on. How can I make sure that I am getting the correct answer in each case? Here is the code that I am using: \begin{code} import random def math_problem(): num1 = random.randint(1, 10) num2 = random.randint(1, 10) problem = f"What is {num1}+{num2}? " return problem def answer_checker(user_answer): num1 = random.randint(1, 10) num2 = random.randint(1, 10) correct_answer = num1 + num2 return correct_answer == int(user_answer) def main(): print("Welcome to the math problem generator!") while True: problem = math_problem() user_answer = input(problem) if answer_checker(user_answer): print("Correct!") else: print("Incorrect. Try again!") if __name__ == "__main__": main() \end{code} My problem is that in the `answer_checker` function, I am generating new random numbers `num1` and `num2` every time I want to check if the user's answer is correct. However, this means that the `answer_checker` function is not comparing the user's answer to the correct answer of the specific problem that was asked, but rather to the correct answer of a new random problem. How can I fix this and ensure that the `answer_checker` function is comparing the user's answer to the correct answer of the specific problem that was asked? Answer: To fix this.... --------------------------------
- oaththrowaway 2y agoOllama is easier to get working on my server with a container, simple as that
- I_am_tiberius 2y agoCan you train models via Ollama or is it just used to run existing/pre-defined models?
- jmorgan 2y agoNot at the moment, although it is a highly requested feature (specifically fine-tuning). There are a few tools that you can (or will soon be able to) use to fine tune a model and then import the resulting adapter layers into Ollama: MLX [1] on macOS, Unsloth [2] on Windows and Linux [1] https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx [2] https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth
- Carrok 2y agoMy best suggestion is use axoltol to tune, convert to HF format with llama.cpp, and serve with Ollama.
- zora_goron 2y agoI see a couple comments comparing llama.cpp and Ollama, and I think both have utility for different purposes. Having used both llama.cpp (which is fantastic) and Ollama, a couple things that I find valuable about Ollama out-of-the-box -- - Automatically loading/unloading models from memory - just running the Ollama server is a relatively small footprint; every time a particular model is called it is loaded into memory, and then unloaded after 5 mins of no further usage. It makes it very convenient to spin up different models for different use-cases without having to worry about memory management or manually shutting down those tools when not in use. - OpenAI API compatibility - I run Ollama on a headless machine that has better hardware and connect via SSH port forwarding from my laptop, and with a 1 line change I can reroute any scripts on my laptop from GPT to Llama-3 (or anything else). Overall, at least for tinkering with multiple local models and building small, personal tools, I've found the utility:maintenance ratio of Ollama to be very positive -- thanks to the team for building something so valuable! :)
- benreesman 2y agoI’m pretty passionate about the space and I’ve seen good diffs from the ‘Ollama’folks, they’ve pushed the ggerganov repo forward in some ways. I’m a bit unsettled about what to me feels like an ambiguous posture on commercial activity. I’ve got no issue with folks doing open-source type stuff to make money, a lot of good code gets written that way. But I’ve learned the hard way that if a project feels kinda squirrelly it’s wise to keep one’s ears up. I still build ggerganov main from source and type out the five parameters.
- etc-hosts 2y agoyou can even immediately unload the model from memory: curl http://localhost:11434/api/generate -d '{"model": "MODELNAME", "keep_alive": 0}'
- DidYaWipe 2y agoIs what?
- forgingahead 2y agoBig kudos to the ollama team, echoing others: It just works. Fiddled with llama.cpp for ages trying to get it to run on my GPU, and ollama was setup and done in literally 3 minutes. The memory management of model loading and unloading is great, and now I can hack around and play with different LLMs from a simple API. Highly recommend that folks try it out, I thought local LLMs would be a pain to setup and use, and ollama made it super easy.
- deleted 2y ago[deleted]
- rcarmo 2y agoI hope it fixes the current bugs around environment variable handling and pointing it to a specific model directory…
- krick 2y agoIs there some llama3-uncensored or something? I'm waiting for it to appear on ollama list of models since Llama 3 was released.
- 5ytihijik 2y agohttps://effect.website/ https://effect.website/