11 ms·
Ollama is now available as an official Docker image
- rovr138 3y agoFrom, https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama > Get up and running with large language models locally. > To run and chat with Llama 2: ollama run llama2
- deleted 3y ago[deleted]
- kevinlinxc 3y agoI havent touched llm stuff because the barrier to entry felt pretty high. This has reduced the barrier so much if it works as promised.
- thatxliner 3y agoIt does.
- yieldcrv 3y agowhy is downloading a docker image and using docker from the command line less of a deterrent for you than downloading a model and using it from the command line?
- lopkeny12ko 3y agoBecause it's impossible to get started as someone who has no background in ML. I tried earlier this year and also failed. 1. Obtain the Llama models. Apparently you have to sign up for access and get a download link? I don't want to do that, found some public download links instead. Ok, now I have a few hundred GBs of model files. 2. Compile llama.cpp. Missing dependencies, took an hour to figure out how to resolve. 3. Quantize the model? What does that even mean? 4. Install pytorch. Run the command in the README. Python exception. 5. Install NVIDIA helper libraries. Doesn't work. Try installing the AMD helpers to run on CPU instead. No instructions for how to do this. Eventually figured it out. 6. Try running pytorch again. Same exception. I gave up after a full day of trying to make this work. The Docker image is for people like me.
- yieldcrv 3y agogot it thanks for the explanation! helps the command line actually work
- imp0cat 3y agoYeah it's great, you can get this running in no time (even without gpu, just to play with it): docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama docker exec -it ollama ollama run llama2 And that's all!
- stefandesu 3y agoIt's nice to have this possibility on Linux now. It wasn't much more complicated on macOS before: brew install ollama brew services start ollama ollama run llama2 Works great on my M1 MacBook Air, although it's definitely not as good as ChatGPT.
- politelemon 3y agoAnd with GPU just do this for the first command docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama You'll need to have the NVidia container toolkit installed.
- ericskiff 3y agoI also highly recommend trying LMStudio. It’s a nice interface for downloading, using, and holding LLMs locally
- dr_kiszonka 3y agoI was wondering if you have come across any models that can answer questions based on information from a single source and that source alone (a website or document)? I asked GPT 4, Claude 1 & 2, and Bard questions about a single documentation page and they all gave both incorrect and hallucinated answers. For example, I asked them to list all functions that accepted integers based on provided documentation. Their responses included functions that didn't accept integers and functions never mentioned in the documentation.
- thebruce87m 3y ago> H2O LLM Studio requires a machine with Ubuntu 16.04+ and at least one recent Nvidia GPU with Nvidia drivers version >= 470.57.02. For larger models, we recommend at least 24GB of GPU memory.
- ilteris 3y agoyes but it's closed source, no?
- backflippinbozo 3y agoThanks, ollama
- vanillax 3y ago[flagged]
- devblogs 3y ago[dead]
- nf17 3y agowhy the downvotes?
- subjectsigma 3y agoHe just explained that he’s using open source tools to take other people’s hard work and re-host it, using an account he set up seemingly just to promote this service. Looking at the website itself I don’t think it’s particularly offensive but I could see why people might not like it.
- rafram 3y agoIt doesn’t look like anything is being rehosted. The links go straight to the original blogs.
- subjectsigma 3y agoAn AI summary of the article by definition contains information from the article; otherwise it would not be a summary. Whether or not you consider this rehosting is muddy but some people might, which is all my comment was saying.
- vanillax 3y agowhat about k8s?
- shaburn 3y agoThis will unlock a ton of Enterprise use cases. I have met with a number of enterprises that have a hard no on sending much of their data into OpenAI, even on Azure. I think this will persist as the brand faces many hurdles with respect for IP, in public perception.
- nottheengineer 3y agoYes, we are currently doing something with T5 for those exact reasons. It could probably benefit a lot from throwing a bigger model at it.
- deleted 3y ago[deleted]
- davidkunz 3y agoOllama is great, I recently integrated it into my text editor (Neovim), now I have local AI assistance when coding/writing text. https://youtu.be/FIZt7MinpMY?si=fzcmKq5cw2bniH26 https://youtu.be/FIZt7MinpMY?si=fzcmKq5cw2bniH26
- davidkunz 3y agoHere's the plugin: https://github.com/David-Kunz/gen.nvim https://github.com/David-Kunz/gen.nvim
- kramerger 3y agoI'm on phone and can't test this right now. What are the system requirements for getting a decent model running in background? Is it something I can run on a laptop with 8-12GB of RAM and not a huge harddrive?
- keonix 3y agoMistral 7B ~ 8 GiB StableLM 3B ~4 GiB You could go even lower with smaller quantization if necessary. I personally wouldn't use anything smaller than 7B and Mistral already pushing it in coherence. Overall it depends on your use case, not everyone needs smart models, or large context that sometimes takes half of required memory. Codellama is also surprisingly good even for non-coding tasks
- ssss11 3y agoWhat is the most simple docker-compose.yml needed to get this running? (No I don’t want to run that command they show I want to use docker compose and am not an expert yet) - thanks
- westernpopular 3y agoYou can use https://www.composerize.com/ https://www.composerize.com/ to see what a `docker run` command would look like as a docker-compose file
- ssss11 3y agoWow I didn’t know! Thanks
- kwertyops 3y agoversion: '3' services: ollama: image: ollama/ollama container_name: ollama ports: - "11434:11434" volumes: - ollama-volume:/root/.ollama
- ssss11 3y agoPerfect thank you
- remmargorp64 3y agoDoes the docker container run on Windows though? I still don't see any official documentation showing how to get Ollama running on Windows?
- sakopov 3y agoI'm running it on windows using this docker image without any issues.
- alexchantavy 3y agoUse Ubuntu on WSL - for me it makes using Windows as a dev machine actually useable
- deleted 3y ago[deleted]
- dvngnt_ 3y agodoes this mean it works on windows too?