jmorgan

11 ms·

61.
by jmorgan 3y ago
That's fast. It's exciting to see more ways to run these models locally. How does this compare to llama.cpp – both in speed and approach?
62.
by jmorgan 3y ago
I'd be interested to see how models behave at different parameter sizes or quantization levels locally with the Ollama integration. For anyone trying promptfoo's local model Ollama provider, Ollama can be found at https:/&#x
63.
Prompting Code Llama(ollama.ai)
10 points by jmorgan 3y ago|0 comments
64.
by jmorgan 3y ago
:-) that's awesome. Thanks! Nice work on this.
65.
by jmorgan 3y ago
This is very cool. Sorry if I missed it (poked around the site and your GitHub repo), but is the script available anywhere for others to run? Would love to publish results of running this against a series of ~10-20 open-source models with d
66.
by jmorgan 3y ago
Q4_0 is often mentioned as being the "tried and true" quantization level to try first. I've heard folks have had good results with 3-bit quantization (Q3_K_M) as well
67.
by jmorgan 3y ago
I work on Ollama. It's a good question since there are quite a few tools emerging in this space. The focus for Ollama is to make downloading and serving a model easy – there's an included `ollama` CLI but it's all powered by
68.
by jmorgan 3y ago
Love how simple of an interface this has. Local LLM tooling can be super daunting, but reducing it to a simple ingest() and then prompt() is really neat. By chance, have you checked out Ollama ( https://github.com/jmorganca&#
69.
Falcon 180B(falconllm.tii.ae)
4 points by jmorgan 3y ago|0 comments
70.
by jmorgan 3y ago
This is true! Although I'm also really excited at the potential speed (both for loading the model and token generation) of a 1B model for things like code completion.
71.
by jmorgan 3y ago
That's quite interesting. I hadn't thought of sparsity in the weights as a way to compress models, although this is an obvious opportunity in retrospect! I started doing some digging and found https://github.com/Sq
72.
by jmorgan 3y ago
Ah, gotcha! I thought you probably meant something else. I've been wondering this too, and it's something I've been meaning to look at. On a related note it doesn't seem like many local runners are leveraging techniques
73.
by jmorgan 3y ago
There are different levels of quantization available for different models (if that's what you mean :). E.g. here are the versions available for Llama 2: https://ollama.ai/library/llama2/tags which go down to
74.
by jmorgan 3y ago
I believe it would have also kept its stars, issues and other data.
75.
by jmorgan 3y ago
Seemed like a great project. Hope to see it come back! There are some great open-source projects in this space – not quite the same – many are focused on local LLMs like Llama2 or Code Llama which was released last week: - https:/&#x2
76.
by jmorgan 3y ago
Continue has a great guide on using the new Code Llama model launched by Facebook last week: https://continue.dev/docs/walkthroughs/codellama Continue also works with various backends and fine-tuned versions of Co
77.
by jmorgan 3y ago
Super cool. I just realized who you were from your handle and wanted say that your work on electron over the years has been absolutely amazing. Thank you for all you've done & built.
78.
by jmorgan 3y ago
Thanks for creating an issue! And sorry for the error folks… working on it!
79.
by jmorgan 3y ago
Yes you'll need the latest version (0.0.16) to run the 34B model. It should run great on that machine! The download (both as a Mac app and standalone binary) is available here: https://github.com/jmorganca/ollama&#
80.
by jmorgan 3y ago
It really is good. Surprisingly it seems to answer instruct-like prompts well! I’ve been using it with Ollama ( https://github.com/jmorganca/ollama ) with prompts like: ollama run phind-codellama "write c code t
81.
by jmorgan 3y ago
A few folks and I have been working on an open-source tool that does some of this (and hopefully more soon!) https://github.com/jmorganca/ollama There's a "PrivateGPT" example in there that is similar to
82.
by jmorgan 3y ago
Indeed! After pulling a model with "ollama pull codellama" you can access it via the REST API: curl -X POST http://localhost:11434/api/generate -d '{ "model": &quo
83.
by jmorgan 3y ago
Sorry, this should be fixed now! To update you'll have to run: ollama pull codellama:7b-instruct
84.
by jmorgan 3y ago
> managed to get infinite streams of near nonsense This should be fixed now! To update you'll have to run: ollama pull codellama:7b-instruct
85.
Run Code Llama Locally(ollama.ai)
12 points by jmorgan 3y ago|4 comments
86.
by jmorgan 3y ago
To run Code Llama locally, the 7B parameter quantized version can be downloaded and run with the open-source tool Ollama: https://github.com/jmorganca/ollama ollama run codellama "write a python function to a
87.
by jmorgan 3y ago
I'm curious about this too. Lately I've been building an open source tool to help bring make pulling + running models easier locally – https://github.com/jmorganca/ollama – right now we work with the awesome
88.
WizardMath solves math problems with Llama 2(ollama.ai)
1 points by jmorgan 3y ago|0 comments
89.
by jmorgan 3y ago
This project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes. A few folks and I have been building a
90.
by jmorgan 3y ago
One benefit of self-hosting LLMs is the wide range of fine-tuned models available, including uncensored models. A popular one over the last weeks was Llama 2 Uncensored by George Sung: https://ollama.ai/blog/run-llama2-

More ›