7 ms·

Business opportunity: I'd pay money for NICE desktop software that can run all these different models (non-subscription, "2-year updates included, then discount

Business opportunity: I'd pay money for NICE desktop software that can run all these different models (non-subscription, "2-year updates included, then discount pricing" modal perhaps). My wishlist: - Easy plug & play model installation, and trivial to change which model once installed. - Runs a local web server, so I can interact with it via any browser - Ability to feed a model a document or multiple documents and be able to ask questions about them (or build a database of some kind?). - Absolute privacy guarantees. Nothing goes off-machine from my prompt/responses (USP over existing cloud/online ones). Routine license/update checks are fine though. I'm not trying to throw shade at the existing ways to running LLMs locally, just saying there may be room for an OPTIONAL commercial piece of software in this space. Most of them are designed for academics to do academic things. I am talking about a turn-key piece of software for everyone else that can give you an "almost" ChatGPT or "almost" CoPilot-like experience for a one time fee that you can feed sensitive private information to.

jmorgan 3y ago
A few folks and I have been working on an open-source tool that does some of this (and hopefully more soon!) https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama There's a "PrivateGPT" example in there that is similar to your third point above: https://github.com/jmorganca/ollama/tree/main/examples/privategpt https://github.com/jmorganca/ollama/tree/main/examples/priva... Would love to know your thoughts
- SubiculumCode 3y ago
  [flagged]
- luma 3y ago
  I'd love to test this out as soon as you get Linux or Windows support going!
  - appel 3y ago
    Me too! I starred the repo and am watching releases, excited to try it.
julianeon 3y ago
What I want is even simpler: just an API that you make requests to and receive answers back. Surprisingly hard to find, outside OpenAI that is.
- BoorishBears 3y ago
  I'm not into open source LLMs in the slightest, and yet even I've trivially found tools to do what both you and the poster above you wanted lmstudio actually does what both of you want: provides an easy GUI and serves up your model over a local endpoint that mirrors the OpenAI API. There's just too much noise in terms of the tooling for LLMs, the solution is fewer higher quality solutions, not more solutions
- ingridpan 3y ago
  https://gradient.ai/ https://gradient.ai/ is doing that with llama2
  - worldsayshi 3y ago
    Looks really promising. I wonder if the similar pricing to OpenAI means that Gradient is also(?) bleeding money even if they get a good customer base. Or are these prices sustainable over time?
    - ingridpan 3y ago
      Good question, esp as Gradient fine-tuning is so much cheaper than Open AI's
      - worldsayshi 3y ago
        Yeah it's even cheaper. Although it looks like it's about the same in proportion to approx model size/expected quality? They haven't launched any >13B model yet, although they plan to.
- MaKey 3y ago
  Oobabooga exposes an API.
noduerme 3y ago
Agreed. After several rounds of setting up various python environments and tinkering with directory structures and debugging glitches and quantizing models just to end up playing around for a few minutes and getting bored, it would be nice to have the experience just be seamless. I wouldn't try to set up a workflow around seriously using what's out there to run on localhost now. That said, non-subscription is essential, and that's probably going to be a heavy lift considering how quickly things are evolving.
- simonw 3y ago
  I've been trying to push things in that direction with my LLM tool - the idea is to have Python plugins which you can install that do all of the irritating details to get a model setup. I've not yet been able to solve the challenge of needing CUDA etc for some models though! Plugins so far: https://llm.datasette.io/en/stable/plugins/directory.html https://llm.datasette.io/en/stable/plugins/directory.html
  - noduerme 3y ago
    Cool! I've followed your instructions and your blog quite a bit as I've experimented with running local LLMs as well as stable diffusion. It's been especially helpful, as python is not my language or usual environment. Your patience at hacking your way through each new iteration and presenting what's important about them is astonishing; I personally think I'd have gone mad, but you've done great work in charting the territory.
tibbon 3y ago
I’ve used llama.cpp easily for some local things, but it does lack a good ui.
MaKey 3y ago
Did you try Oobabooga (https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui) yet?
- max51 3y ago
  Oobabooga is a great tool but it still has a long way to go in term of user-friendliness. It's absolutely not plug and play the way that chatgpt is; It requires research, trial and error, and knowledge of the tech to make the model work to its full potential. It's great once you finish setting it up, but it does not compare to what you would expect from a commercial product aimed at normal end-users. Things like bad default values, no tooltips, an no curated model list to one-click download is what separates a tool like Oobabooga from a paid commercial product. These things require time/money and it would be very unlikely that an open source tool could find resources for all the testing and R&D. I think there is a big market for products where you pay and can just start chatting with the model without having to ever go to the settings tab or google anything unless you need to do something out of the ordinary.
alsobrsp 3y ago
I have been using refact.ai on my laptop, it has been quite good. https://github.com/smallcloudai/refact/blob/main/README.md https://github.com/smallcloudai/refact/blob/main/README.md
- thewataccount 3y ago
  Refact has worked for me. Hopefully exllama will support CodeLlama.
- firecall 3y ago
  I wish these things worked with anything other than VSCode or JerBrains tools! VSCode is such a bloated hog of an editor! Every time I open VSCode it’s bugging with badges to update extensions… and it’s so slow!
  - kateklink 3y ago
    having more plugin support is in our plans for sure. We're also open for contributions.
irrational 3y ago
I work for a Fortune 100 company with 80,000+ employees. All of us are explicitly forbidden from using any sort of AI/LLM tool without written permission from the head of legal AND the CEO. In other words, nobody is going to get permission. The concerns are 2 fold - 1. We might inadvertently use someone else’s intellectual property. 2. Someone else might gain access to our intellectual property. What you are describing would help alleviate the concern about issue 2, but I’m not sure if it would help alleviate the concerns with issue 1.
- roguas 3y ago
  Change company. Honestly. If you go as far as to forbid your partners in crime (workers sigh..) to explore new uncharted territory at all - well ya know someone will/might just win by not doing that.
  - sshine 3y ago
    Working for an 80.000+ employee company, one has already accepted a certain degree of inertia.
  - flatline 3y ago
    This is particular, specifically problematic territory. I cannot imagine handing over proprietary data to a third party without a contract in place for how that data is stored and used. It’s not about innovation, it’s about using someone else’s tools without ownership. For the other case, it’s both about integrity in owning your own work, and a shield from legal consequences. These things should be very relevant to any business. I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy. I am more bullish on LLMs than most of my coworkers. I think there is a lot of potential there. I do not see that potential in the current commercial offerings, and the financial outlay to fine-tune an open-source model and run it at scale is…prohibitive.
    - haldujai 3y ago
      > I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy. Really? I'm academic now but I find Copilot at least moderately helpful when I'm writing a library. It's pretty good a lot of boilerplate functions, docstrings, regex, etc. I certainly don't want to go back to not using it, my code is a lot closer to production quality now and looks nicer. Thinking back to my days in back-end it seems like it would have been very helpful/sped things up so I'm surprised to hear it's just a toy but I've been out of the professional game for a while now. What's the main criticism?
ssalka 3y ago
GPT4All satisfies these requirements, except for (AFAIK) running a web server https://gpt4all.io/index.html https://gpt4all.io/index.html
- simonw 3y ago
  It runs a web server too - if you start up the desktop app it can run a web server with an API on a port for you.
jupp0r 3y ago
You want a polished product and something that can run all sorts of different models. I don't know that both are possible at the same time.
biwills 3y ago
We are doing this with https://Faraday.dev https://Faraday.dev - would love to chat more - my email is on my profile!
deleted 3y ago
[deleted]
kateklink 3y ago
we're going in this direction for code models with Refact https://github.com/smallcloudai/refact/ https://github.com/smallcloudai/refact/ - right now you self-host code models, fine-tune them on local files, get the model running locally inside your IDE

jmorgan 3y ago

A few folks and I have been working on an open-source tool that does some of this (and hopefully more soon!) https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama There's a "PrivateGPT" example in there that is similar to your third point above: https://github.com/jmorganca/ollama/tree/main/examples/privategpt https://github.com/jmorganca/ollama/tree/main/examples/priva... Would love to know your thoughts

SubiculumCode 3y ago
[flagged]
luma 3y ago
I'd love to test this out as soon as you get Linux or Windows support going!
- appel 3y ago
  Me too! I starred the repo and am watching releases, excited to try it.

julianeon 3y ago

What I want is even simpler: just an API that you make requests to and receive answers back. Surprisingly hard to find, outside OpenAI that is.

BoorishBears 3y ago
I'm not into open source LLMs in the slightest, and yet even I've trivially found tools to do what both you and the poster above you wanted lmstudio actually does what both of you want: provides an easy GUI and serves up your model over a local endpoint that mirrors the OpenAI API. There's just too much noise in terms of the tooling for LLMs, the solution is fewer higher quality solutions, not more solutions
ingridpan 3y ago
https://gradient.ai/ https://gradient.ai/ is doing that with llama2
- worldsayshi 3y ago
  Looks really promising. I wonder if the similar pricing to OpenAI means that Gradient is also(?) bleeding money even if they get a good customer base. Or are these prices sustainable over time?
  - ingridpan 3y ago
    Good question, esp as Gradient fine-tuning is so much cheaper than Open AI's
    - worldsayshi 3y ago
      Yeah it's even cheaper. Although it looks like it's about the same in proportion to approx model size/expected quality? They haven't launched any >13B model yet, although they plan to.
MaKey 3y ago
Oobabooga exposes an API.

noduerme 3y ago

Agreed. After several rounds of setting up various python environments and tinkering with directory structures and debugging glitches and quantizing models just to end up playing around for a few minutes and getting bored, it would be nice to have the experience just be seamless. I wouldn't try to set up a workflow around seriously using what's out there to run on localhost now. That said, non-subscription is essential, and that's probably going to be a heavy lift considering how quickly things are evolving.

simonw 3y ago
I've been trying to push things in that direction with my LLM tool - the idea is to have Python plugins which you can install that do all of the irritating details to get a model setup. I've not yet been able to solve the challenge of needing CUDA etc for some models though! Plugins so far: https://llm.datasette.io/en/stable/plugins/directory.html https://llm.datasette.io/en/stable/plugins/directory.html
- noduerme 3y ago
  Cool! I've followed your instructions and your blog quite a bit as I've experimented with running local LLMs as well as stable diffusion. It's been especially helpful, as python is not my language or usual environment. Your patience at hacking your way through each new iteration and presenting what's important about them is astonishing; I personally think I'd have gone mad, but you've done great work in charting the territory.

tibbon 3y ago

I’ve used llama.cpp easily for some local things, but it does lack a good ui.

MaKey 3y ago

Did you try Oobabooga (https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui) yet?

max51 3y ago
Oobabooga is a great tool but it still has a long way to go in term of user-friendliness. It's absolutely not plug and play the way that chatgpt is; It requires research, trial and error, and knowledge of the tech to make the model work to its full potential. It's great once you finish setting it up, but it does not compare to what you would expect from a commercial product aimed at normal end-users. Things like bad default values, no tooltips, an no curated model list to one-click download is what separates a tool like Oobabooga from a paid commercial product. These things require time/money and it would be very unlikely that an open source tool could find resources for all the testing and R&D. I think there is a big market for products where you pay and can just start chatting with the model without having to ever go to the settings tab or google anything unless you need to do something out of the ordinary.

alsobrsp 3y ago

I have been using refact.ai on my laptop, it has been quite good. https://github.com/smallcloudai/refact/blob/main/README.md https://github.com/smallcloudai/refact/blob/main/README.md

thewataccount 3y ago
Refact has worked for me. Hopefully exllama will support CodeLlama.
firecall 3y ago
I wish these things worked with anything other than VSCode or JerBrains tools! VSCode is such a bloated hog of an editor! Every time I open VSCode it’s bugging with badges to update extensions… and it’s so slow!
- kateklink 3y ago
  having more plugin support is in our plans for sure. We're also open for contributions.

irrational 3y ago

I work for a Fortune 100 company with 80,000+ employees. All of us are explicitly forbidden from using any sort of AI/LLM tool without written permission from the head of legal AND the CEO. In other words, nobody is going to get permission. The concerns are 2 fold - 1. We might inadvertently use someone else’s intellectual property. 2. Someone else might gain access to our intellectual property. What you are describing would help alleviate the concern about issue 2, but I’m not sure if it would help alleviate the concerns with issue 1.

roguas 3y ago
Change company. Honestly. If you go as far as to forbid your partners in crime (workers sigh..) to explore new uncharted territory at all - well ya know someone will/might just win by not doing that.
- sshine 3y ago
  Working for an 80.000+ employee company, one has already accepted a certain degree of inertia.
- flatline 3y ago
  This is particular, specifically problematic territory. I cannot imagine handing over proprietary data to a third party without a contract in place for how that data is stored and used. It’s not about innovation, it’s about using someone else’s tools without ownership. For the other case, it’s both about integrity in owning your own work, and a shield from legal consequences. These things should be very relevant to any business. I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy. I am more bullish on LLMs than most of my coworkers. I think there is a lot of potential there. I do not see that potential in the current commercial offerings, and the financial outlay to fine-tune an open-source model and run it at scale is…prohibitive.
  - haldujai 3y ago
    > I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy. Really? I'm academic now but I find Copilot at least moderately helpful when I'm writing a library. It's pretty good a lot of boilerplate functions, docstrings, regex, etc. I certainly don't want to go back to not using it, my code is a lot closer to production quality now and looks nicer. Thinking back to my days in back-end it seems like it would have been very helpful/sped things up so I'm surprised to hear it's just a toy but I've been out of the professional game for a while now. What's the main criticism?

ssalka 3y ago

GPT4All satisfies these requirements, except for (AFAIK) running a web server https://gpt4all.io/index.html https://gpt4all.io/index.html

simonw 3y ago
It runs a web server too - if you start up the desktop app it can run a web server with an API on a port for you.

jupp0r 3y ago

You want a polished product and something that can run all sorts of different models. I don't know that both are possible at the same time.

biwills 3y ago

We are doing this with https://Faraday.dev https://Faraday.dev - would love to chat more - my email is on my profile!

deleted 3y ago

[deleted]

kateklink 3y ago

we're going in this direction for code models with Refact https://github.com/smallcloudai/refact/ https://github.com/smallcloudai/refact/ - right now you self-host code models, fine-tune them on local files, get the model running locally inside your IDE