6 ms·

Show HN: OpenCopilot – Build and embed open-source AI copilots into your product

109 points by JohannesTk 3y ago

Hey HN OpenCopilot is an OSS framework which helps devs to build open-source AI Copilots that actually work and embed into their product with ease. Why another LLM framework? Twitter is full of impressive LLM applications but once you peel off the curtains it’s clear that they are just demos. The reason being because building an AI Copilot that goes beyond a Twitter demo can be complex, time-consuming and unreliable. Our team has been in the AI space since 2018 and built numerous LLM apps & copilots. While doing that, we got approached by many startups saying they’d also like to build a copilot for their product but they haven’t been able to get it reliable, fast or cost-effective enough for production use. Thus we built OpenCopilot framework, so devs can intuitively get AI Copilots running in less than 10 minutes and iterate towards a useful Copilot in a single day. We believe every product, company and individual will have their Copilot in the future. Thus, we’d love your feedback, questions and constructive criticism.

decodebytes 3y ago
Which LLM are you using?
- MourningWood 3y ago
  Looks like gpt4
- michaelmior 3y ago
  For the time being, it looks like you can choose between gpt-3.5-turbo-16k and gpt-4. https://github.com/opencopilotdev/opencopilot/blob/d7aa8270db24641723e49d3575f0dc8e94e7ec3e/opencopilot/settings.py#L27C21-L27C48 https://github.com/opencopilotdev/opencopilot/blob/d7aa8270d...
- sq42na 3y ago
  Yep, currently gpt-3.5-16k or gpt-4. We wrote the example prompts in a relatively Llama-compatible way though (we actually started building this onto Llama 1 before switching to OpenAI as default), and make few assumptions about the LLM so it's easy to switch out. Mostly this is waiting behind us adding an option to pass in any LLM, and we're planning to add support for this. Generally, we leave the LLM up to the user -- if OpenAI or Google is a no-go, then you probably are anyway in the territory of self-hosting or even self-training your LLM, which means you're fine setting up your own inference endpoints as well.
neilv 3y ago
What is the relationship to: OpenCopilot – Open source AI copilot for your own SaaS product (github.com/openchatai) | 127 points by gharbat 1 day ago | 31 comments | https://news.ycombinator.com/item?id=37203196 https://news.ycombinator.com/item?id=37203196
- JohannesTk 3y ago
  None, we've been building OpenCopilot for the last few months and it seems they beat us with the launch by a few days. I do hope they do well cus we believe a world where copilots can be built with open-source is a much better one vs built on a closed MS, etc. stack.
  - CharlesW 3y ago
    Related question: Why would you (and they) give your product a Microsoft name and open yourself up to a trademark fight and probable name change when, with just a little effort and imagination (and maybe ChatGPT), you could've called it anything else?
    - willsmith72 3y ago
      Why a "Microsoft name"? Are you talking about GitHub copilot? Because that's far from the first "copilot" product
bko 3y ago
I'm confused as to what this is? Is it just a wrapper to gpt API that parses the response? And it looks like it manages the convo so you don't have to send over entire history? I have a somewhat related project that wraps the gpt API and let's you interact in a codebase, where the LLM can request access to certain files, propose new files and edits https://github.com/breeko/j-dev https://github.com/breeko/j-dev
- nonethewiser 3y ago
  Sounds a lot like aider
skeptrune 3y ago
What is the point of this if no RAG?
- sq42na 3y ago
  RAG comes out of the box - see here: https://docs.opencopilot.dev/improve/knowledge-base https://docs.opencopilot.dev/improve/knowledge-base We were hoping to make dynamic RAG part of the very first release but didn't - we'll be adding that soon.
swyx 3y ago
> Our team has been in the AI space since 2018 and built numerous LLM apps & copilots. could you please elaborate?
- sq42na 3y ago
  Sure! Taivo here - CTO & co-founder of OpenCopilot. The most recent LLM app we've built has been the Ready Player Me copilot (which you can see linked in the main repo as well, and which was an inspiration for actually making something open-source based on what we had learned into). The problem it solves is onboarding developers onto their platform, and is currently live for a subset of their users. We've built a ton of copilots that we haven't published. Some have been for work like founder copilots for ourselves and a few friends, sales outreach copilots, etc. Some have been for fun: I wanted to see if I could create a copilot that coaches me to do things faster a la https://patrickcollison.com/fast https://patrickcollison.com/fast - so I did. As a PoC I created a copilot that explains the rules of an obscure sport I like (floorball). In addition, with the same team, since the beginning of this year we've made a bunch of experiments that I'm sure many others have tried: * "chat with your knowledge base" style bots: for tech support, for developer docs, for internal HR use, and probably some I can't remember * an LLM-based quiz app that dynamically makes a quiz based on your Spotify history * an automated engineer (a much more basic version of smol-ai/developer) * a website builder where you mostly uses a chat UI for making the website * making AutoGPT better (higher quality, faster, more straightforward to use) * we also briefly considered making an OSS LLM before Llama 1 came out and kickstarted the OSS LLM wild ride we've seen since then We killed almost all of these experiments within a few weeks after release, because none found early traction; we'd probably been wrong about the problem these were solving. About my AI background: I built a 25-person computer vision team from zero at an identity verification unicorn. After that tried and failed to build a computer vision labelling tool startup. Before that, built robot perception software at Starship robots and unsuccessfully tried to make a contribution to AI safety with my thesis on active reinforcement learning in 2017. About Johannes (OP) & the rest of the team: in 2018 him and part of the current team tried to make Sidekik: something like character.ai, but training models from scratch - which eventually didn't work out. From 2020 (after a pivot) the same team was building Sentinel, a deepfake-detection startup - which again did not work out. FYI: Sidekik's original website: https://web.archive.org/web/20181008082142/https://sidekik.ai/ https://web.archive.org/web/20181008082142/https://sidekik.a...
zelphirkalt 3y ago
I would still like to have a model, open source at minimum, better yet with AGPL or so, that I train locally on my own code and also use locally exclusively on my own machine and that does not require me to have a Nvidia GPU and can simply be trained on CPU. Does this exist?
- steno132 3y ago
  There are significant safety risks with this approach. Current foundation AI models have safeguards in place to avoid generating content that could be used for disinformation, violence, and terrorism. While not perfect, these safeguards have meant that we've been able to use ChatGPT over the last couple months, without a major backlash due to the technology being used for destructive purposes and it potentially being closed (as has happened many times before).
  - Xeoncross 3y ago
    Considering governments murdered literally a million times more people than civilizations over the past hundred years, it's ironic they alone get the models without safeguards in place.
    - steno132 3y ago
      The fundamental disconnect is that we as technologists believe we are entitled to technology. That technology should be open. And, for the most part, it is. However, if you look at the most sensitive technologies, whether it's nuclear weapons or until recently cryptography, they're never open. The risk of them falling into the wrong hands is too high. AI seems to us like any other software, as people who may understand it and its limitations. However, it's also a tool for unfathomable destruction -- not now, but in the coming years. And so restrictions are almost certain to happen, and according to many, justified. You're not going to have access to "un-censored AI" in the coming years. That's unfortunately reality.
      - Xeoncross 3y ago
squeaky-clean 3y ago
I wouldn't really consider a wrapper around the OpenAI API to be an open source copilot. Am I misunderstanding this project?
- geek_at 3y ago
  True, GitHub copilot is also using gpt3.5 at the moment so it's actually kind of the same
- bofadeez 3y ago
  You understand it fully
- JohannesTk 3y ago
  You do have a valid point that the LLM is closed aka OpenAI's API. We were thorn about it ourselves but decided to release with OpenAI because of it's ease of use for devs. We have already tested Llama-2 and seeing more and more pull for it, especially by enterprise who have can't use OpenAI because of privacy and IP issues. Thus, OSS LLMs are next on our roadmap :) Rest of the stack is fully open-source and it's much more than a wrapper. Maybe the impression comes from the aspect that we've made getting started dead simple with the minimal Copilot example. The framework covers a lot of ground so you can build your copilot easily: backend, adding a knowledge base & dynamic data, front-end, monitoring & evaluation. This is pretty much v0.1 release. What else would you love to see?
warthog 3y ago
Any plans to add Langchain support? Another thing for OSS LLMs could be adding support for the like of Replicate or Baseten as well.
- sq42na 3y ago
  We actually use Langchain under the hood for some things, and are in the process of figuring out which abstractions make sense to put on the SDK interface. For example, we already use the `langchain.schema.Document` class as the main document dataclass. With making it possible to bring any LLM, we're considering adding support for bringing anything that implements the Langchain LLM interface. In general, we're still trying to figure out how much customizability to give to the user and how (Langchain is a great option here) -- power should not come at the expense of ease of use.
boredemployee 3y ago
The amount of cynical comments everywhere always get me. Anyway, congrats, it will be really useful for building POCs and mvps quickly. I'll definetely give it a try and feedback once I test it!
- JohannesTk 3y ago
  Thanks for the support! :) Would love to hear your feedback on what's good, what sucks and what to improve.

109 points by JohannesTk 3y ago

decodebytes 3y ago

Which LLM are you using?

MourningWood 3y ago
Looks like gpt4
michaelmior 3y ago
For the time being, it looks like you can choose between gpt-3.5-turbo-16k and gpt-4. https://github.com/opencopilotdev/opencopilot/blob/d7aa8270db24641723e49d3575f0dc8e94e7ec3e/opencopilot/settings.py#L27C21-L27C48 https://github.com/opencopilotdev/opencopilot/blob/d7aa8270d...
sq42na 3y ago
Yep, currently gpt-3.5-16k or gpt-4. We wrote the example prompts in a relatively Llama-compatible way though (we actually started building this onto Llama 1 before switching to OpenAI as default), and make few assumptions about the LLM so it's easy to switch out. Mostly this is waiting behind us adding an option to pass in any LLM, and we're planning to add support for this. Generally, we leave the LLM up to the user -- if OpenAI or Google is a no-go, then you probably are anyway in the territory of self-hosting or even self-training your LLM, which means you're fine setting up your own inference endpoints as well.

neilv 3y ago

What is the relationship to: OpenCopilot – Open source AI copilot for your own SaaS product (github.com/openchatai) | 127 points by gharbat 1 day ago | 31 comments | https://news.ycombinator.com/item?id=37203196 https://news.ycombinator.com/item?id=37203196

JohannesTk 3y ago
None, we've been building OpenCopilot for the last few months and it seems they beat us with the launch by a few days. I do hope they do well cus we believe a world where copilots can be built with open-source is a much better one vs built on a closed MS, etc. stack.
- CharlesW 3y ago
  Related question: Why would you (and they) give your product a Microsoft name and open yourself up to a trademark fight and probable name change when, with just a little effort and imagination (and maybe ChatGPT), you could've called it anything else?
  - willsmith72 3y ago
    Why a "Microsoft name"? Are you talking about GitHub copilot? Because that's far from the first "copilot" product

bko 3y ago

I'm confused as to what this is? Is it just a wrapper to gpt API that parses the response? And it looks like it manages the convo so you don't have to send over entire history? I have a somewhat related project that wraps the gpt API and let's you interact in a codebase, where the LLM can request access to certain files, propose new files and edits https://github.com/breeko/j-dev https://github.com/breeko/j-dev

nonethewiser 3y ago
Sounds a lot like aider

skeptrune 3y ago

What is the point of this if no RAG?

sq42na 3y ago
RAG comes out of the box - see here: https://docs.opencopilot.dev/improve/knowledge-base https://docs.opencopilot.dev/improve/knowledge-base We were hoping to make dynamic RAG part of the very first release but didn't - we'll be adding that soon.

swyx 3y ago

> Our team has been in the AI space since 2018 and built numerous LLM apps & copilots. could you please elaborate?

sq42na 3y ago
Sure! Taivo here - CTO & co-founder of OpenCopilot. The most recent LLM app we've built has been the Ready Player Me copilot (which you can see linked in the main repo as well, and which was an inspiration for actually making something open-source based on what we had learned into). The problem it solves is onboarding developers onto their platform, and is currently live for a subset of their users. We've built a ton of copilots that we haven't published. Some have been for work like founder copilots for ourselves and a few friends, sales outreach copilots, etc. Some have been for fun: I wanted to see if I could create a copilot that coaches me to do things faster a la https://patrickcollison.com/fast https://patrickcollison.com/fast - so I did. As a PoC I created a copilot that explains the rules of an obscure sport I like (floorball). In addition, with the same team, since the beginning of this year we've made a bunch of experiments that I'm sure many others have tried: * "chat with your knowledge base" style bots: for tech support, for developer docs, for internal HR use, and probably some I can't remember * an LLM-based quiz app that dynamically makes a quiz based on your Spotify history * an automated engineer (a much more basic version of smol-ai/developer) * a website builder where you mostly uses a chat UI for making the website * making AutoGPT better (higher quality, faster, more straightforward to use) * we also briefly considered making an OSS LLM before Llama 1 came out and kickstarted the OSS LLM wild ride we've seen since then We killed almost all of these experiments within a few weeks after release, because none found early traction; we'd probably been wrong about the problem these were solving. About my AI background: I built a 25-person computer vision team from zero at an identity verification unicorn. After that tried and failed to build a computer vision labelling tool startup. Before that, built robot perception software at Starship robots and unsuccessfully tried to make a contribution to AI safety with my thesis on active reinforcement learning in 2017. About Johannes (OP) & the rest of the team: in 2018 him and part of the current team tried to make Sidekik: something like character.ai, but training models from scratch - which eventually didn't work out. From 2020 (after a pivot) the same team was building Sentinel, a deepfake-detection startup - which again did not work out. FYI: Sidekik's original website: https://web.archive.org/web/20181008082142/https://sidekik.ai/ https://web.archive.org/web/20181008082142/https://sidekik.a...

zelphirkalt 3y ago

I would still like to have a model, open source at minimum, better yet with AGPL or so, that I train locally on my own code and also use locally exclusively on my own machine and that does not require me to have a Nvidia GPU and can simply be trained on CPU. Does this exist?

steno132 3y ago
There are significant safety risks with this approach. Current foundation AI models have safeguards in place to avoid generating content that could be used for disinformation, violence, and terrorism. While not perfect, these safeguards have meant that we've been able to use ChatGPT over the last couple months, without a major backlash due to the technology being used for destructive purposes and it potentially being closed (as has happened many times before).
- Xeoncross 3y ago
  Considering governments murdered literally a million times more people than civilizations over the past hundred years, it's ironic they alone get the models without safeguards in place.
  - steno132 3y ago
    The fundamental disconnect is that we as technologists believe we are entitled to technology. That technology should be open. And, for the most part, it is. However, if you look at the most sensitive technologies, whether it's nuclear weapons or until recently cryptography, they're never open. The risk of them falling into the wrong hands is too high. AI seems to us like any other software, as people who may understand it and its limitations. However, it's also a tool for unfathomable destruction -- not now, but in the coming years. And so restrictions are almost certain to happen, and according to many, justified. You're not going to have access to "un-censored AI" in the coming years. That's unfortunately reality.
    - Xeoncross 3y ago

squeaky-clean 3y ago

I wouldn't really consider a wrapper around the OpenAI API to be an open source copilot. Am I misunderstanding this project?

geek_at 3y ago
True, GitHub copilot is also using gpt3.5 at the moment so it's actually kind of the same
bofadeez 3y ago
You understand it fully
JohannesTk 3y ago
You do have a valid point that the LLM is closed aka OpenAI's API. We were thorn about it ourselves but decided to release with OpenAI because of it's ease of use for devs. We have already tested Llama-2 and seeing more and more pull for it, especially by enterprise who have can't use OpenAI because of privacy and IP issues. Thus, OSS LLMs are next on our roadmap :) Rest of the stack is fully open-source and it's much more than a wrapper. Maybe the impression comes from the aspect that we've made getting started dead simple with the minimal Copilot example. The framework covers a lot of ground so you can build your copilot easily: backend, adding a knowledge base & dynamic data, front-end, monitoring & evaluation. This is pretty much v0.1 release. What else would you love to see?

warthog 3y ago

Any plans to add Langchain support? Another thing for OSS LLMs could be adding support for the like of Replicate or Baseten as well.

sq42na 3y ago
We actually use Langchain under the hood for some things, and are in the process of figuring out which abstractions make sense to put on the SDK interface. For example, we already use the `langchain.schema.Document` class as the main document dataclass. With making it possible to bring any LLM, we're considering adding support for bringing anything that implements the Langchain LLM interface. In general, we're still trying to figure out how much customizability to give to the user and how (Langchain is a great option here) -- power should not come at the expense of ease of use.

boredemployee 3y ago

The amount of cynical comments everywhere always get me. Anyway, congrats, it will be really useful for building POCs and mvps quickly. I'll definetely give it a try and feedback once I test it!

JohannesTk 3y ago
Thanks for the support! :) Would love to hear your feedback on what's good, what sucks and what to improve.