7 ms·

Awesome-totally-open-ChatGPT: A list of open alternatives to ChatGPT

343 points by pallas_athena 4y ago

amelius 4y ago
Cool, but a pity there are no benchmarks so we can compare them.
- lionkor 4y ago
  If OpenAI would release more than masturbation papers, yes, we could compare them. Right now, we only know GPT-4 does things, and looks smart when you ask it stuff.
  - isaacfrond 4y ago
    What do you mean? The paper [1] is full of benchmarks? [1] GPT-4 Technical Report, https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf
    - brabel 4y ago
      Just looking at the pictures and graphs in that paper is enough to become amazed by what they're achieving. The example where they show 3 pictures of an old monitor plug being connected to an iphone to recharge it, and then GPT4 is asked what's funny about it, and answers incredibly accurately, is amazing.
      - amrb 4y ago
        Since we don't have access to this feature lets be skeptical, its feels like "leading the witness," if your asks what be the funny here. Also if the image is from a forum or sub with funny images is that able to give it away? Having multiple tests would be a stronger test say with example prompts: "whats going on in this picture", "what would a person think seeing this image" etc.. gpt4 is cool as a numbers box but this is not reasoning logic and without papers hasn't been proven either.
    - amrb 4y ago
      Problem being its not a Research Paper, which they where doing previously. This is very bad state as you're not detailing anything that external parties can recreate or prove the scientific method. They can exclaim the model says 40% less "xbox live gamer words" which people outside the company couldn't validate. tl:dr OpenAi is now a business Worth watch Yannic talk about the problem and other cool ML topics too. https://www.youtube.com/watch?v=2zW33LfffPc https://www.youtube.com/watch?v=2zW33LfffPc
      - siva7 4y ago
        It's not like a closed model only available to scientists you can't benchmark yourself. Benchmarking should also be done by a 3rd party otherwise we have a conflict of interest.
        amrb 4y ago
        If this was a cpu/graphic cards sure lets benchmark it, worst case you getting less frames. Here we'd need to see more about its design and safety, else you may be getting recipes for veggie dishes when what you really wanted was fried chicken.
        brookst 4y ago
        How would knowing the architecture or safety mechanisms help you decide if it’s going to give incorrect results more than actual testing would? I’m no LLM expert, but I don’t think you can eyeball the arch and say “that’s going to confuse veggies for fried chicken”.
    - kgarten 4y ago
      https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks https://aisnakeoil.substack.com/p/gpt-4-and-professional-ben... "GPT-4 and professional benchmarks: the wrong answer to the wrong question OpenAI may have tested on the training data. Besides, human benchmarks are meaningless for bots."
  - sebzim4500 4y ago
    There are plenty of people with access to GPT-4 through the API. They can benchmark it however they want.
    - mdaniel 4y ago
      Your suggestion made me wonder: are the OpenAI API responses deterministic? I would expect that if one gets a different reply to the same prompt each time (as I would expect, given the probabilistic nature of LLM), that makes "benchmarking" kind of weird. Or maybe the solution is just to run the prompt 100 times and mean/average across whatever metric is being used
lgbr 4y ago
It's absolutely fantastic that we have so many runtimes, so quickly, to the point where we have an awesome list. However, given that the usefulness of chatbots depends more on the model being used, what I would find a lot more useful is a ranking of the various models that are available. Currently I'm having to rely on comments on the internet to find out if Alpaca 7B or LlaMA 65B is genuinely productive to use. As new models come out, I'd love it if I knew how well it tells jokes, answers complicated questions, or generates code.
- LASR 4y ago
  We have a whole team of folks just watching for these to come out and then go evaluate them. Short answer: none of them do as well as the OG Davinci-003. Not even close. Even the 3.5 Turbo models from OpenAI don’t do as well. We throw some sophisticated prompts at them to attempt chain of thought reasoning.
  - inciampati 4y ago
    Do you have a citation for that?
    - sirlunchalot 4y ago
      would be interested in that as well
  - dotancohen 4y ago
    Could you expand on this a bit more? What types of prompts? What are your evaluation criteria? This actually sounds fascinating. Not unlike birdwatching! ))
  - dr_dshiv 4y ago
    We need open benchmarks, clearly. Know any projects in that space?
  - joenot443 4y ago
    That’s interesting - what about 4?
not-chatgpt 4y ago
I've tried most of these and I gotta say, ChatGLM is actually pretty good. The Chinese responses are sometimes indistinguishable from those of ChatGPT, and it's only a 6B model. Sadly, whenever I use harder questions or logical ones, it performs terribly. It also occasionally uses Chinese adjectives (albeit correctly) when responding in English.
- Jensson 4y ago
  I wonder if their censorship means that the Chinese corpus has less spam than the rest of the internet? Would be interesting if that turns out to be a huge advantage for making AI.
  - brookst 4y ago
    Chinese censorship is both subtractive and additive (“flood the zone with shit”). There will be plenty of spam in the corpus.
runka 4y ago
I feel like we need to invest in open alternatives otherwise we are gonna create something worse than IBM used to be.
- elorant 4y ago
  What we need is a crowdsource project where we dedicate gpu resources to build a model from scratch. There are thousands of us in here, and thousands more in the wide community.
  - inawarminister 4y ago
    Open-assistant.io is what you want. They've gotten more than 100k RLHF training dataset and is actively tuning up a Chat model based on GPT-NeoX (?) from what I've heard.
Garlef 4y ago
What would be great: A list of performance benchmarks: What's the model good at?
- VadimPR 4y ago
  Those exist - the trouble is that they're not for the average mortal...
mg 4y ago
Comparing search engines has been my passion for a while now. I built various tools around it (https://www.gnod.com/search/ https://www.gnod.com/search/) With the advent of LLMs, I also started doing it for AI engines recently: https://www.gnod.com/search/ai https://www.gnod.com/search/ai Will keep it updated with new LLMs as they come on the scene. If anybody runs one of these open ChatGPT alternatives as a service, let me know. I would love to include your service in the AI comparison.
- efficientsticks 4y ago
  Good idea. A couple more non-AI engines: https://search.marginalia.nu/ https://search.marginalia.nu/ and my own https://highbrow.se/ https://highbrow.se/
  - marginalia_nu 4y ago
    Something in Marginalia's query processing is almost comically broken right now. Only really works well for single-term queries. Got a fix in the pipe, but it'll be a week or so.
- toyg 4y ago
  Nice, I've found engines that our corporate firewall doesn't block, woot =)
- notRobot 4y ago
  Prompt: What would happen if the Eiffel tower was submerged in a massive tub of liquid nitrogen? ---- YouChat: Based on the provided search results, there does not appear to be any reliable information on what would happen if the Eiffel Tower was submerged in a massive tub of liquid nitrogen. It is important to note that submerging the Eiffel Tower in such extreme conditions could potentially cause damage or destruction to the tower, and it is not recommended to attempt such an experiment. If you have any other questions or concerns, please let me know and I will do my best to assist you. ---- Phind: Liquid nitrogen is a colorless, odorless, and non-flammable substance that is used in many industrial processes. It has a boiling point of −196 °C (−321 °F), which is much lower than the freezing point of water. If the Eiffel Tower were submerged in a massive tub of liquid nitrogen, several things would happen [4]. The liquid nitrogen would quickly evaporate into gas since the Eiffel Tower is much warmer than −196 °C. This would create a cloud of nitrogen gas around the tower, which could obscure its visibility. The tower's metal structure would initially contract due to the extreme cold, which could cause it to make cracking or popping sounds. However, the contraction would be minimal since the tower is made of wrought iron, which has a low coefficient of thermal expansion [5]. The tower's paint would become brittle and could crack due to the sudden change in temperature. However, the paint layer is not thick enough to affect the overall structural integrity of the tower [1]. The tower's micro-movements due to wind and sun exposure would be reduced since the metal structure would be much colder and less prone to expansion and contraction. However, this effect would be temporary since the tower would eventually warm up again [1]. The tower's lights would still be visible since they are powered by electricity, which is not affected by the temperature of the tower [3]. In summary, submerging the Eiffel Tower in a massive tub of liquid nitrogen would not cause any significant damage to the tower's structure. However, it could create a temporary cloud of nitrogen gas and cause the tower's paint to become brittle. [1]: https://www.toureiffel.paris/en/news/130-years/how-long-can-tower-last https://www.toureiffel.paris/en/news/130-years/how-long-can-... [2]: <missing> [3]: https://www.toureiffel.paris/en/news/history-and-culture/15-essential-things-know-about-eiffel-tower https://www.toureiffel.paris/en/news/history-and-culture/15-... [4]: https://quizlet.com/501418121/assignment-4-eiffel-flash-cards/ https://quizlet.com/501418121/assignment-4-eiffel-flash-card... [5]: https://www.snexplores.org/article/how-science-saved-eiffel-tower https://www.snexplores.org/article/how-science-saved-eiffel-... ---- Perplexity: Submerging the Eiffel Tower in a massive tub of liquid nitrogen would not be possible as it would require an enormous amount of liquid nitrogen, which is not readily available. However, if such a scenario were to occur, the tower would become extremely cold and brittle due to the low temperature of liquid nitrogen (-196°C). The metal structure of the tower would contract and shrink, causing it to become more fragile and prone to breaking. Additionally, any water or moisture on the tower would freeze instantly, leading to potential damage or destruction of the structure[1][2]. [1]: https://youtube.com/watch?v=u4gobiTvFg0 https://youtube.com/watch?v=u4gobiTvFg0 [2]: https://insh.world/science/what-if-we-poured-a-pool-of-liquid-nitrogen-into-a-volcano/ https://insh.world/science/what-if-we-poured-a-pool-of-liqui...
siwakotisaurav 4y ago
ChatGLM is the only one on the list whose performance feels similar to gpt3. I tried out Alpaca, wasn’t as good yet but I’ve heard on Twitter someone has a better dataset and has finetuned it, let’s see if they release the model weights
intalentive 4y ago
No! We can’t have open alternatives! That would be “unsafe”. We need regulatory capt— I mean, responsible government regulation now! As a MSFT shareholder I demand it.
- nashashmi 4y ago
  Sam Altman: "We definitely need more regulation on AI" Regulation could prevent indie AI projects.
  - BizarroLand 4y ago
    Just like regulation stopped piracy and 3d printed weaponry and prostitution and the war on drugs and prohibition stopped the drug and alcohol trade in America.
    - lisp-pornstar 4y ago
      Regulation can do bad things/Regulation can do good things ? Is this what you meant ?
      - bigfishrunning 4y ago
        I suspect that gp meant that, for a lot of things, regulation is ineffective and costly.
- mhuffman 4y ago
  OpenAI: This technology is too power to be put in the hands of mere mortals! Also OpenAI: ... unless, they pay us $20 per month!
satvikpendem 4y ago
I posted this yesterday too but I've found OpenAssistant to be the best of the OSS ones so far (Alpaca and LLaMA are non-commercial so not truly open): https://huggingface.co/spaces/olivierdehaene/chat-llm-streaming https://huggingface.co/spaces/olivierdehaene/chat-llm-stream...
- intalentive 4y ago
  What's nice about Open Assistant is that it's continuously improving and the community can contribute to the training data. It is potentially MUCH better than ChatGPT.

343 points by pallas_athena 4y ago

amelius 4y ago

Cool, but a pity there are no benchmarks so we can compare them.

lionkor 4y ago
If OpenAI would release more than masturbation papers, yes, we could compare them. Right now, we only know GPT-4 does things, and looks smart when you ask it stuff.
- isaacfrond 4y ago
  What do you mean? The paper [1] is full of benchmarks? [1] GPT-4 Technical Report, https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf
  - brabel 4y ago
    Just looking at the pictures and graphs in that paper is enough to become amazed by what they're achieving. The example where they show 3 pictures of an old monitor plug being connected to an iphone to recharge it, and then GPT4 is asked what's funny about it, and answers incredibly accurately, is amazing.
    - amrb 4y ago
      Since we don't have access to this feature lets be skeptical, its feels like "leading the witness," if your asks what be the funny here. Also if the image is from a forum or sub with funny images is that able to give it away? Having multiple tests would be a stronger test say with example prompts: "whats going on in this picture", "what would a person think seeing this image" etc.. gpt4 is cool as a numbers box but this is not reasoning logic and without papers hasn't been proven either.
  - amrb 4y ago
    Problem being its not a Research Paper, which they where doing previously. This is very bad state as you're not detailing anything that external parties can recreate or prove the scientific method. They can exclaim the model says 40% less "xbox live gamer words" which people outside the company couldn't validate. tl:dr OpenAi is now a business Worth watch Yannic talk about the problem and other cool ML topics too. https://www.youtube.com/watch?v=2zW33LfffPc https://www.youtube.com/watch?v=2zW33LfffPc
    - siva7 4y ago
      It's not like a closed model only available to scientists you can't benchmark yourself. Benchmarking should also be done by a 3rd party otherwise we have a conflict of interest.
      - amrb 4y ago
        If this was a cpu/graphic cards sure lets benchmark it, worst case you getting less frames. Here we'd need to see more about its design and safety, else you may be getting recipes for veggie dishes when what you really wanted was fried chicken.
        brookst 4y ago
        How would knowing the architecture or safety mechanisms help you decide if it’s going to give incorrect results more than actual testing would? I’m no LLM expert, but I don’t think you can eyeball the arch and say “that’s going to confuse veggies for fried chicken”.
  - kgarten 4y ago
    https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks https://aisnakeoil.substack.com/p/gpt-4-and-professional-ben... "GPT-4 and professional benchmarks: the wrong answer to the wrong question OpenAI may have tested on the training data. Besides, human benchmarks are meaningless for bots."
- sebzim4500 4y ago
  There are plenty of people with access to GPT-4 through the API. They can benchmark it however they want.
  - mdaniel 4y ago
    Your suggestion made me wonder: are the OpenAI API responses deterministic? I would expect that if one gets a different reply to the same prompt each time (as I would expect, given the probabilistic nature of LLM), that makes "benchmarking" kind of weird. Or maybe the solution is just to run the prompt 100 times and mean/average across whatever metric is being used

lgbr 4y ago

It's absolutely fantastic that we have so many runtimes, so quickly, to the point where we have an awesome list. However, given that the usefulness of chatbots depends more on the model being used, what I would find a lot more useful is a ranking of the various models that are available. Currently I'm having to rely on comments on the internet to find out if Alpaca 7B or LlaMA 65B is genuinely productive to use. As new models come out, I'd love it if I knew how well it tells jokes, answers complicated questions, or generates code.

LASR 4y ago
We have a whole team of folks just watching for these to come out and then go evaluate them. Short answer: none of them do as well as the OG Davinci-003. Not even close. Even the 3.5 Turbo models from OpenAI don’t do as well. We throw some sophisticated prompts at them to attempt chain of thought reasoning.
- inciampati 4y ago
  Do you have a citation for that?
  - sirlunchalot 4y ago
    would be interested in that as well
- dotancohen 4y ago
  Could you expand on this a bit more? What types of prompts? What are your evaluation criteria? This actually sounds fascinating. Not unlike birdwatching! ))
- dr_dshiv 4y ago
  We need open benchmarks, clearly. Know any projects in that space?
- joenot443 4y ago
  That’s interesting - what about 4?

not-chatgpt 4y ago

I've tried most of these and I gotta say, ChatGLM is actually pretty good. The Chinese responses are sometimes indistinguishable from those of ChatGPT, and it's only a 6B model. Sadly, whenever I use harder questions or logical ones, it performs terribly. It also occasionally uses Chinese adjectives (albeit correctly) when responding in English.

Jensson 4y ago
I wonder if their censorship means that the Chinese corpus has less spam than the rest of the internet? Would be interesting if that turns out to be a huge advantage for making AI.
- brookst 4y ago
  Chinese censorship is both subtractive and additive (“flood the zone with shit”). There will be plenty of spam in the corpus.

runka 4y ago

I feel like we need to invest in open alternatives otherwise we are gonna create something worse than IBM used to be.

elorant 4y ago
What we need is a crowdsource project where we dedicate gpu resources to build a model from scratch. There are thousands of us in here, and thousands more in the wide community.
- inawarminister 4y ago
  Open-assistant.io is what you want. They've gotten more than 100k RLHF training dataset and is actively tuning up a Chat model based on GPT-NeoX (?) from what I've heard.

Garlef 4y ago

What would be great: A list of performance benchmarks: What's the model good at?

VadimPR 4y ago
Those exist - the trouble is that they're not for the average mortal...

mg 4y ago

Comparing search engines has been my passion for a while now. I built various tools around it (https://www.gnod.com/search/ https://www.gnod.com/search/) With the advent of LLMs, I also started doing it for AI engines recently: https://www.gnod.com/search/ai https://www.gnod.com/search/ai Will keep it updated with new LLMs as they come on the scene. If anybody runs one of these open ChatGPT alternatives as a service, let me know. I would love to include your service in the AI comparison.

efficientsticks 4y ago
Good idea. A couple more non-AI engines: https://search.marginalia.nu/ https://search.marginalia.nu/ and my own https://highbrow.se/ https://highbrow.se/
- marginalia_nu 4y ago
  Something in Marginalia's query processing is almost comically broken right now. Only really works well for single-term queries. Got a fix in the pipe, but it'll be a week or so.
toyg 4y ago
Nice, I've found engines that our corporate firewall doesn't block, woot =)
notRobot 4y ago
Prompt: What would happen if the Eiffel tower was submerged in a massive tub of liquid nitrogen? ---- YouChat: Based on the provided search results, there does not appear to be any reliable information on what would happen if the Eiffel Tower was submerged in a massive tub of liquid nitrogen. It is important to note that submerging the Eiffel Tower in such extreme conditions could potentially cause damage or destruction to the tower, and it is not recommended to attempt such an experiment. If you have any other questions or concerns, please let me know and I will do my best to assist you. ---- Phind: Liquid nitrogen is a colorless, odorless, and non-flammable substance that is used in many industrial processes. It has a boiling point of −196 °C (−321 °F), which is much lower than the freezing point of water. If the Eiffel Tower were submerged in a massive tub of liquid nitrogen, several things would happen [4]. The liquid nitrogen would quickly evaporate into gas since the Eiffel Tower is much warmer than −196 °C. This would create a cloud of nitrogen gas around the tower, which could obscure its visibility. The tower's metal structure would initially contract due to the extreme cold, which could cause it to make cracking or popping sounds. However, the contraction would be minimal since the tower is made of wrought iron, which has a low coefficient of thermal expansion [5]. The tower's paint would become brittle and could crack due to the sudden change in temperature. However, the paint layer is not thick enough to affect the overall structural integrity of the tower [1]. The tower's micro-movements due to wind and sun exposure would be reduced since the metal structure would be much colder and less prone to expansion and contraction. However, this effect would be temporary since the tower would eventually warm up again [1]. The tower's lights would still be visible since they are powered by electricity, which is not affected by the temperature of the tower [3]. In summary, submerging the Eiffel Tower in a massive tub of liquid nitrogen would not cause any significant damage to the tower's structure. However, it could create a temporary cloud of nitrogen gas and cause the tower's paint to become brittle. [1]: https://www.toureiffel.paris/en/news/130-years/how-long-can-tower-last https://www.toureiffel.paris/en/news/130-years/how-long-can-... [2]: <missing> [3]: https://www.toureiffel.paris/en/news/history-and-culture/15-essential-things-know-about-eiffel-tower https://www.toureiffel.paris/en/news/history-and-culture/15-... [4]: https://quizlet.com/501418121/assignment-4-eiffel-flash-cards/ https://quizlet.com/501418121/assignment-4-eiffel-flash-card... [5]: https://www.snexplores.org/article/how-science-saved-eiffel-tower https://www.snexplores.org/article/how-science-saved-eiffel-... ---- Perplexity: Submerging the Eiffel Tower in a massive tub of liquid nitrogen would not be possible as it would require an enormous amount of liquid nitrogen, which is not readily available. However, if such a scenario were to occur, the tower would become extremely cold and brittle due to the low temperature of liquid nitrogen (-196°C). The metal structure of the tower would contract and shrink, causing it to become more fragile and prone to breaking. Additionally, any water or moisture on the tower would freeze instantly, leading to potential damage or destruction of the structure[1][2]. [1]: https://youtube.com/watch?v=u4gobiTvFg0 https://youtube.com/watch?v=u4gobiTvFg0 [2]: https://insh.world/science/what-if-we-poured-a-pool-of-liquid-nitrogen-into-a-volcano/ https://insh.world/science/what-if-we-poured-a-pool-of-liqui...

siwakotisaurav 4y ago

ChatGLM is the only one on the list whose performance feels similar to gpt3. I tried out Alpaca, wasn’t as good yet but I’ve heard on Twitter someone has a better dataset and has finetuned it, let’s see if they release the model weights

intalentive 4y ago

No! We can’t have open alternatives! That would be “unsafe”. We need regulatory capt— I mean, responsible government regulation now! As a MSFT shareholder I demand it.

nashashmi 4y ago
Sam Altman: "We definitely need more regulation on AI" Regulation could prevent indie AI projects.
- BizarroLand 4y ago
  Just like regulation stopped piracy and 3d printed weaponry and prostitution and the war on drugs and prohibition stopped the drug and alcohol trade in America.
  - lisp-pornstar 4y ago
    Regulation can do bad things/Regulation can do good things ? Is this what you meant ?
    - bigfishrunning 4y ago
      I suspect that gp meant that, for a lot of things, regulation is ineffective and costly.
mhuffman 4y ago
OpenAI: This technology is too power to be put in the hands of mere mortals! Also OpenAI: ... unless, they pay us $20 per month!

satvikpendem 4y ago

I posted this yesterday too but I've found OpenAssistant to be the best of the OSS ones so far (Alpaca and LLaMA are non-commercial so not truly open): https://huggingface.co/spaces/olivierdehaene/chat-llm-streaming https://huggingface.co/spaces/olivierdehaene/chat-llm-stream...

intalentive 4y ago
What's nice about Open Assistant is that it's continuously improving and the community can contribute to the training data. It is potentially MUCH better than ChatGPT.