8 ms·

In some way, I'd be grateful if they screwed up ChatGPT (even though I really like to use it). The best way to be sure that no corporation can mess with one of

by t_mann 3y ago

In some way, I'd be grateful if they screwed up ChatGPT (even though I really like to use it). The best way to be sure that no corporation can mess with one of your most important work tools is to host it yourself, and correct for the shortcomings of the likely smaller models by finetuning/RAG'ing/[whatever cool techniques exist out there and are still to come] it to your liking. And I think having a community around open source models for what promises to be a very important class of tech is an important safeguard against SciFi dystopias where we depend on ad-riddled products by a few megacorps. As long as ChatGPT is the best product out there that I'll never match, there's simply little reason to do so. If they continue to mess it up, that might give lazy bums like me the kick they need to get started.

dvfjsdhgfv 3y ago
The "open source" LLMs are already good enough for simple tasks GPT-3.5 was used for. I see no reason why they can't catch up with GPT-4 one day.
- throwawaybbq1 3y ago
  I assume you are referring to Llama 2? Is there a way to compare models? e.g. what is Llama-7b equivalent to in OpenAI land? Perplexity scores? Also, does ChatGPT use GPT 4 under the hood or 3.5?
  - Tiberium 3y ago
    Actually, there have been new model releases after LLaMA 2. For example, for small models Mistral 7B is simply unbeatable, with a lot of good fine-tunes available for it. Usually people compare models with all the different benchmarks, but of course sometimes models get trained on benchmark datasets, so there's no true way of knowing except if you have a private benchmark or just try the model yourself. I'd say that Mistral 7B is still short of gpt-3.5-turbo, but Mixtral 7x8B (the Mixture-of-Experts one) is comparable. You can try them all at https://chat.lmsys.org/ https://chat.lmsys.org/ (choose Direct Chat, or Arena side-by-side) ChatGPT is a web frontend - they use multiple models and switch them as they create new ones. Currently, the free ChatGPT version is running 3.5, but if you get ChatGPT Plus, you get (limited by messages/hour) access to 4, which is currently served with their GPT-4-Turbo model.
    - mark_l_watson 3y ago
      I agree with your comments and want to add re: benchmarks: I don’t pay too much attention to benchmarks, but I have the advantage of now being retired so I can spend time experimenting with a variety of local models I run with Ollama and commercial offerings. I spend time to build my own, very subjective, views of what different models are good for. One kind of model analysis that I do like are the circle displays on Hugging Face that show how a model benchmarks for different capabilities (word problems, coding, etc.)
  - dkarras 3y ago
    no it's mistral. mistral 7b and mixtral 8x7b MoE which is almost on par (or better than) chatgpt 3.5. Mistral 7b itself packs a punch as well.
    - mark_l_watson 3y ago
      Mixtral 8x7b continues to amaze me, even though I have to run it with 3 bit quantization on my Mac (I just have 32G memory). When I run this model on commercial services with 4 or more bits of quantization I definitely notice, subjectively, better results. I like to play around with smaller models and regular app code in Common Lisp or Racket, and Mistral 7b is very good for that. Mixing and matching old fashioned coding with the NLP, limited world knowledge, and data manipulation capabilities of LLMs.
      - throwawaybbq1 3y ago
        This is neat to know. On Ollama, I see mistral and mixtral. Is the latter one the MoE model?
        dkarras 3y ago
        yes, mixtral is the MoE model.
      - dkarras 3y ago
        There is also MiQu (stands for mi(s|x)tral quantized I think?) which is a leaked and older mistral medium model. I have not been able to try it as it needs some RAM / VRAM I don't have but people say it is very good.
  - tarruda 3y ago
    > Is there a way to compare models? This is what I like to use for comparing models: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... It is an ELO system based on users voting LLM answers to real questions > what is Llama-7b equivalent to in OpenAI land? I don't think Llama 7b compares with OpenAI models, but if you look in the rank I linked above, there are some 7B models which rank higher than early versions of GPT 3.5. those models are Mistral 7b fine tunes.
  - guappa 3y ago
    llama 2 isn't open source
  - int_19h 3y ago
    Miqu (the leaked large Mistral model) and its finetunes seem to be the most coherent currently, and I'd say they beat GPT-3.5 handily. There are no models comparable to GPT-4, open source or not. Not even close.
- imtringued 3y ago
  The opensource ones are already competitive to GPT3.5 in terms of "reasoning" and instruction following. They tend to be significantly worse in knowledge tasks though, due to their lack of parameters. GPT 3.5 is five times bigger than mixtral after all.
- herbst 3y ago
  It's been a few months since I tested but as far as commercially useable AIs go nothing could beat GPT 3.5 for conversations and staying in a character. Llama 2 and other available clones were way to technical (good at that tho)
chx 3y ago
> for what promises to be a very important class of tech What I see here is the automated plagiarism machine can't give you the answer only what the answer would sound like. So you need to countercheck everything it gives you and if you need to do so then why bother using it at all? I am totally baffled by the hype.
- x0x0 3y ago
  For things that are well covered on stack overflow, it's a strictly better search engine. eg say you don't remember the syntax for a rails migration, or a regex, or something you're coding in bash, or processpool arguments in python. ChatGPT will often do a shockingly good job at answering those without you searching through random docs, stack overflow, all the bullshit google loves to throw at the top of search queries, etc yourself. You can even paste in a bunch of your code and ask it to fill in something with context, at which it regularly does a shockingly good job. Or paste code and say you want a test that hits some specific aspect of the code. And yeah, I don't really care if they train on the code I share -- figuring out the interaction of some stupid file upload lib with aws and cloudflare is not IP that I care about, and i chatgpt uses this to learn and save anyone else from the issues I was having, even a competitor, I'm happy for them. For a real example: > can you show me how to build a css animation? I'd like a bar, perhaps 20 pixels high, with a light blue (ideally bootstrap 5.3 colors) small gradient both vertically and horizontally, that 1 - fades in; 2 - starts on the left of the div and takes perhaps 20% of the div; 3 - grows to the right of the div; and 4 - loops This got me 95% of where I wanted; I fiddled with the keyframe percents a bit and we use this in our product today. It spat out 30 lines of css that I absolutely could not have produced in under 2 hours.
  - skerit 3y ago
    > This got me 95% of where I wanted Exactly. Even when it gives an answer that contains many mistakes, or doesn't work at all, I still get some valuable information out of it that does in the end save me a lot of time. I'm so tired of constantly seeing remarks that basically boil down to "Look, I asked ChatGPT to do my job for me and it failed! What a piece of garbage! Ban AI!", which funnily enough mostly comes from people that fear that their job will be 100% replaced by an AI.

by t_mann 3y ago

dvfjsdhgfv 3y ago

The "open source" LLMs are already good enough for simple tasks GPT-3.5 was used for. I see no reason why they can't catch up with GPT-4 one day.

throwawaybbq1 3y ago
I assume you are referring to Llama 2? Is there a way to compare models? e.g. what is Llama-7b equivalent to in OpenAI land? Perplexity scores? Also, does ChatGPT use GPT 4 under the hood or 3.5?
- Tiberium 3y ago
  Actually, there have been new model releases after LLaMA 2. For example, for small models Mistral 7B is simply unbeatable, with a lot of good fine-tunes available for it. Usually people compare models with all the different benchmarks, but of course sometimes models get trained on benchmark datasets, so there's no true way of knowing except if you have a private benchmark or just try the model yourself. I'd say that Mistral 7B is still short of gpt-3.5-turbo, but Mixtral 7x8B (the Mixture-of-Experts one) is comparable. You can try them all at https://chat.lmsys.org/ https://chat.lmsys.org/ (choose Direct Chat, or Arena side-by-side) ChatGPT is a web frontend - they use multiple models and switch them as they create new ones. Currently, the free ChatGPT version is running 3.5, but if you get ChatGPT Plus, you get (limited by messages/hour) access to 4, which is currently served with their GPT-4-Turbo model.
  - mark_l_watson 3y ago
    I agree with your comments and want to add re: benchmarks: I don’t pay too much attention to benchmarks, but I have the advantage of now being retired so I can spend time experimenting with a variety of local models I run with Ollama and commercial offerings. I spend time to build my own, very subjective, views of what different models are good for. One kind of model analysis that I do like are the circle displays on Hugging Face that show how a model benchmarks for different capabilities (word problems, coding, etc.)
- dkarras 3y ago
  no it's mistral. mistral 7b and mixtral 8x7b MoE which is almost on par (or better than) chatgpt 3.5. Mistral 7b itself packs a punch as well.
  - mark_l_watson 3y ago
    Mixtral 8x7b continues to amaze me, even though I have to run it with 3 bit quantization on my Mac (I just have 32G memory). When I run this model on commercial services with 4 or more bits of quantization I definitely notice, subjectively, better results. I like to play around with smaller models and regular app code in Common Lisp or Racket, and Mistral 7b is very good for that. Mixing and matching old fashioned coding with the NLP, limited world knowledge, and data manipulation capabilities of LLMs.
    - throwawaybbq1 3y ago
      This is neat to know. On Ollama, I see mistral and mixtral. Is the latter one the MoE model?
      - dkarras 3y ago
        yes, mixtral is the MoE model.
    - dkarras 3y ago
      There is also MiQu (stands for mi(s|x)tral quantized I think?) which is a leaked and older mistral medium model. I have not been able to try it as it needs some RAM / VRAM I don't have but people say it is very good.
- tarruda 3y ago
  > Is there a way to compare models? This is what I like to use for comparing models: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... It is an ELO system based on users voting LLM answers to real questions > what is Llama-7b equivalent to in OpenAI land? I don't think Llama 7b compares with OpenAI models, but if you look in the rank I linked above, there are some 7B models which rank higher than early versions of GPT 3.5. those models are Mistral 7b fine tunes.
- guappa 3y ago
  llama 2 isn't open source
- int_19h 3y ago
  Miqu (the leaked large Mistral model) and its finetunes seem to be the most coherent currently, and I'd say they beat GPT-3.5 handily. There are no models comparable to GPT-4, open source or not. Not even close.
imtringued 3y ago
The opensource ones are already competitive to GPT3.5 in terms of "reasoning" and instruction following. They tend to be significantly worse in knowledge tasks though, due to their lack of parameters. GPT 3.5 is five times bigger than mixtral after all.
herbst 3y ago
It's been a few months since I tested but as far as commercially useable AIs go nothing could beat GPT 3.5 for conversations and staying in a character. Llama 2 and other available clones were way to technical (good at that tho)

chx 3y ago

> for what promises to be a very important class of tech What I see here is the automated plagiarism machine can't give you the answer only what the answer would sound like. So you need to countercheck everything it gives you and if you need to do so then why bother using it at all? I am totally baffled by the hype.

x0x0 3y ago
For things that are well covered on stack overflow, it's a strictly better search engine. eg say you don't remember the syntax for a rails migration, or a regex, or something you're coding in bash, or processpool arguments in python. ChatGPT will often do a shockingly good job at answering those without you searching through random docs, stack overflow, all the bullshit google loves to throw at the top of search queries, etc yourself. You can even paste in a bunch of your code and ask it to fill in something with context, at which it regularly does a shockingly good job. Or paste code and say you want a test that hits some specific aspect of the code. And yeah, I don't really care if they train on the code I share -- figuring out the interaction of some stupid file upload lib with aws and cloudflare is not IP that I care about, and i chatgpt uses this to learn and save anyone else from the issues I was having, even a competitor, I'm happy for them. For a real example: > can you show me how to build a css animation? I'd like a bar, perhaps 20 pixels high, with a light blue (ideally bootstrap 5.3 colors) small gradient both vertically and horizontally, that 1 - fades in; 2 - starts on the left of the div and takes perhaps 20% of the div; 3 - grows to the right of the div; and 4 - loops This got me 95% of where I wanted; I fiddled with the keyframe percents a bit and we use this in our product today. It spat out 30 lines of css that I absolutely could not have produced in under 2 hours.
- skerit 3y ago
  > This got me 95% of where I wanted Exactly. Even when it gives an answer that contains many mistakes, or doesn't work at all, I still get some valuable information out of it that does in the end save me a lot of time. I'm so tired of constantly seeing remarks that basically boil down to "Look, I asked ChatGPT to do my job for me and it failed! What a piece of garbage! Ban AI!", which funnily enough mostly comes from people that fear that their job will be 100% replaced by an AI.