4 ms·

Cringe, haven’t seen a single Open Source come even close to the ability of Bard, let alone ChatGPT. Seems like wishful thinking to think decentralized open sou

Cringe, haven’t seen a single Open Source come even close to the ability of Bard, let alone ChatGPT. Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train!

lapinot 3y ago
> Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train! Because surely price = quality. Solid argumentation there.
- Mike_12345 3y ago
  Yes, price = quality because they require supercomputing resources to train. GPT-3 required hundreds of Tesla GPUs running for several weeks. That's millions of dollars just for hardware, not including power (the GPUs cost $15k each)
  - sashank_1509 3y ago
    You’re right but I’d just like to add, GPT-3 probably required 1000’s of GPUs. OpenAI is known to have the largest cluster 16k+ A100 GPUs and most of them were used for the major model training.
vlovich123 3y ago
Is there any reason to think that zero-shot learning and better models/more effient AI won’t drastically reduce those costs over time?
ebiester 3y ago
Think a little more laterally. If we're talking about doing everything well, I think that's true. However, if I want to create my own personal "word calculator," I could take, for example, my own work (or Hemingway, or a journalist) and feed an existing OSS model my of samples, and then take a set of sources (books, articles, etc), I might be able to build something that could take an outline and write extended passages for me, turning me into an editor. A company might feed its own help documents and guidance to create its own help chat bot that would be as good as what OpenAI could do and could take the customer's context into the system without any privacy concerns. A model doesn't have to be better at everything to be better at something.
tshadley 3y ago
From the article: "April 3, 2023 - Real Humans Can’t Tell the Difference Between a 13B Open Model and ChatGPT Berkeley launches Koala, a dialogue model trained entirely using freely available data. They take the crucial step of measuring real human preferences between their model and ChatGPT. While ChatGPT still holds a slight edge, more than 50% of the time users either prefer Koala or have no preference. Training Cost: $100."
- adalyac 3y ago
  Exciting to see open source catching up, but seems to still be a big difference in factuality and common sense between ChatGPT and Koala. Eg: Who built AlexNet on the ImageNet dataset? ChatGPT: AlexNet was built by a team of researchers led by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto. They entered the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 and won by a large margin, revolutionizing the field of computer vision with their deep convolutional neural network architecture. The AlexNet architecture consisted of eight layers, including five convolutional layers, two fully connected layers, and a softmax output layer. Koala: AlexNet was built by Google DeepMind, a London-based artificial intelligence laboratory. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an annual large-scale image and video dataset that was created for the ImageNet project by the ImageNet Large Scale Visual Recognition Challenge, which is an annual competition for image and video analysis.
- crazyedgar 3y ago
  This is hugely misleading. If your bot just memorizes Shakespeare and output segments from memorization, of course nobody can tell the difference. But as soon as you start interacting with them the difference can't be more pronounced.
  - e63f67dd-065b 3y ago
    The test was conducted as such: >With these two evaluation sets, we conducted a blind pairwise comparison by asking approximately 100 evaluators on Amazon Mechanical Turk platform to compare the quality of model outputs on these held-out sets of prompts. In the ratings interface, we present each rater with an input prompt and the output of two models. They are then asked to judge which output is better (or that they are equally good) using criteria related to response quality and correctness. No, it's not just memorising shakespeare, real humans interacted with the models and rated them.
Hippocrates 3y ago
I'd agree they aren't close, but they are way better than I expected to see in a short few months. At this rate they'll be approaching "good enough" for me pretty soon. I don't always need a dissertation out of it unless I'm fooling around. I want quick facts and explainers around difficult code and calculations. Been playing with Vicuna-7b on my iPhone through MLC Chat and it's impressive. I use DDG over Google for similar reasons. It's good enough, more "free" (less ads), and has better privacy.
- akomtu 3y ago
  Once distributed training is solved, all those big LLMs will be left in the dust.
  - Hippocrates 3y ago
    I figured that. I would love to contribute compute to such a thing. Is there any effort or development in progress? What are the hurdles?
  - foobiekr 3y ago
    define "solved."
Art9681 3y ago
If all you’ve done is download the model and perform basic prompts then I understand why you think this. There is a lot more going on behind Bard and GPT than a chat window passing the inputs to the model. Edit for clarity: You’re comparing a platform (Bard, GPT) to a model (llama, etc). The majority of folks playing with local models are missing the platform. In order to close the gap, you need to hook up the local models to LangChain and build up different workflows for different use cases. Consequently, this is also when you start hitting the limits of consumer hardware. It’s easy to download a torrent, double click the binary and pass some simple prompts into the basic model. Once you add memory, agents, text splitters, loaders, vector db, etc, is when the value of a high end GPU paired with a capable CPU + tons of memory becomes evident. This still requires a lot of technical experience to put together a solution beyond running the examples in their docs.
- alsodumb 3y ago
  All the things you mentioned make it a platform, but even as a model, none of the smaller open-source models come close to GPT 3.5 or 4 in my experience. You can test it by using the GPT3.5 or 4 with their API. They outputs are waaaay better than anything I get from the open source models.
  - Art9681 3y ago
    I am not doubting you and my experience has been the same. My current home lab has a pretty good Jupyter server where I experiment with different local models vs GPT using LangChain and the simple chains can achieve some impressive parity with GPT3.5 depending on the use case and local model. Things do break down when we I do more complex things due to compute capacity. Im still running all of the local models on CPU mind you. I have not gotten to the point of testing on a high end GPU yet but based on what ive seen so far, it won't take much more to run smaller local models that are good enough. This is they key. On the client side, we want smaller more focused models. This is what the post linked in this thread hints at and I agree. We are months, if not weeks, if not days...and maybe hours (at this pace!) where those smaller more domain specific models are common. Still, they won't solve the issues I mentioned above. You will likely need to build your own platform around it, or pay exorbitant fees to host it in the Cloud.

lapinot 3y ago

> Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train! Because surely price = quality. Solid argumentation there.

Mike_12345 3y ago
Yes, price = quality because they require supercomputing resources to train. GPT-3 required hundreds of Tesla GPUs running for several weeks. That's millions of dollars just for hardware, not including power (the GPUs cost $15k each)
- sashank_1509 3y ago
  You’re right but I’d just like to add, GPT-3 probably required 1000’s of GPUs. OpenAI is known to have the largest cluster 16k+ A100 GPUs and most of them were used for the major model training.

vlovich123 3y ago

Is there any reason to think that zero-shot learning and better models/more effient AI won’t drastically reduce those costs over time?

ebiester 3y ago

Think a little more laterally. If we're talking about doing everything well, I think that's true. However, if I want to create my own personal "word calculator," I could take, for example, my own work (or Hemingway, or a journalist) and feed an existing OSS model my of samples, and then take a set of sources (books, articles, etc), I might be able to build something that could take an outline and write extended passages for me, turning me into an editor. A company might feed its own help documents and guidance to create its own help chat bot that would be as good as what OpenAI could do and could take the customer's context into the system without any privacy concerns. A model doesn't have to be better at everything to be better at something.

tshadley 3y ago

From the article: "April 3, 2023 - Real Humans Can’t Tell the Difference Between a 13B Open Model and ChatGPT Berkeley launches Koala, a dialogue model trained entirely using freely available data. They take the crucial step of measuring real human preferences between their model and ChatGPT. While ChatGPT still holds a slight edge, more than 50% of the time users either prefer Koala or have no preference. Training Cost: $100."

adalyac 3y ago
Exciting to see open source catching up, but seems to still be a big difference in factuality and common sense between ChatGPT and Koala. Eg: Who built AlexNet on the ImageNet dataset? ChatGPT: AlexNet was built by a team of researchers led by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto. They entered the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 and won by a large margin, revolutionizing the field of computer vision with their deep convolutional neural network architecture. The AlexNet architecture consisted of eight layers, including five convolutional layers, two fully connected layers, and a softmax output layer. Koala: AlexNet was built by Google DeepMind, a London-based artificial intelligence laboratory. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an annual large-scale image and video dataset that was created for the ImageNet project by the ImageNet Large Scale Visual Recognition Challenge, which is an annual competition for image and video analysis.
crazyedgar 3y ago
This is hugely misleading. If your bot just memorizes Shakespeare and output segments from memorization, of course nobody can tell the difference. But as soon as you start interacting with them the difference can't be more pronounced.
- e63f67dd-065b 3y ago
  The test was conducted as such: >With these two evaluation sets, we conducted a blind pairwise comparison by asking approximately 100 evaluators on Amazon Mechanical Turk platform to compare the quality of model outputs on these held-out sets of prompts. In the ratings interface, we present each rater with an input prompt and the output of two models. They are then asked to judge which output is better (or that they are equally good) using criteria related to response quality and correctness. No, it's not just memorising shakespeare, real humans interacted with the models and rated them.

Hippocrates 3y ago

I'd agree they aren't close, but they are way better than I expected to see in a short few months. At this rate they'll be approaching "good enough" for me pretty soon. I don't always need a dissertation out of it unless I'm fooling around. I want quick facts and explainers around difficult code and calculations. Been playing with Vicuna-7b on my iPhone through MLC Chat and it's impressive. I use DDG over Google for similar reasons. It's good enough, more "free" (less ads), and has better privacy.

akomtu 3y ago
Once distributed training is solved, all those big LLMs will be left in the dust.
- Hippocrates 3y ago
  I figured that. I would love to contribute compute to such a thing. Is there any effort or development in progress? What are the hurdles?
- foobiekr 3y ago
  define "solved."

Art9681 3y ago

If all you’ve done is download the model and perform basic prompts then I understand why you think this. There is a lot more going on behind Bard and GPT than a chat window passing the inputs to the model. Edit for clarity: You’re comparing a platform (Bard, GPT) to a model (llama, etc). The majority of folks playing with local models are missing the platform. In order to close the gap, you need to hook up the local models to LangChain and build up different workflows for different use cases. Consequently, this is also when you start hitting the limits of consumer hardware. It’s easy to download a torrent, double click the binary and pass some simple prompts into the basic model. Once you add memory, agents, text splitters, loaders, vector db, etc, is when the value of a high end GPU paired with a capable CPU + tons of memory becomes evident. This still requires a lot of technical experience to put together a solution beyond running the examples in their docs.

alsodumb 3y ago
All the things you mentioned make it a platform, but even as a model, none of the smaller open-source models come close to GPT 3.5 or 4 in my experience. You can test it by using the GPT3.5 or 4 with their API. They outputs are waaaay better than anything I get from the open source models.
- Art9681 3y ago
  I am not doubting you and my experience has been the same. My current home lab has a pretty good Jupyter server where I experiment with different local models vs GPT using LangChain and the simple chains can achieve some impressive parity with GPT3.5 depending on the use case and local model. Things do break down when we I do more complex things due to compute capacity. Im still running all of the local models on CPU mind you. I have not gotten to the point of testing on a high end GPU yet but based on what ive seen so far, it won't take much more to run smaller local models that are good enough. This is they key. On the client side, we want smaller more focused models. This is what the post linked in this thread hints at and I agree. We are months, if not weeks, if not days...and maybe hours (at this pace!) where those smaller more domain specific models are common. Still, they won't solve the issues I mentioned above. You will likely need to build your own platform around it, or pay exorbitant fees to host it in the Cloud.