4 ms·

TLDR: openai still crushing the competition. Competing models beat benchmarks, but are borderline worthless on real tasks.

dmezzetti 3y ago
Going to call BS on this. I've delivered multiple projects this year using open models.
- apwell23 3y ago
  > delivered multiple projects Very curious about the type of projects you've delivered.
  - dmezzetti 3y ago
    You can look at https://github.com/neuml/txtai https://github.com/neuml/txtai. Biggest thing of 2023 was RAG with models like Mistral.
    - apwell23 3y ago
      Got it. Thank you. I thought you had delivered consumer facing applications when i read that comment.
      - dmezzetti 3y ago
        I've delivered consulting projects using RAG and Mistral 7B. Not saying it's easy to get the prompts right but I've been able to build systems that work well. Just trying to counter the myth that paid APIs are the only way to do this.
  - benrow 3y ago
    Also curious. I'm a sample of 1 and also relatively inexperienced. But I felt I quickly reached the limits of what was possible when I tried doing sentence classification with OSS sentence embedding models. The issue was with negation. I'd attributed too much magic to embedding models - they don't really understand language. Not to say there isn't very capable tech out there. Just to add a datapoint that "sentiment analysis"-like approaches in blogs don't always scale to your particular use-case. Edit: conscious I've drifted from the topic of chatbot type models, but felt relevant somehow.
ewweezdsd 3y ago
How do you define "real tasks"? Even small models that can be run on consumer CPU can produce a coherent summary of an email, as an example. Isn't that a real task?
ilaksh 3y ago
I think that might have been more true several 8-12 months ago. But now it feels like the momentum has swung towards open source. Multiple models are close to or exceeding GPT 3.5 now. They can do a lot of useful things. I have come to the conclusion that as much as possible I should try to wean myself off of OpenAI immediately. Because it's just not necessary or desirable to be tied to a single vendor anymore for many tasks. And in 2024 the open source capabilities will continue to increase. Soon everyone with a relatively new computer will be running things like LLMs locally. Within a couple of years it will be integrated into every OS or browser.
- Kiro 3y ago
  Why are people benchmarking against 3.5? To me, the real race started with GPT-4. 3.5 and 4 are completely different beasts and that open models are catching up to 3.5 doesn't mean much. I've yet to see anything come close to 4.
  - JanSt 3y ago
    Because 3.5 is a very useful model. Getting there with an open model already is pretty cool.
    - ironhaven 3y ago
      You are only allowed to criticize Open AI if you are paying a monthly subscription to Open AI
- YetAnotherNick 3y ago
  > Multiple models are close to or exceeding GPT 3.5 now Only in benchmarks as GP said. All the models I have tried either hallucinates like crazy or refuse to answer everything. Nowhere close to GPT 3.5.

dmezzetti 3y ago

Going to call BS on this. I've delivered multiple projects this year using open models.

apwell23 3y ago
> delivered multiple projects Very curious about the type of projects you've delivered.
- dmezzetti 3y ago
  You can look at https://github.com/neuml/txtai https://github.com/neuml/txtai. Biggest thing of 2023 was RAG with models like Mistral.
  - apwell23 3y ago
    Got it. Thank you. I thought you had delivered consumer facing applications when i read that comment.
    - dmezzetti 3y ago
      I've delivered consulting projects using RAG and Mistral 7B. Not saying it's easy to get the prompts right but I've been able to build systems that work well. Just trying to counter the myth that paid APIs are the only way to do this.
- benrow 3y ago
  Also curious. I'm a sample of 1 and also relatively inexperienced. But I felt I quickly reached the limits of what was possible when I tried doing sentence classification with OSS sentence embedding models. The issue was with negation. I'd attributed too much magic to embedding models - they don't really understand language. Not to say there isn't very capable tech out there. Just to add a datapoint that "sentiment analysis"-like approaches in blogs don't always scale to your particular use-case. Edit: conscious I've drifted from the topic of chatbot type models, but felt relevant somehow.

ewweezdsd 3y ago

How do you define "real tasks"? Even small models that can be run on consumer CPU can produce a coherent summary of an email, as an example. Isn't that a real task?

ilaksh 3y ago

I think that might have been more true several 8-12 months ago. But now it feels like the momentum has swung towards open source. Multiple models are close to or exceeding GPT 3.5 now. They can do a lot of useful things. I have come to the conclusion that as much as possible I should try to wean myself off of OpenAI immediately. Because it's just not necessary or desirable to be tied to a single vendor anymore for many tasks. And in 2024 the open source capabilities will continue to increase. Soon everyone with a relatively new computer will be running things like LLMs locally. Within a couple of years it will be integrated into every OS or browser.

Kiro 3y ago
Why are people benchmarking against 3.5? To me, the real race started with GPT-4. 3.5 and 4 are completely different beasts and that open models are catching up to 3.5 doesn't mean much. I've yet to see anything come close to 4.
- JanSt 3y ago
  Because 3.5 is a very useful model. Getting there with an open model already is pretty cool.
  - ironhaven 3y ago
    You are only allowed to criticize Open AI if you are paying a monthly subscription to Open AI
YetAnotherNick 3y ago
> Multiple models are close to or exceeding GPT 3.5 now Only in benchmarks as GP said. All the models I have tried either hallucinates like crazy or refuse to answer everything. Nowhere close to GPT 3.5.