5 ms·

Man this LLM stuff gets released faster than I can keep up. Is there a centralized list somewhere that tests "use this for x purpose, use that for y?"

brucethemoose2 3y ago
Even the OSS trainers/finetuners themselves don't really know the use case for their models. They just hot drop them and move on, lol.
- htrp 3y ago
  They train and drop them.... and rely on you to figure it out.
lxe 3y ago
This is a finetune of the torrent-dropped moe model from a few days ago. Not a "release" per se, but rather a display of community excitement over this.
behnamoh 3y ago
> Is there a centralized list somewhere that tests "use this for x purpose, use that for y?" Yeah, "don't use these models for production, use OpenAI for production, ignore Claude/Gemini/etc.".
- holoduke 3y ago
  Why not. For simple things like converting, translating its quite handy and the costs are with you. I use open models for removing backgrounds from images. Also use models to translate texts. Openai would be too expensive for my usecases.
- PostOnce 3y ago
  "Put all your eggs in one basket" isn't the traditional advice. Any one of these companies can at any time change their API, pricing, access rules, or even swap the model out for a dumber, cheaper one at the same price. You'll have no recourse if you don't have several backends available, or control your own. At a minimum, you should have available several hot-swappable backends/APIs if you want to remain viable in an indeterminate future.
  - Der_Einzige 3y ago
    I want someone to offer a large sum of money to anyone who can leak the full GPT-4 model to prevent ClosedAIs lunacy. We already got a model leak from NAI and it forever propelled AI waifus. Why can’t some Good Samaritan take one for the team and give the world access to the closest thing we have to AGI? Whoever does it will be remembered even more positively than Aaron Swartz Why should I respect copyright on a model when the model training didn’t respect copyright? To be clear, I’m a copyright abolitionist so I’m down with not accepting copyright. I want someone to force OpenAI to be actually open.
- lannisterstark 3y ago
  None of this is about production. I'm not building anything with LLMs. (well, apart from a couple of IRC bots). Most of the reason I'm asking is to use them for personal usage. Not a huge fan of closed models in general.
adt 3y ago
Best I can do, tree fiddy: https://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/
gregsadetsky 3y ago
This isn't exactly what you're looking for, but I've found this ranked list of llms useful -- https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Something more domain/use-specific would be great to have
- stavros 3y ago
  Keep in mind that the rankings are just that: an order of "best" to "worst". There's nothing on there that's an indication of how much better one model is from another.
  - WhitneyLand 3y ago
    That’s exactly what it does. It’s an Elo scoring system, like chess. By definition the difference between two entries represents how often it beats the other entry. https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system
    - stavros 3y ago
      So, GPT-4 beats the next best LLM. Is GPT-4 twice as good, or a hundred times as good as the next one?
      - furyofantares 3y ago
        Looks like 62 points better, which under the Elo model implies about 59/41 matchup (aka GPT-4-Turbo is about 59% chance to beat Claude on a random prompt). You can look up an Elo probability calculator to plug in an Elo difference and see this number. For example plug in two Elo ratings here: https://sandhoefner.github.io/chess.html https://sandhoefner.github.io/chess.html - really only the difference between them matters. Btw the similarity of the numbers 59 and 62 here is coincidence, other differences won't be nearly the same as the probability.
appplication 3y ago
Honestly none of them are particularly incredible compared to neither. At this point everything is a prototype, there’s no sense in building knowledge specific to them unless you’re also building LLM products. It’s just distributed ADHD at this point. LLMs are new and cool and each new release will be significantly better than the last, but as with any emerging tech, we’re on an exponential curve so there’s no sense in falling in love with specifics or products until things stabilize.
- lannisterstark 3y ago
  >It’s just distributed ADHD at this point I just want a FOSS 'ADHD' model that I can ask questions to and it gives me mostly accurate answers as well as help me debug shit lol.

brucethemoose2 3y ago

Even the OSS trainers/finetuners themselves don't really know the use case for their models. They just hot drop them and move on, lol.

htrp 3y ago
They train and drop them.... and rely on you to figure it out.

lxe 3y ago

This is a finetune of the torrent-dropped moe model from a few days ago. Not a "release" per se, but rather a display of community excitement over this.

behnamoh 3y ago

> Is there a centralized list somewhere that tests "use this for x purpose, use that for y?" Yeah, "don't use these models for production, use OpenAI for production, ignore Claude/Gemini/etc.".

holoduke 3y ago
Why not. For simple things like converting, translating its quite handy and the costs are with you. I use open models for removing backgrounds from images. Also use models to translate texts. Openai would be too expensive for my usecases.
PostOnce 3y ago
"Put all your eggs in one basket" isn't the traditional advice. Any one of these companies can at any time change their API, pricing, access rules, or even swap the model out for a dumber, cheaper one at the same price. You'll have no recourse if you don't have several backends available, or control your own. At a minimum, you should have available several hot-swappable backends/APIs if you want to remain viable in an indeterminate future.
- Der_Einzige 3y ago
  I want someone to offer a large sum of money to anyone who can leak the full GPT-4 model to prevent ClosedAIs lunacy. We already got a model leak from NAI and it forever propelled AI waifus. Why can’t some Good Samaritan take one for the team and give the world access to the closest thing we have to AGI? Whoever does it will be remembered even more positively than Aaron Swartz Why should I respect copyright on a model when the model training didn’t respect copyright? To be clear, I’m a copyright abolitionist so I’m down with not accepting copyright. I want someone to force OpenAI to be actually open.
lannisterstark 3y ago
None of this is about production. I'm not building anything with LLMs. (well, apart from a couple of IRC bots). Most of the reason I'm asking is to use them for personal usage. Not a huge fan of closed models in general.

adt 3y ago

Best I can do, tree fiddy: https://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/

gregsadetsky 3y ago

This isn't exactly what you're looking for, but I've found this ranked list of llms useful -- https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Something more domain/use-specific would be great to have

stavros 3y ago
Keep in mind that the rankings are just that: an order of "best" to "worst". There's nothing on there that's an indication of how much better one model is from another.
- WhitneyLand 3y ago
  That’s exactly what it does. It’s an Elo scoring system, like chess. By definition the difference between two entries represents how often it beats the other entry. https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system
  - stavros 3y ago
    So, GPT-4 beats the next best LLM. Is GPT-4 twice as good, or a hundred times as good as the next one?
    - furyofantares 3y ago
      Looks like 62 points better, which under the Elo model implies about 59/41 matchup (aka GPT-4-Turbo is about 59% chance to beat Claude on a random prompt). You can look up an Elo probability calculator to plug in an Elo difference and see this number. For example plug in two Elo ratings here: https://sandhoefner.github.io/chess.html https://sandhoefner.github.io/chess.html - really only the difference between them matters. Btw the similarity of the numbers 59 and 62 here is coincidence, other differences won't be nearly the same as the probability.

appplication 3y ago

Honestly none of them are particularly incredible compared to neither. At this point everything is a prototype, there’s no sense in building knowledge specific to them unless you’re also building LLM products. It’s just distributed ADHD at this point. LLMs are new and cool and each new release will be significantly better than the last, but as with any emerging tech, we’re on an exponential curve so there’s no sense in falling in love with specifics or products until things stabilize.

lannisterstark 3y ago
>It’s just distributed ADHD at this point I just want a FOSS 'ADHD' model that I can ask questions to and it gives me mostly accurate answers as well as help me debug shit lol.