3 ms·

> ...consistently at the top of every key benchmark, but more importantly the clear winner in terms of “vibes”. Can anybody explain what is meant by "vibes" he

> ...consistently at the top of every key benchmark, but more importantly the clear winner in terms of “vibes”. Can anybody explain what is meant by "vibes" here? Is it an industry term?

rvz 3y ago
Translation: "mind share"
swyx 3y ago
qualitative evaluations from simply trying the thing. many things not captured by the quantitative benchmarks, which are easy to game.
minimaxir 3y ago
It's a term to represent that in the real-world, the quality of a model can't be measured objectively so the most common way to evaluate models personally is to see whether you like the output, with all the selection, anchoring, and recency biases that entails.
sixhobbits 3y ago
First time I have heard the term (and I have done quite a bit of llm benchmarking) but I love it! Similar to the xkcd "wtfs per minute" measure of code quality measure I guess. Everyone I know who plays with LLMs says things like - it didn't answer my special secret benchmark question better than previous models - it surprised me by being able to give a satisfying answer for x that I wasn't expecting - it always seems to give answers that are [Some negative quality] Obviously these things are super subjective but still important and useful in a way that "answers 5th grade maths questions more accurately than previous models" sometimes isn't
- gremlinsinc 3y ago
  when I have a new use for ai I'll try Claude, Bing, and Gemini and go with whoever is best for that use case. sometimes I'll have another model check and offer suggestions for improvement like for Etsy listings title or description etc...
simonw 3y ago
It's a bit of an industry in-joke. The problem with LLM benchmarks is that they don't really help you get a "feel" for what it's like using them. A model might score off the charts but not be particularly useful for your personal style of promoting.

> ...consistently at the top of every key benchmark, but more importantly the clear winner in terms of “vibes”. Can anybody explain what is meant by "vibes" here? Is it an industry term?

rvz 3y ago

Translation: "mind share"

swyx 3y ago

qualitative evaluations from simply trying the thing. many things not captured by the quantitative benchmarks, which are easy to game.

minimaxir 3y ago

It's a term to represent that in the real-world, the quality of a model can't be measured objectively so the most common way to evaluate models personally is to see whether you like the output, with all the selection, anchoring, and recency biases that entails.

sixhobbits 3y ago

First time I have heard the term (and I have done quite a bit of llm benchmarking) but I love it! Similar to the xkcd "wtfs per minute" measure of code quality measure I guess. Everyone I know who plays with LLMs says things like - it didn't answer my special secret benchmark question better than previous models - it surprised me by being able to give a satisfying answer for x that I wasn't expecting - it always seems to give answers that are [Some negative quality] Obviously these things are super subjective but still important and useful in a way that "answers 5th grade maths questions more accurately than previous models" sometimes isn't

gremlinsinc 3y ago
when I have a new use for ai I'll try Claude, Bing, and Gemini and go with whoever is best for that use case. sometimes I'll have another model check and offer suggestions for improvement like for Etsy listings title or description etc...

simonw 3y ago

It's a bit of an industry in-joke. The problem with LLM benchmarks is that they don't really help you get a "feel" for what it's like using them. A model might score off the charts but not be particularly useful for your personal style of promoting.