3 ms·

I'd say definitely do not do that. That would make the benchmark look more serious while still being problematic for knowledge cutoff reasons. Your prompt has b

I'd say definitely do not do that. That would make the benchmark look more serious while still being problematic for knowledge cutoff reasons. Your prompt has become popular even outside your blog, so the odds of some SVG pelicans on bicycles making it into the training data have been going up and up. Karpathy used it as an example in a recent interview: https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3-other-models-to-draw-pelican-riding-bicycle-see-results/ar-AA1zhfox https://www.msn.com/en-in/health/other/ai-expert-asks-grok-3...

diggan 1y ago
Yeah, this is the problem with benchmarks where the questions/problems are public. They're valuable for some months, until it bleeds into the training set. I'm certain a lot of the "improvements" we're seeing are just benchmarks leaking into the training set.
- travisgriggs 1y ago
  That’s ok, once bicycle “riding” pelicans become normative, we can ask it for images of pelicans humping bicycles. The number of subject-verb-objects are near infinite. All are imaginable, but most are not plausible. A plausibility machine (LLM) will struggle with the implausible, until it can abstract well.
  - diggan 1y ago
    > The number of subject-verb-objects are near infinite. All are imaginable, but most are not plausible Until there is enough unique/new subject-verb-objects examples/benchmarks so the trained model actually generalized it just like you did. (Public) Benchmarks needs to constantly evolve, otherwise they stop being useful.
    - demosthanos 1y ago
      To be fair, once it does generalize the pattern then the benchmark is actually measuring something useful for deciding if the model will be able to product a subject-verb-object SVG.
  - zahlman 1y ago
    I can't fathom this working, simply because building a model that relates the word "ride" to "hump" seems like something that would be orders of magnitude easier for an LLM than visualizing the result of SVG rendering.
6LLvveMx2koXfwn 1y ago
I would definitely say he had no intention of doing that and was doubling down on the original joke.
- colecut 1y ago
  The road to hell is paved with the best intentions clarification: I enjoyed the pelican on a bike and don't think it's that bad =p
- deleted 1y ago
  [deleted]
throwaway31131 1y ago
I’d say it doesn’t really matter. There is no universally good benchmark and really they should only be used to answer very specific questions which may or may not be relevant to you. Also, as the old saying goes, the only thing worse than using benchmarks is not using benchmarks.
telotortium 1y ago
Yeah, Simon needs to release a new benchmark under a pen name, like Stephen King did with Richard Bachman.
- Breza 1y ago
  Richard Bachman, you say? https://chatgpt.com/share/684c3f20-575c-800a-9ea2-889dd3deaf26 https://chatgpt.com/share/684c3f20-575c-800a-9ea2-889dd3deaf...

diggan 1y ago

Yeah, this is the problem with benchmarks where the questions/problems are public. They're valuable for some months, until it bleeds into the training set. I'm certain a lot of the "improvements" we're seeing are just benchmarks leaking into the training set.

travisgriggs 1y ago
That’s ok, once bicycle “riding” pelicans become normative, we can ask it for images of pelicans humping bicycles. The number of subject-verb-objects are near infinite. All are imaginable, but most are not plausible. A plausibility machine (LLM) will struggle with the implausible, until it can abstract well.
- diggan 1y ago
  > The number of subject-verb-objects are near infinite. All are imaginable, but most are not plausible Until there is enough unique/new subject-verb-objects examples/benchmarks so the trained model actually generalized it just like you did. (Public) Benchmarks needs to constantly evolve, otherwise they stop being useful.
  - demosthanos 1y ago
    To be fair, once it does generalize the pattern then the benchmark is actually measuring something useful for deciding if the model will be able to product a subject-verb-object SVG.
- zahlman 1y ago
  I can't fathom this working, simply because building a model that relates the word "ride" to "hump" seems like something that would be orders of magnitude easier for an LLM than visualizing the result of SVG rendering.

6LLvveMx2koXfwn 1y ago

I would definitely say he had no intention of doing that and was doubling down on the original joke.

colecut 1y ago
The road to hell is paved with the best intentions clarification: I enjoyed the pelican on a bike and don't think it's that bad =p
deleted 1y ago
[deleted]

throwaway31131 1y ago

I’d say it doesn’t really matter. There is no universally good benchmark and really they should only be used to answer very specific questions which may or may not be relevant to you. Also, as the old saying goes, the only thing worse than using benchmarks is not using benchmarks.

telotortium 1y ago

Yeah, Simon needs to release a new benchmark under a pen name, like Stephen King did with Richard Bachman.

Breza 1y ago
Richard Bachman, you say? https://chatgpt.com/share/684c3f20-575c-800a-9ea2-889dd3deaf26 https://chatgpt.com/share/684c3f20-575c-800a-9ea2-889dd3deaf...