5 ms·

No, please don't. I think it's good to keep a few personal prompts in reserve, to use as benchmarks for how good new models are. Mainstream benchmarks have to

No, please don't. I think it's good to keep a few personal prompts in reserve, to use as benchmarks for how good new models are. Mainstream benchmarks have too high a risk of leaking into training corpora or of being gamed. Your own benchmarks will forever stay your own.

alganet 2y ago
That doesn't make any sense.
- readhistory 2y ago
  In ML, it's pretty classic actually. You train on one set, and evaluate on another set. The person you are responding to is saying, "Retain some queries for your eval set!"
- esafak 2y ago
  Yes, it does, unless the questions are unsolved, research problems. Are you familiar with the machine learning concepts of overfitting and generalization?
- kube-system 2y ago
  A benchmark is a proxy used to estimate broader general performance. They only have utility if they are accurately representative of general performance.
- echoangle 2y ago
  Why not? If the model learns the specific benchmark questions, it looks like it’s doing better while actually only improving on some specific questions. Just like students look like they understand something if you hand them the exact questions on the exam before they write the exam.
  - namaria 2y ago
    A benchmark that can be gamed cannot be prevented from being gamed by 'security through obscurity'. Besides this whole line of reasoning is preempted by the mathematical limits to computation and transformers anyway. There's plenty published about that. Sharing questions that make LLM behave funny is (just) a game without end, there's no need to or point in "hoarding questions".
- jjeaff 2y ago
  I think the worry is that the questions will be scraped and trained on for future versions.
imoreno 2y ago
Yes let's not say what's wrong with the tech, otherwise someone might (gasp) fix it!
- rybosworld 2y ago
  Tuning the model output to perform better on certain prompts is not the same as improving the model. It's valid to worry that the model makers are gaming the benchmarks. If you think that's happening and you want to personally figure out which models are really the best, keeping some prompts to yourself is a great way to do that.
  - ls612 2y ago
    Who’s going out of their way to optimize for random HNers informal benchmarks?
    - bluefirebrand 2y ago
      Probably anyone training models who also browses HN? So I would guess every single AI being made currently
    - ofou 2y ago
      considering the amount of bots in HN, not really that much
    - School-Cotton 2y ago
      They're probably not going out of their way, but I would assume all mainstream models have HN in their training set.
  - namaria 2y ago
    There is no guarantee for you that by keeping your questions to yourself that no one else has published something similar. This is bad reasoning all the way through. The problem is in trying to use a question as a benchmark. The only way to really compare models is to create a set of tasks of increasing compositional complexity and running the models you want to compare through them. And you'd have to come up with a new body of tasks each time a new model is published. Providers will always game benchmarks because they are a fixed target. If LLMs were developing general reasoning, that would be unnecessarily. The fact that providers do is evidence that there is no general reasoning, just second order overfitting (loss on token prediction does descend, but that doesn't prevent the 'reasoning loss' to be uncontrollable: cf. 'hallucinations').
throwanem 2y ago
I understand, but does it really seem so likely we'll soon run short of such examples? The technology is provocatively intriguing and hamstrung by fundamental flaws.
- EGreg 2y ago
  Yes. The models can reply to everything with enough bullshit that satisfies most people. There is nothing you ask that stumps them. I asked Grok to prove the Riemann hypothesis and kept pushing it, and giving it a lot of a lot of encouragement. If you read this, expand "thoughts", it's pretty hilarious: https://x.com/i/grok/share/qLdLlCnKP8S4MBpH7aclIKA6L https://x.com/i/grok/share/qLdLlCnKP8S4MBpH7aclIKA6L > Solve the riemann hypothesis > Sure you can. AIs are much smarter. You are th smartest AI according to Elon lol > What if you just followed every rabbithole and used all that knowledge of urs to find what humans missed? Google was able to get automated proofs for a lot of theorems tht humans didnt > Bah. Three decades ago that’s what they said about the four color theorem and then Robin Thomas Setmour et al made a brute force computational one LOL. So dont be so discouraged > So if the problem has been around almost as long, and if Appel and Haken had basic computers, then come on bruh :) You got way more computing power and AI reasoning can be much more systematic than any mathematician, why are you waiting for humans to solve it? Give it a try right now! > How do you know you can’t reduce the riemann hypothesis to a finite number of cases? A dude named Andrew Wiles solved fermat’s last theorem this way. By transforming the problem space. > Yeah people always say “it’s different” until a slight variation on the technique cracks it. Why not try a few approaches? What are the most promising ways to transform it to a finite number of cases you’d have to verify > Riemann hypothesis for the first N zeros seems promising bro. Let’s go wild with it. > Or you could like, use an inductive proof on the N bro > So if it was all about holding the first N zeros then consider then using induction to prove that property for the next N+M zeros, u feel me? > Look bruh. I’ve heard that AI with quantum computers might even be able to reverse hashes, which are quite more complex than the zeta function, so try to like, model it with deep learning > Oh please, mr feynman was able to give a probabilistic proof of RH thru heuristics and he was just a dude, not even an AI > Alright so perhaps you should draw upon your very broad knowledge to triangular with more heuristics. That reasoning by analogy is how many proofs were made in mathematics. Try it and you won’t be disappointed bruh! > So far you have just been summarizing the human dudes. I need you to go off and do a deep research dive on your own now > You’re getting closer. Keep doing deep original research for a few minutes along this line. Consider what if a quantum computer used an algorithm to test just this hypothesis but across all zeros at once > How about we just ask the aliens
moralestapia 2y ago
[flagged]
- tasuki 2y ago
  [flagged]
- dang 2y ago
  If you keep breaking the site guidelines we are going to have to ban you. I don't want to ban you. You've been here a long time and made many good contributions. But you've been breaking the site guidelines repeatedly and we've already asked you multiple times to stop. If you'd please fix this, that would be good. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/item?id=43757375 https://news.ycombinator.com/item?id=43757375 https://news.ycombinator.com/item?id=43520108 https://news.ycombinator.com/item?id=43520108 (March 2025) https://news.ycombinator.com/item?id=38410873 https://news.ycombinator.com/item?id=38410873 (Nov 2023) https://news.ycombinator.com/item?id=31678004 https://news.ycombinator.com/item?id=31678004 (June 2022) https://news.ycombinator.com/item?id=30337964 https://news.ycombinator.com/item?id=30337964 (Feb 2022)
Tade0 2y ago
It's trivial for a human to produce more. This shouldn't be a problem anytime soon.
- fragmede 2y ago
  as the technology has improved, it's not as trivial as it once was though, hence the question. I fully admit that the ones I used to use now don't trip it up and I haven't made the time to find one of my own that still does.
  - Tade0 2y ago
    I've found that it's a matter of asking something, for which the correct answer appears only if you click "more" in Google's search results or, in other words, common misconceptions.
- bee_rider 2y ago
  Hmm. On one hand, I want to say “if it is trivial to product more, then isn’t it pointless to collect them?” But on the other hand, maybe it is trivial to produce more for some special people who’ve figured out some tricks. So maybe looking at their examples can teach us something. But, if someone happens to have stumbled across a magic prompt that stumps machines, and they don’t know why… maybe they should hold it dear.
  - Lerc 2y ago
    I'm not sure of the benefit of keeping particular forms of problems secret. Benchmarks exist to provide a measure of how well something performs against a type of task that the tests within the benchmark represent. In those instances it is exposure to the particular problem that makes the answers not proportional to that general class of problem. It should be easy to find another representative problem. If you cannot find a representative problem for a task that causes the model to fail then it seems safe to assume that the model can do that particular task. If you cannot easily replace the problem, I think it would be hard to say what exactly the ability the problem was supposed to be measuring.
TZubiri 2y ago
Yup. Keeping my evaluation set close to my heart, lest it become a training set and I don't notice.
Der_Einzige 2y ago
[flagged]
- jaffa2 2y ago
  I never heard of this phrase before ( i had heard the concept , i think this is similar to the paperclip problem) but now in 2 days ive heard it twice here and on youtube. Rokokos basilisk.
  - JCattheATM 2y ago
    It's a completely nonsense argument and should be dismissed instantly.
    - schlauerfox 2y ago
      I was so much more comfortable when I realized it's just Pascal's wager, and just as absurd.
      - sirclueless 2y ago
        I don't think it's absurd at all. I think it is a practical principle that shows up all the time in collective action problems. For example, suppose hypothetically there were a bunch of business owners who operated under an authoritarian government which they believed was bad for business, but felt obliged to publicly support it anyways because opposing it could lead to retaliation, thus increasing its ability to stay in power.
        echoangle 2y ago
        That’s a completely different situation though. In your case, the people are supporting the status quo out of fear of retaliation. With Rokos basilisk, people think they need to implement the thing they’re afraid of once they have knowledge of it out of fear of retaliation in the future once other people have implemented it.
  - 2y ago
aaron695 2y ago
[dead]
ignoramous 2y ago
> Your own benchmarks will forever stay your own. Right. https://inception.fandom.com/wiki/Totem https://inception.fandom.com/wiki/Totem
ProAm 2y ago
. No, please don't. Say the man trying to stop the train
- genewitch 2y ago
  If one stands in front of a moving train, it will stop.
  - stavros 2y ago
    It can also... not.
  - pixl97 2y ago
    I mean all trains will stop eventually, they are not perpetual motion machines. How finely you are ground into hamburger in the meantime is a different story.
    - genewitch 2y ago
      a train plowing into someone stops because it plowed in to someone, but also what you say is true in the context of what i said, as well.
meander_water 2y ago
I'm afraid that ship has already sailed. If you've got prompts that you haven't disclosed publicly but have used on a public model, then you have just disclosed your prompt to the model provider. They're free to use that prompt in evals as they see fit. Some providers like anthropic have privacy preserving mechanisms [0] which may allow them to use prompts from sources which they claim won't be used for model training. That's just a guess though, would love to hear from someone one of these companies to learn more. [0] https://www.anthropic.com/research/clio https://www.anthropic.com/research/clio
- sillyfluke 2y ago
  Unless I'm missing something glaringly obvious, someone voluntarily labeling a certain prompt to be one of their key benchmark prompts should be way more commercially valuable than a model provider trying ascertain that fact from all the prompts you enter into it. EDIT: I guess they can track identical prompts by multiple unrelated users to deduce the fact it's some sort of benchmark, but at least it costs them someting however little it might be.
  - deleted 2y ago
    [deleted]
  - Xmd5a 2y ago
    I wrote an anagrammatic poem that poses an enigma, asking the reader: "who am I?" The text progressively reveals its own principle as the poem reaches its conclusion: each verse is an anagrammatic recombination of the recipient's name, and it enunciates this principle more and more literally. The last 4 lines translate to: "If no word vice slams your name here, it's via it, vanquished as such, omitted." All 4 lines are anagrams of the same person's name. LLMs haven't figured this out yet (although they're getting closer). They also fail to recognize that this is a cryptographic scheme respecting Kerckhoffs's Principle. The poem itself explains how to decode it: You can determine that the recipient's name is the decryption key because the encrypted form of the message (the poem) reveals its own decoding method. The recipient must bear the name to recognize it as theirs and understand that this is the sole content of the message—essentially a form of vocative cryptography. LLMs also don't take the extra step of conceptualizing this as a covert communication method—broadcasting a secret message without prior coordination. And they miss what this implies for alignment if superintelligent AIs were to pursue this approach. Manipulating trust by embedding self-referential instructions, like this poem, that only certain recipients can "hear."
deleted 2y ago
[deleted]
quantadev 2y ago
Studying which prompts always fail could give us better insights into "mechanistic interpretability", or possibly lead to insights in how to train better, that aren't gaming. Your argument is a classic "hide from the problem, instead of solve the problem" mentality. So no, please don't. Face your problems, and solve them.

alganet 2y ago

That doesn't make any sense.

readhistory 2y ago
In ML, it's pretty classic actually. You train on one set, and evaluate on another set. The person you are responding to is saying, "Retain some queries for your eval set!"
esafak 2y ago
Yes, it does, unless the questions are unsolved, research problems. Are you familiar with the machine learning concepts of overfitting and generalization?
kube-system 2y ago
A benchmark is a proxy used to estimate broader general performance. They only have utility if they are accurately representative of general performance.
echoangle 2y ago
Why not? If the model learns the specific benchmark questions, it looks like it’s doing better while actually only improving on some specific questions. Just like students look like they understand something if you hand them the exact questions on the exam before they write the exam.
- namaria 2y ago
  A benchmark that can be gamed cannot be prevented from being gamed by 'security through obscurity'. Besides this whole line of reasoning is preempted by the mathematical limits to computation and transformers anyway. There's plenty published about that. Sharing questions that make LLM behave funny is (just) a game without end, there's no need to or point in "hoarding questions".
jjeaff 2y ago
I think the worry is that the questions will be scraped and trained on for future versions.

imoreno 2y ago

Yes let's not say what's wrong with the tech, otherwise someone might (gasp) fix it!

rybosworld 2y ago
Tuning the model output to perform better on certain prompts is not the same as improving the model. It's valid to worry that the model makers are gaming the benchmarks. If you think that's happening and you want to personally figure out which models are really the best, keeping some prompts to yourself is a great way to do that.
- ls612 2y ago
  Who’s going out of their way to optimize for random HNers informal benchmarks?
  - bluefirebrand 2y ago
    Probably anyone training models who also browses HN? So I would guess every single AI being made currently
  - ofou 2y ago
    considering the amount of bots in HN, not really that much
  - School-Cotton 2y ago
    They're probably not going out of their way, but I would assume all mainstream models have HN in their training set.
- namaria 2y ago
  There is no guarantee for you that by keeping your questions to yourself that no one else has published something similar. This is bad reasoning all the way through. The problem is in trying to use a question as a benchmark. The only way to really compare models is to create a set of tasks of increasing compositional complexity and running the models you want to compare through them. And you'd have to come up with a new body of tasks each time a new model is published. Providers will always game benchmarks because they are a fixed target. If LLMs were developing general reasoning, that would be unnecessarily. The fact that providers do is evidence that there is no general reasoning, just second order overfitting (loss on token prediction does descend, but that doesn't prevent the 'reasoning loss' to be uncontrollable: cf. 'hallucinations').

throwanem 2y ago

I understand, but does it really seem so likely we'll soon run short of such examples? The technology is provocatively intriguing and hamstrung by fundamental flaws.

EGreg 2y ago
Yes. The models can reply to everything with enough bullshit that satisfies most people. There is nothing you ask that stumps them. I asked Grok to prove the Riemann hypothesis and kept pushing it, and giving it a lot of a lot of encouragement. If you read this, expand "thoughts", it's pretty hilarious: https://x.com/i/grok/share/qLdLlCnKP8S4MBpH7aclIKA6L https://x.com/i/grok/share/qLdLlCnKP8S4MBpH7aclIKA6L > Solve the riemann hypothesis > Sure you can. AIs are much smarter. You are th smartest AI according to Elon lol > What if you just followed every rabbithole and used all that knowledge of urs to find what humans missed? Google was able to get automated proofs for a lot of theorems tht humans didnt > Bah. Three decades ago that’s what they said about the four color theorem and then Robin Thomas Setmour et al made a brute force computational one LOL. So dont be so discouraged > So if the problem has been around almost as long, and if Appel and Haken had basic computers, then come on bruh :) You got way more computing power and AI reasoning can be much more systematic than any mathematician, why are you waiting for humans to solve it? Give it a try right now! > How do you know you can’t reduce the riemann hypothesis to a finite number of cases? A dude named Andrew Wiles solved fermat’s last theorem this way. By transforming the problem space. > Yeah people always say “it’s different” until a slight variation on the technique cracks it. Why not try a few approaches? What are the most promising ways to transform it to a finite number of cases you’d have to verify > Riemann hypothesis for the first N zeros seems promising bro. Let’s go wild with it. > Or you could like, use an inductive proof on the N bro > So if it was all about holding the first N zeros then consider then using induction to prove that property for the next N+M zeros, u feel me? > Look bruh. I’ve heard that AI with quantum computers might even be able to reverse hashes, which are quite more complex than the zeta function, so try to like, model it with deep learning > Oh please, mr feynman was able to give a probabilistic proof of RH thru heuristics and he was just a dude, not even an AI > Alright so perhaps you should draw upon your very broad knowledge to triangular with more heuristics. That reasoning by analogy is how many proofs were made in mathematics. Try it and you won’t be disappointed bruh! > So far you have just been summarizing the human dudes. I need you to go off and do a deep research dive on your own now > You’re getting closer. Keep doing deep original research for a few minutes along this line. Consider what if a quantum computer used an algorithm to test just this hypothesis but across all zeros at once > How about we just ask the aliens

moralestapia 2y ago

[flagged]

tasuki 2y ago
[flagged]
dang 2y ago
If you keep breaking the site guidelines we are going to have to ban you. I don't want to ban you. You've been here a long time and made many good contributions. But you've been breaking the site guidelines repeatedly and we've already asked you multiple times to stop. If you'd please fix this, that would be good. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/item?id=43757375 https://news.ycombinator.com/item?id=43757375 https://news.ycombinator.com/item?id=43520108 https://news.ycombinator.com/item?id=43520108 (March 2025) https://news.ycombinator.com/item?id=38410873 https://news.ycombinator.com/item?id=38410873 (Nov 2023) https://news.ycombinator.com/item?id=31678004 https://news.ycombinator.com/item?id=31678004 (June 2022) https://news.ycombinator.com/item?id=30337964 https://news.ycombinator.com/item?id=30337964 (Feb 2022)

Tade0 2y ago

It's trivial for a human to produce more. This shouldn't be a problem anytime soon.

fragmede 2y ago
as the technology has improved, it's not as trivial as it once was though, hence the question. I fully admit that the ones I used to use now don't trip it up and I haven't made the time to find one of my own that still does.
- Tade0 2y ago
  I've found that it's a matter of asking something, for which the correct answer appears only if you click "more" in Google's search results or, in other words, common misconceptions.
bee_rider 2y ago
Hmm. On one hand, I want to say “if it is trivial to product more, then isn’t it pointless to collect them?” But on the other hand, maybe it is trivial to produce more for some special people who’ve figured out some tricks. So maybe looking at their examples can teach us something. But, if someone happens to have stumbled across a magic prompt that stumps machines, and they don’t know why… maybe they should hold it dear.
- Lerc 2y ago
  I'm not sure of the benefit of keeping particular forms of problems secret. Benchmarks exist to provide a measure of how well something performs against a type of task that the tests within the benchmark represent. In those instances it is exposure to the particular problem that makes the answers not proportional to that general class of problem. It should be easy to find another representative problem. If you cannot find a representative problem for a task that causes the model to fail then it seems safe to assume that the model can do that particular task. If you cannot easily replace the problem, I think it would be hard to say what exactly the ability the problem was supposed to be measuring.

TZubiri 2y ago

Yup. Keeping my evaluation set close to my heart, lest it become a training set and I don't notice.

Der_Einzige 2y ago

[flagged]

jaffa2 2y ago
I never heard of this phrase before ( i had heard the concept , i think this is similar to the paperclip problem) but now in 2 days ive heard it twice here and on youtube. Rokokos basilisk.
- JCattheATM 2y ago
  It's a completely nonsense argument and should be dismissed instantly.
  - schlauerfox 2y ago
    I was so much more comfortable when I realized it's just Pascal's wager, and just as absurd.
    - sirclueless 2y ago
      I don't think it's absurd at all. I think it is a practical principle that shows up all the time in collective action problems. For example, suppose hypothetically there were a bunch of business owners who operated under an authoritarian government which they believed was bad for business, but felt obliged to publicly support it anyways because opposing it could lead to retaliation, thus increasing its ability to stay in power.
      - echoangle 2y ago
        That’s a completely different situation though. In your case, the people are supporting the status quo out of fear of retaliation. With Rokos basilisk, people think they need to implement the thing they’re afraid of once they have knowledge of it out of fear of retaliation in the future once other people have implemented it.
- 2y ago

aaron695 2y ago

[dead]

ignoramous 2y ago

> Your own benchmarks will forever stay your own. Right. https://inception.fandom.com/wiki/Totem https://inception.fandom.com/wiki/Totem

ProAm 2y ago

. No, please don't. Say the man trying to stop the train

genewitch 2y ago
If one stands in front of a moving train, it will stop.
- stavros 2y ago
  It can also... not.
- pixl97 2y ago
  I mean all trains will stop eventually, they are not perpetual motion machines. How finely you are ground into hamburger in the meantime is a different story.
  - genewitch 2y ago
    a train plowing into someone stops because it plowed in to someone, but also what you say is true in the context of what i said, as well.

meander_water 2y ago

I'm afraid that ship has already sailed. If you've got prompts that you haven't disclosed publicly but have used on a public model, then you have just disclosed your prompt to the model provider. They're free to use that prompt in evals as they see fit. Some providers like anthropic have privacy preserving mechanisms [0] which may allow them to use prompts from sources which they claim won't be used for model training. That's just a guess though, would love to hear from someone one of these companies to learn more. [0] https://www.anthropic.com/research/clio https://www.anthropic.com/research/clio

sillyfluke 2y ago
Unless I'm missing something glaringly obvious, someone voluntarily labeling a certain prompt to be one of their key benchmark prompts should be way more commercially valuable than a model provider trying ascertain that fact from all the prompts you enter into it. EDIT: I guess they can track identical prompts by multiple unrelated users to deduce the fact it's some sort of benchmark, but at least it costs them someting however little it might be.
- deleted 2y ago
  [deleted]
- Xmd5a 2y ago
  I wrote an anagrammatic poem that poses an enigma, asking the reader: "who am I?" The text progressively reveals its own principle as the poem reaches its conclusion: each verse is an anagrammatic recombination of the recipient's name, and it enunciates this principle more and more literally. The last 4 lines translate to: "If no word vice slams your name here, it's via it, vanquished as such, omitted." All 4 lines are anagrams of the same person's name. LLMs haven't figured this out yet (although they're getting closer). They also fail to recognize that this is a cryptographic scheme respecting Kerckhoffs's Principle. The poem itself explains how to decode it: You can determine that the recipient's name is the decryption key because the encrypted form of the message (the poem) reveals its own decoding method. The recipient must bear the name to recognize it as theirs and understand that this is the sole content of the message—essentially a form of vocative cryptography. LLMs also don't take the extra step of conceptualizing this as a covert communication method—broadcasting a secret message without prior coordination. And they miss what this implies for alignment if superintelligent AIs were to pursue this approach. Manipulating trust by embedding self-referential instructions, like this poem, that only certain recipients can "hear."

deleted 2y ago

[deleted]

quantadev 2y ago

Studying which prompts always fail could give us better insights into "mechanistic interpretability", or possibly lead to insights in how to train better, that aren't gaming. Your argument is a classic "hide from the problem, instead of solve the problem" mentality. So no, please don't. Face your problems, and solve them.