4 ms·
In this context, "performance" means "does it do what we want it to do" not "does it do it quickly". Quality of output is what they're measuring, speed is not a
by yorwba 9mo ago
In this context, "performance" means "does it do what we want it to do" not "does it do it quickly". Quality of output is what they're measuring, speed is not a consideration.
- vidarh 9mo agoThe point is that whether it does what you tell it in a single iteration is less important then whether it avoids stupid mistakes. Any serious use will put it in a harness.
- yorwba 9mo agoMy point is that you misread the comment you replied to. (By the way, on page 2 of the paper: "we evaluate each LLM only within its corresponding harness.")
- vidarh 9mo ago> My point is that you misread the comment you replied to. I'm not the person you replied to. > (By the way, on page 2 of the paper: "we evaluate each LLM only within its corresponding harness.") That has zero relevance to my comment or to the type of harnesses I talked about in the comment you replied to, nor in my comment up-thread.
- yorwba 9mo agoThe only people I have replied to in this thread were vidarh, vidarh, and now vidarh again. I thought you were all the same person?