8 ms·
> Testers preferred o3-mini's responses to o1-mini 56% of the time I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefe
by jen729w 2y ago
> Testers preferred o3-mini's responses to o1-mini 56% of the time
I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'.
They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both; I'm usually just selecting the one that answered first.
That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a coin-flip.
- danielmarkbruce 2y agoRLUHF, U = useless.
- brookst 2y agoThose prompts are so irritating and so frequent that I’ve taken to just quickly picking whichever one looks worse at a cursory glance. I’m paying them, they shouldn’t expect high quality work from me.
- deleted 2y ago[deleted]
- apparent 2y agoHave you considered the possibility that your feedback is used to choose what type of response to give to you specifically in the future? I would not consider purposely giving inaccurate feedback for this reason alone.
- isaacremuant 2y agoAlternatively, I'll use the tool that is most user friendly and provides the most value for my money. Wasting time on an anti pattern is not value nor is it trying to outguess the way that selection mechanism is used.
- MattDaEskimo 2y agoI don't want a model that's customized to my preferences. My preferences and understanding changes all the time. I want a single source model that's grounded in base truth. I'll let the model know how to structure it in my prompt.
- szundi 2y agoConstang meh and fixing prompts to the right direction vs unable to escape the bubble
- kenjackson 2y agoYou know there's no such as base truth here? You want to write something like this to start your prompts, "Respond in English, using standard capitalization and punctuation, following rules of grammar as written by Strunk & White, where numbers are represented using arabic numerals in base 10 notation...."???
- AutistiCoder 2y agoactually, I might appreciate that. I like precision of language, so maybe just have a system prompt that says "use precise language (ex: no symbolism of any kind)"
- MattGaiser 2y agoA lot of preferences have nothing to do with any truth. Do you like code segments or full code? Do you like paragraphs or bullet points? Heck, do you want English or Japanaese?
- orbital-decay 2y agoWhat is base truth for e.g. creative writing?
- francis_lewis 2y agoI think my awareness that this may influence future responses has actually been detrimental to my response rate. The responses are often so similar that I can imagine preferring either in specific circumstances. While I’m sure that can be guided by the prompt, I’m often hesitant to click on a specific response as I can see the value of the other response in a different situation and I don’t want to bias the future responses. Maybe with more specific prompting this wouldn’t be such an issue, or maybe more of an understanding of how inter-chat personalisation is applied (maybe I’m missing some information on this too).
- Der_Einzige 2y agoSpotted the pissed off OpenAI RLHF engineer! Hahahahaha!
- Tenoke 2y agoThat's such a counter-productive and frankly dumb thing to do. Just don't vote on them.
- explain 2y agoYou have to pick one to continue the chat.
- apparent 2y agoWhy not always pick the one on the left, for example? I understand wanting to speed through and not spend time doing labor for OpenAI, but it seems counter-productive to spend any time feeding it false information.
- brookst 2y agoMy assumption is they measure the quality of user feedback, either on a per user basis or in an aggregate. I want them to interrupt me less, so I want them to either decide I’m a bad teacher or that users in general are bad teachers.
- deleted 2y ago[deleted]
- DiggyJohnson 2y agoI know for a fact that as of yesterday I did not have to pick one to continue the conversation. It just maximizes the second choice and displayed a 2/2 below the response.
- jackbrookes 2y agoYes I'd bet most users just 50/50 it, which actually makes it more remarkable that there was a 56% selection rate
- cgriswald 2y agoI read the one on the left but choose the shorter one. The interface wastes so much screen real estate already and the answers are usually overly verbose unless I've given explicit instructions on how to answer.
- ljm 2y agoThe default level of verbosity you get without explicitly prompting for it to be succinct makes me think there’s an office full of workers getting paid by the token.
- internetter 2y agoIn my experience the verbosity significantly improves output quality
- deleted 2y ago[deleted]
- johnneville 2y agothey also pay contractors to do these evaluations with much more detailed metrics, no idea which their number is based on though
- dkjaudyeqooe 2y agoIt's kind of strange that they gave that stat. Maybe they thought people would somehow think about "56% better" or something. Because when you think about it, it really is quite damning. Minus statistical noise it's no better.
- fsndz 2y agoexactly I was surprised as well
- Powdering7082 2y agoThat would be 12%, why would you assume that is eaten by statistical noise?
- senorrib 2y agoThe OPs comment is probably a testament of that. With such a poorly designed A/B test I doubt this has a p-value of < 0.10.
- throwaway287391 2y agoErm, why not? A 0.56 result with n=1000 ratings is statistically significantly better than 0.5 with a p-value of 0.00001864, well beyond any standard statistical significance threshold I've ever heard of. I don't know how many ratings they collected but 1000 doesn't seem crazy at all. Assuming of course that raters are blind to which model is which and the order of the 2 responses is randomized with every rating -- or, is that what you meant by "poorly designed"? If so, where do they indicate they failed to randomize/blind the raters?
- n2d4 2y agoBecause you're not testing "will a user click the left or right button" (for which asking a thousand users to click a button would be a pretty good estimation), you're testing "which response is preferred". If 10% of people just click based on how fast the response was because they don't want to read both outputs, your p-value for the latter hypothesis will be atrocious, no matter how large the sample is.
- mikeInAlaska 2y agoMaybe we should take both answers, paste them into a new chat and ask for a summary amalgamation of them
- sharkweek 2y agoFunny - I had ChatGPT document some stuff for me this week and asked which responses I preferred as well. Didn’t bother reading either of them, just selected one and went on with my day. If it were me I would have set up a “hey do you mind if we give you two results and you can pick your favorite?” prompt to weed out people like me.
- usef- 2y agoI'm surprised how many people claim to do this. You can just not select one.
- rubyn00bie 2y agoI think it’s somewhat natural and am not personally surprised. It’s easy to quickly select an option, that has no consequence, compared to actively considering that not selecting something is an option. Not selecting something feels more like actively participating than just checking a box and moving on. /shrug
- ssl-3 2y agoWe -- the people who live in front of a computer -- have been training ourselves to avoid noticing annoyances like captchas, advertising, and GDPR notices for quite a long time. We find what appears to be the easiest combination "Fuck off, go away" buttons and use them without a moment of actual consideration. (This doesn't mean that it's actually the easiest method.)
- grahamj 2y agoI can't even believe how many times in a day I frustratedly think "whatever, go away!"
- apparent 2y agoI wonder if they down-weight responses that come in too fast to be meaningful, or without sufficient scrolling.
- 2y ago
- danilocesar 2y agoI almost always pick the second one, because it's closer to the submit button and the one I read first.
- arijo 2y agoPeople could be flipping a coin and the score would be the same.
- brianstrimp 2y agoA 12% margin is literally the opposite of a coin flip. Unless you have a really bad coin.
- buggy6257 2y agoYou're being downvoted for 3 reasons: 1) Coming off as a jerk, and from a new account is a bad look 2) "Literally the opposite of a coin flip" would probably be either 0% or 100% 3) Your reasoning doesn't stand up without further info; it entirely depends on the sample size. I could have 5 coin flips all come up heads, but over thousands or millions it averages to 50%. 56% on a small sample size is absolutely within margin of error/noise. 56% on a MASSIVE sample size is _statistically_ significant, but isn't even still that much to brag about for something that I feel like they probably intended to be a big step forward.
- brianstrimp 2y agoI'm a little puzzled by your response. 1. The message was net-upvoted. Whether there are downvotes in there I can't tell, but the final karma is positive. A similarly spirited message of mine in the same thread was quite well receive as well. 2. I can't see how my message would come across as a jerk? I wrote 2 simple sentences, not using any offensive language, stating a mere fact of statistics. Is that being jerk? And a long-winded berating of a new member of the community isn't? 3. A coin flip is 50%. Anything else is not, once you have a certain sample size. So, this was not. That was my statement. I don't know why you are building a strawman of 5 coin flips. 56% vs 44% is a margin of 12%, as I stated, and with a huge sample size, which they had, that's massive in a space where the returns are deep in "diminishing" territory.
- arijo 2y agoI wasn't expecting for my comment to be red so literally but ok. We're talking about the most cost-efficient model, the competition here is on price, not on a 12% incremental performance (which would make sense for the high end model). To my knowledge deepseek is the cheaper service which is what matters on the low-end (unless the increase in performance was in such magnitude that the extra-charge would be worth the money).
- deleted 2y ago[deleted]
- teeray 2y agoThis prompt is like "See Attendant" on the gas pump. I'm just going to use another AI instead for this chat.
- ninkendo 2y agoGlad to know I’m not the only person who just drives to the next station when I see a “see attendant” message.
- janalsncm 2y ago> I'm usually just selecting the one that answered first Which is why you randomize the order. You aren’t a tester. 56% vs 44% may not be noise. That’s why we have p values. It depends on sample size.
- jhardy54 2y agoThe order doesn't matter. They often generate tokens at different speeds, and produce different lengths of text. "The one that answered first" != "The first option"
- letmevoteplease 2y agoThe article says "expert testers." "Evaluations by expert testers showed that o3-mini produces more accurate and clearer answers, with stronger reasoning abilities, than OpenAI o1-mini. Testers preferred o3-mini's responses to o1-mini 56% of the time and observed a 39% reduction in major errors on difficult real-world questions. W"
- threatripper 2y agoThose are two different sentences. The second sentence doesn't refer to experts explicitly.
- resters 2y agoI too have questioned the approach of showing the long side-by-side answers from two different models. 1) sometimes I wanted the short answer, and so even though the long answer is better I picked the short one. 2) sometimes both contain code that is different enough that I am inclined to go with the one that is more similar to what I already had, even if the other approach seems a bit more solid. 3) Sometimes one will have less detail but more big picture awareness and the other will have excellent detail but miss some overarching point that is valuable. Depending on my mood I sometimes choose but it is annoying to have to do so because I am not allowed to say why I made the choice. The area of human training methodology seems to be a big part of what got Deepseek's model so strong. I read the explanation of the test results as an acknowledgement by OpenAI of some weaknesses in its human feedback paradigm. IMO the way it should work is that the thumbs up or down should be read in context by a reasoning being and a more in-depth training case should be developed that helps future models learn whatever insight the feedback should have triggered. Feedback that A is better or worse than B is definitely not (in my view) sufficient except in cases where a response is a total dud. Usually the responses have different strengths and weaknesses and it's pretty subjective which one is better.
- brianstrimp 2y agoThat makes the result stronger though. Even though many people click randomly, there is still a 12% margin between both groups. Not the world, but still quite a lot.
- dionian 2y agoi enjoy it, i like getting two answers for free - often one of them is significantly better and probably the newer model
- mullingitover 2y agoYou know you can configure default instructions to your prompts, right? I have something like “always be terse and blunt with your answers.”
- gcanyon 2y agoI don't think they make it clear: I wonder if they mean testers prefer o3 mini 56% of the time when they express an opinion, or overall? Some percentage of people don't choose; if that number is 10% and they aren't excluded, that means 56% of the time people prefer o3 mini, 34% of the time people prefer o1 mini, and 10% of the time people don't choose. I'm not sure I think it would be reasonable to present the data that way, but it seems possible.
- ricardobeat 2y agoThis is just a way to prove, statistically, that one model is better than another as part of its validation. It's not collected from normal people using ChatGPT, you don't ever get shown two responses from different models at once.
- directevolve 2y agoAlso, it's not clear if the preference comes from the quality of the 'meat' of the answer, or the way it reports its thinking and the speed with which it responds. With o1, I get a marked feeling of impatience waiting for it to spit something out, and the 'progress of thought' is in faint grey text I can't read. With o3, the 'progress of thought' comes quickly, with more to read, and is more engaging even if I don't actually get anything more than entertainment value. I'm not going to say there's nothing substantive about o3 vs. o1, but I absolutely do not put it past Sam Altman to juice the stats every chance he gets.
- shombaboor 2y agoIt seems like the the first response must get chosen a majority of the time just to account for friction
- energy123 2y agoThen 56% is even more impressive. Example: if 80% choose randomly and 20% choose carefully, that implies an 80% preference rate for o3-mini (0.8*0.2 + 0.5*0.8 = 0.56)