3 ms·
Why would the CTO/lead engineer admit that they nerfed the model even if they did? It’s all closed, how does admitting it benefit them? I would much rather trus
by polygamous_bat 2y ago
Why would the CTO/lead engineer admit that they nerfed the model even if they did? It’s all closed, how does admitting it benefit them? I would much rather trust the people using it everyday.
- refulgentis 2y agoI wouldn't recommend that, it is tempting, but leaves you self-peasantizing and avoiding learnings.
- hackerlight 2y agoIt's not a random sample of people. You're sampling the 10 most noisy people out of a million users, and those 10 people could be mistaken. Claude 3 hasn't dropped Elo on the lmsys leaderboard which supports the CTO's claim.
- CuriouslyC 2y agoBeyond that, to people who interact with the models regularly the "nerf" issue is pretty obvious. It was pretty clear when a new model rollout caused ChatGPT4 to try and stick to the "leadup, answer, explanation" response model and also start to get lazy about longer responses.
- swores 2y agoThat's a different company's model, so while it may have been obvious it is not relevant to whether Claude 3 has been nerfed or not is it?
- CuriouslyC 2y agoI use claude3 opus daily and I haven't noticed a change in its outputs, I think it's more likely that there's a discontinuity in the inputs the user is providing to claude which is tipping it over a threshold into a response type they find incorrect. When GPT4 got lobotomized, you had to work hard to avoid the new behavior, it popped up everywhere. People claiming claude got lobotomized seem to be cherry picking example.
- swores 2y agoOh my bad, sorry, I misinterpreted your previous comment as meaning "it was obvious with GPT4 and therefore if people say the same about Claude 3 it must equally be obvious and true", rather than what you meant which was half the opposite.