> From a predict-the-future point of view, I’ll say that Anthropic’s progress this year precisely backs your point that taste might in fact matter. Last year I

> From a predict-the-future point of view, I’ll say that Anthropic’s progress this year precisely backs your point that taste might in fact matter. Last year I would have said safety concerns had irrevocably ruined their product, but I would have been wrong. Yes. Claude-2 underperformed ChatGPT... But you could still see that it was much more pleasant to use for creative writing and nonfiction, while not seeming to be much worse in terms of jailbreaking/safety, and so their RLAIF seemed like the right approach compared to RLHF. But meanwhile, OA appears to be in complete denial as an organization that there is anything wrong whatsoever with ChatGPT, and disinclined to 'delve' into what might be the issue with their approach. This is despite the peculiar and alarming side-effects of their RLHF, like the rhyming verse, and our general inability to understand the full implications and all side-effects of decisions like BPE or RLHF. It's not hard to imagine that these sorts of hacks might eventually limit the potential of OA's models - maybe you don't care about non-rhyming poetry, but what else is being handicapped by it? And if no benchmark told you about the rhyming thing, why do you think that they will tell you about other things? How would you know if your new models are incapable of making genuine scientific breakthroughs (which always require a lot of creativity and daring) because your RLHF was wrong and you've built up too much data & infrastructure and now have sunk-cost bias ruling everything? It's one of those Feynman things: RLHF was not supposed to destroy poetry, so the fact that it did tells you that something has gone alarmingly wrong in your understanding of what you are doing. And when an organization decides it's not interested in how its rubber seals perform in unusual conditions... And if an organization can't be honest about these sorts of things, that bodes poorly for the long run. As Karpathy says, 'neural networks want to work', even if your code has severe flaws; they will just fall perpetually well short of where they could if you fixed those hidden, silent, invisible flaws. OA could wind up puzzling over its frontier models and wondering why the $10b run didn't yield all that impressive a result and not willing to risk the $100b AGI run, while Anthropic just smoothly scales up with a much better bang for buck, and the tortoise passes the hare.