> the RLHF data isn't There are open sources of RLHF data - for example, the LMSys Chatbot Arena dataset (for same prompt two different responses along with wh

> the RLHF data isn't There are open sources of RLHF data - for example, the LMSys Chatbot Arena dataset (for same prompt two different responses along with which the human preferred), ShareGPT It also isn’t hard to use existing leading AIs like GPT-4 or Claude-3 to generate synthetic RLHF datasets. They’ll put words in their terms of service to say you can’t use a dataset generated in that way to a train a competing model, but their ability to enforce those terms in practice is very open to question