Is conversation data really that valuable? If I have the LLM translate a text from French to English... what is there to learn from that? Maybe the translation

Is conversation data really that valuable? If I have the LLM translate a text from French to English... what is there to learn from that? Maybe the translation is great maybe it's awful, but there's no "correct" translation to evaluate the LLM against to improve it. If I ask the chatbot for working code and it can't provide it, again, there's no "correct" code to train it against found in the conversation. If I ask an LLM to interpret a bible passage, whether it does a good job or a terrible job there's no "correct" answer that the provider has to use as the gold standard, just the noise of people chatting with arbitrary answers.