I suppose you wouldn't train it using a traditional LLM type of training and you might not have an LLM type of model either. American Fuzzy Lop usually manages

I suppose you wouldn't train it using a traditional LLM type of training and you might not have an LLM type of model either. American Fuzzy Lop usually manages to generate valid files of any format you want, using a genetic algorithm that the author didn't even call ML. AlphaGo trained against itself. Such things aren't impossible when you can automate the reward function, though you might have to come up with novel techniques.