Since you are going down this route, I would recommend you guys to build some sort of unit test driven fine tuning framework, where you may provide input output

Since you are going down this route, I would recommend you guys to build some sort of unit test driven fine tuning framework, where you may provide input output examples expressed as simple function calls. You could then let the LLM generate examples and check them using the unit tests and keep the valid results to build up a valid data set. For bonus points, the unit tests themselves could also call the LLM to check if the output passes criteria expressed in natural language or not.