I think that's misunderstanding of what the Open Source Definition refers to. The way I read it, you must be allowed to run the binary to further any goal. But

I think that's misunderstanding of what the Open Source Definition refers to. The way I read it, you must be allowed to run the binary to further any goal. But it doesn't mean you can share the source code however you want. Most FOSS licenses add restrictions on that, that's why there are so many of them. To be fair OP might need to reword a little to clarify that you can use the binary to accomplish training but doesn't allow you to create derived works of the source or binary by training on them.