I think the big limiting factor is having it learn what the infinite number of parameters in different synths, etc actually make it sound like, and then being a

I think the big limiting factor is having it learn what the infinite number of parameters in different synths, etc actually make it sound like, and then being able to make the model be able to produce music by using and configuring them instead of just spaffing out crap music. If it could do that it'd give the tweakability it needs, but I think it would basically involve training a model for each instrument.