I think one possible problem with this is that you don't really have the worldspace in the traditional sense, thus making many things like convincing reflection

I think one possible problem with this is that you don't really have the worldspace in the traditional sense, thus making many things like convincing reflections impossible. So probably they have to be rendered by the conventional pipeline, then augmented with the NN.