That's the GPU parallelization they offer, but if you had a problem where say the objective function itself was really big and cumbersome and benefited from GPU

That's the GPU parallelization they offer, but if you had a problem where say the objective function itself was really big and cumbersome and benefited from GPU parallelization, then you could put that objective function on the GPU just fine. I think this use case isn't really discussed in the paper because the solver doesn't need to specialize on that case, it should 'just work' without any intervention needed.