I think it's you who missed the point. If your algorithm requires any serial coordination between execution units, you're hosed, whether you're using CPU, GPU

I think it's you who missed the point. If your algorithm requires any serial coordination between execution units, you're hosed, whether you're using CPU, GPU or really anything that can be used for computation. There are a set of algorithms that are "embarrassingly parallel", and that's where for example many graphics tasks fall. Same is not true in general.