This is totally an issue and I am glad there is some emerging discussion. It's true that practically you can run SGD on a CPU but it is really not feasible as y

This is totally an issue and I am glad there is some emerging discussion. It's true that practically you can run SGD on a CPU but it is really not feasible as your training time might blow up by a factor of 50 and it is not unusual to train a network for a week using nvidia GPUs (and their proprietary drivers and frameworks like cudnn). At this point in time you just can't do some of the DL stuff without proprietary nvidia tech.