Actually 2012 was when GPU training was just taking off. A team from the University of Toronto entered one of the larger competitions and won by a large margin

Actually 2012 was when GPU training was just taking off. A team from the University of Toronto entered one of the larger competitions and won by a large margin by using GPU training. They used 2 GTX 580 over the course of 6 days to train their network on millions of images. Here is the original paper: http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf