The network that you train in that tutorial is quite small. Your networks have to get quite large before the overhead of transferring the network onto the GPU i

The network that you train in that tutorial is quite small. Your networks have to get quite large before the overhead of transferring the network onto the GPU is negligible to the cost of training it. You'll see the benefit of the GPU with larger models.