A good way to get around the long iteration cycles is to develop on subsets of the data or simpler tasks. With computer vision you can either start with somethi

A good way to get around the long iteration cycles is to develop on subsets of the data or simpler tasks. With computer vision you can either start with something like MNIST (which might not always be ideal because you'll need to adjust your architecture) or take a subset of about a 1000 examples from your real training set and get it to overfit. Another thing that helps is having a way to queue up multiple experiments. It's not that hard to set up with celery. Also I'd really recommend starting with the dumbest baseline model possible and getting the evaluation pipeline around it working. You can then focus on iterating on it and measure how much each change moves the needle.