Great meta article. Some other general advice concerning iterative development: - Test your ideas on a smaller data sample at first to iterate and debug more

Great meta article. Some other general advice concerning iterative development: - Test your ideas on a smaller data sample at first to iterate and debug more rapidly. - Caching your augmented data to disk may also improve iteration speed. Or it may not, depending on kernal disk cache vs disk bandwidth vs compute tradeoffs. - Wrap your functions in timers so you can notice when a new processing stage is unnaturally slow, rather than having to back out later what happened. - Make sure you're actually setup and using the GPU and not accidentally running on the CPU.