"The reported accuracy is simply the cumulative total number of errors divided by the number of steps." Use a moving average. I use: m <-- m - (2/t) (m - x_

"The reported accuracy is simply the cumulative total number of errors divided by the number of steps." Use a moving average. I use: m <-- m - (2/t) (m - x_t) when estimating the current training error. 1/t would be the exact historical average. 2/t gives more weight to recent events, which is good when your distribution is non-stationary (as is the case when your model is changing). With a constant learning rate (independent of t) you get an exponential moving average.