When I did analysis of hundreds of millions of predictions to analyze our prediction algorithm for trucks, I had a couple of metrics I would look at: MAE: mean

When I did analysis of hundreds of millions of predictions to analyze our prediction algorithm for trucks, I had a couple of metrics I would look at: MAE: mean absolute error: easy to understand metric. Given a stable system, this was my go-to standard. Bias: you put this one in. Generally, you want your predictions to be slightly pessimistic. RMSE: root mean square error. Most important one. Being 5 minutes off consistently is better than 30 minutes occasionally. Precision: larger trips can have larger errors. Here you divide your error by the predicted (or actual, or both) and then aggregate. Then obviously I would separate it into classes. Each 15 minute longer trip would have its own class. Highway vs suburb, region in the Netherlands, time of day and weekday.