Having developed similar systems in the past, I would like to advice against arbitrary boundaries on accuracy. We used a couple of metrics to assess the qualit

Having developed similar systems in the past, I would like to advice against arbitrary boundaries on accuracy. We used a couple of metrics to assess the quality of the predictions. The first one is sMAPE (Scaled Mean Absolute Precision Error), which tells us quickly where and how our predictions fail. We plot this with the precision on the y-axis and the minutes till arrival on x. Also plotted is bias, which is important, especially since you want to be slightly biased towards being too early. Similar axes. Other metrics we used are MAE (Mean Absolute Error), RSE (Root Square Error) and 2D kernel density plots. In the end, for a contract, I would take into account the density of use of these predictions. It's nice that you can predict perfectly during the middle of the night, but if that line is not used, it is next to useless. So something like sMAPE * passengers or sth like that. Also, even though I cannot relay this one-to-one to the passengers, a confidence interval on predictions is gold.