The problem with that it only tells us how good they were on election day. It'd be great to somehow evaluate their earlier predictions, which is probably only p

The problem with that it only tells us how good they were on election day. It'd be great to somehow evaluate their earlier predictions, which is probably only possible by looking at an aggregate of election. (or maybe, since there are predictions and results for every state and every senate race, it's actually possible to use those for an evaluation. Note that it's not as easy as scoring a "won by trump" in favor of a forecaster who was giving trump a 60% chance. You need a measure that expects your 60% predictions to be wrong 40% of the time, like the Brier score)