I don't think this is a weird idea. In fact, I really like this idea. It seems like a rating agency could be given a "score" based on the historical accuracy

I don't think this is a weird idea. In fact, I really like this idea. It seems like a rating agency could be given a "score" based on the historical accuracy of their measurements, since although ongoing books are business secrets I would imagine you could get SOME proxy at least for examining the veracity of the ratings in hindsight. It could be done with very little overhead, and would be a first step in some small sort of accountability, at least, if only an accountability to the data. Could anyone with an actual familiarity with this domain say if something like this exists, or if there's sufficient audit data for it to exist?