You need to have a known-good test dataset that is as representative as possible. Something that you are absolutely sure is golden - i.e. someone has gone throu

You need to have a known-good test dataset that is as representative as possible. Something that you are absolutely sure is golden - i.e. someone has gone through it manually and verified all the labels are correct and complete. Then it doesn't matter quite so much if your input labels are noisy, because if you perform well at test time, your model is working. If you have no idea of the accuracy of any of your data then you're probably asking the wrong question of it. You can do things like test for consistency using cross-validation, e.g. does half of your dataset predict the other half with the same kind of performance? But that can't detect the same errors repeated throughout your data.