Using accuracy to measure PoS taggers makes results look good, but is ensnaring due to their huge bias: Tagging every word by the majority tag found during trai

Using accuracy to measure PoS taggers makes results look good, but is ensnaring due to their huge bias: Tagging every word by the majority tag found during training and everything else as either NNP or NNPS (with suffix -s) means the statistical baseline is already well beyond 90% accuracy. However, my point was that from the results shown its not clear to me if the gains in attachment score you saw when using Gold Standard PoS tags would be lost in a "real-world" usage when you have to rely on the tagger's own PoS tags. In such a case, it could be that your embeddings contribute much less "new knowledge" than what you see in your results, using independent (Gold) PoS tags. This might be mitigated by using two independently trained and set up PoS taggers, however. But this finally gets us back to my initial concern: How much performance gain really is in there from all this added complexity and is that "worth the effort"?