2 ms·
I imagine that would work pretty well given an adequate and representative body of annotated sample data. Though that is also not easy to come by.
by marginalia_nu 1y ago
I imagine that would work pretty well given an adequate and representative body of annotated sample data. Though that is also not easy to come by.
- ted_dunning 1y agoActually, it is easy to come up with reasonably decent heuristics that can auto-tag a corpus. From that you can look for anomalies and adjust your tagging system. The problem of getting a representative body is (surprisingly) much harder than the annotation. I know. I spent quite some time years ago doing this.
- wrs 1y agoBut if you believe in your manual heuristics enough to ship them, you must already have a body of tests that you're happy with, right? Also seems like this is a case where generating synthetic data would be a big help. You don't have to use only real-world documents for training, just examples of the sorts of things real-world documents have in them. Make a vast corpus of semi-random documents in semi-random fonts and settings, printed from Word, Pandoc, LaTeX, etc.