4 ms·
It's a hard problem; one you can't ever fully solve. You need to accept some degree of error, as pretty soon you'll hit the point where every incremental improv
by svat 8y ago
It's a hard problem; one you can't ever fully solve. You need to accept some degree of error, as pretty soon you'll hit the point where every incremental improvement requires you to basically double the amount of effort. (You can start by incorporating some signals like whether the word/title occurs in uppercase, whether it has any special formatting like italics, whether the author's name occurs nearby, and so on.)
But what you can do is be very up front about the error: you can be explicit about your methods and describe their modes of failure, give examples of some of the things you might miss, try to analyze your error and how bad it is, and so on, and finally leave it to the reader to decide how seriously to take your results. (If you see some recent papers they include a “Threats to Validity” section, e.g. Section 3.4 here: https://people.engr.ncsu.edu/ermurph3/papers/seip18.pdf https://people.engr.ncsu.edu/ermurph3/papers/seip18.pdf)