4 ms·
I agree that a few misclassifications is not a huge problem, but it suggests a bigger methodological flaw. Looking at your source code, it seems your definition
by benkuykendall 10y ago
I agree that a few misclassifications is not a huge problem, but it suggests a bigger methodological flaw. Looking at your source code, it seems your definition of engineer is
grepl("engineer", tolower(job_title)) | grepl("developer", tolower(job_title)
This seems like a poor definition of engineer to use. It includes "VP Engineering" but excludes "CTO", it includes "Growth Engineers" (marketers) but excludes "Code Ninjas" (developers). Short of word2vec, I don't think you're going to be able to automatically classify people as engineers based on job title.
- minimaxir 10y agoThat heuristic is good for 99%+ of the data set (and probably higher accuracy than a word2vec approach anyways). That's more than acceptable for this analysis.
- jonathankoren 10y agoYou don't even need word2vec good old cosine sim or an svm on the descriptions would classify theses. But if you're stuck with just the titles, just make some more patterns. There's not much you can do without building a simple dictionary.