4 ms·

There are plenty of engineering jobs in the "non-engineering" graph. ("Fullstack Javascript Ninja" 3% 50K) ("Avionic Software Lead" 6.25% 120k) ("Softwar

• minimaxir 10y ago
  3 jobs out of 3,000 in a category does not invalidate the results and is within acceptable error (and even if there was a super-mega-ultra outlier as a result, that's why the median is used instead of the average). Although I made my best effort, it's infeasible for anyone to get 100% perfect categorization, which is why I would not have made this visualization without having a sufficiently large sample size.
  • benkuykendall 10y ago
    I agree that a few misclassifications is not a huge problem, but it suggests a bigger methodological flaw. Looking at your source code, it seems your definition of engineer is grepl("engineer", tolower(job_title)) | grepl("developer", tolower(job_title) This seems like a poor definition of engineer to use. It includes "VP Engineering" but excludes "CTO", it includes "Growth Engineers" (marketers) but excludes "Code Ninjas" (developers). Short of word2vec, I don't think you're going to be able to automatically classify people as engineers based on job title.
    • minimaxir 10y ago
      That heuristic is good for 99%+ of the data set (and probably higher accuracy than a word2vec approach anyways). That's more than acceptable for this analysis.
    • jonathankoren 10y ago
      You don't even need word2vec good old cosine sim or an svm on the descriptions would classify theses. But if you're stuck with just the titles, just make some more patterns. There's not much you can do without building a simple dictionary.