3 ms·

Don't do that then? Resume screening with an LLM is obviously a bad idea, but maybe this study will be more convincing.

_heimdall 2y ago
Automated screenings have been used for quite a few years now. I never liked the practice, but an LLM can't do any worse than a basic algorithm that attempts to scrub text out of a PDF or Word document and filter by keywords.
- tbrownaw 2y ago
  > an LLM can't do any worse than a basic algorithm that attempts to scrub text out of a PDF or Word document and filter by keywords. Keyword filtering looks at what you explicitly tell it to look at. Magic LLM filtering looks at who knows what, especially if you use is as more than just a fancier entity extractor.
  - _heimdall 2y ago
    But both work poorly in my experience. We'd need to define metrics and a lot more data to really know which one performs worse. Keyword filtering is only as good as those who decide the keywords to filter on and how well the algorithm can match keywords and variations of keywords. I've worked with plenty of tech recruiters over the years, they rarely understood anything about the jobs they recruited for. I never trusted the keywords and parameters they would set on any automated screenings.
    - tbrownaw 2y ago
      The magic black box can have ~unlimited downside, if you give it a question that's too open-ended and for some reason it decides to key off of something you're not allowed to look at. Dumb keyword filtering may be useless, but it shouldn't have that extra risk.
      - _heimdall 2y ago
        The unlimited downside is really driven by how people use the tool and what they do with the results rather than how the tool itself works. If any hiring manager uses a tool poorly and trusts bad information that comes out the other end, that's on them not the tool. That said, I very much am concerned with how quickly and blindly we're trying to develop and use AI. LLMs being used for narrow tasks like filtering results are at least limited a bit in risk, but what isn't limited is all the dangerous things we can do as we learn more from how these algorithms do those tasks well or poorly.
tbrownaw 2y ago
Yes, but maybe it takes a while for non-experts to figure out that the thing that's been sold as magic and sure looks magical when you play with it for a bit turns out to not in fact be magic under more intensive use. I think that's where that "trough of disillusionment" in the hype cycle comes from.
jart 2y ago
Just get rid of the whole resume / application system. Don't 90% of people get their jobs by knowing people anyway? Or some recruiter finding your GitHub. Applying for jobs with strangers where your competition is a stack of paper was never a dignified experience for anyone. Even being a hobo backpacking through Europe is better than living a single day of that humiliation.

Automated screenings have been used for quite a few years now. I never liked the practice, but an LLM can't do any worse than a basic algorithm that attempts to scrub text out of a PDF or Word document and filter by keywords.

tbrownaw 2y ago
> an LLM can't do any worse than a basic algorithm that attempts to scrub text out of a PDF or Word document and filter by keywords. Keyword filtering looks at what you explicitly tell it to look at. Magic LLM filtering looks at who knows what, especially if you use is as more than just a fancier entity extractor.
- _heimdall 2y ago
  But both work poorly in my experience. We'd need to define metrics and a lot more data to really know which one performs worse. Keyword filtering is only as good as those who decide the keywords to filter on and how well the algorithm can match keywords and variations of keywords. I've worked with plenty of tech recruiters over the years, they rarely understood anything about the jobs they recruited for. I never trusted the keywords and parameters they would set on any automated screenings.
  - tbrownaw 2y ago
    The magic black box can have ~unlimited downside, if you give it a question that's too open-ended and for some reason it decides to key off of something you're not allowed to look at. Dumb keyword filtering may be useless, but it shouldn't have that extra risk.
    - _heimdall 2y ago
      The unlimited downside is really driven by how people use the tool and what they do with the results rather than how the tool itself works. If any hiring manager uses a tool poorly and trusts bad information that comes out the other end, that's on them not the tool. That said, I very much am concerned with how quickly and blindly we're trying to develop and use AI. LLMs being used for narrow tasks like filtering results are at least limited a bit in risk, but what isn't limited is all the dangerous things we can do as we learn more from how these algorithms do those tasks well or poorly.

tbrownaw 2y ago

Yes, but maybe it takes a while for non-experts to figure out that the thing that's been sold as magic and sure looks magical when you play with it for a bit turns out to not in fact be magic under more intensive use. I think that's where that "trough of disillusionment" in the hype cycle comes from.

jart 2y ago

Just get rid of the whole resume / application system. Don't 90% of people get their jobs by knowing people anyway? Or some recruiter finding your GitHub. Applying for jobs with strangers where your competition is a stack of paper was never a dignified experience for anyone. Even being a hobo backpacking through Europe is better than living a single day of that humiliation.