3 ms·
Any naive use of an LLM is not likely to produce good results, even with the best models. You need a process - a sequence of steps, and appropriately safeguarde
by observationist 3y ago
Any naive use of an LLM is not likely to produce good results, even with the best models. You need a process - a sequence of steps, and appropriately safeguarded prompts at each step. AI will eventually reach a point when you can get all the subtle nuance and quality in task performance you might desire, but right now, you have to dumb things down and be very explicit. Assumptions will bite you in the ass.
Naive, superficial one shot prompting, even with CoT or other clever techniques, or using big context, is insufficient to achieve quality, predictable results.
Dropping the resume into a prompt with few-shot examples can get you a little consistency, but what really needs to be done is repeated discrete operations, that link the relevant information to the relevant decisions. You'd want to do something like tracking years of experience, age, work history, certifications, and so on, completely discarding any information not specifically relevant to the decision of whether to proceed in the hiring process. Once you have that information separated out, you consider each in isolation, scoring from 1 to 10, with a short justification for each scoring based on many-shot examples. Then you build a process iteratively with the bot, asking it which variables should be considered in context of the others, and incorporate a -5 to 5 modifier based on each clustering of variables (8 companies in the last 2 years might be a significant negative score, but maybe there's an interesting success story involved, so you hold off on scoring until after the interview.)
And so on, down the line, through the whole hiring process. Any time a judgment or decision has to be made, break it down into component parts, and process each of the parts with their own prompts and processes, until you have a cohesive whole, any part of which you can interrogate and inspect for justifiable reasoning.
The output can then be handled by a human, adjusted where it might be reasonable to do so, and you avoid the endless maze of mode collapse pits and hallucinated dragons.
LLMs are not minds - they're incapable of acting like minds, unless you build a mind-like process around them. If you want a reasonable, rational, coherent, explainable process, you can't achieve that with zero or one shot prompting. Complex and impactful decisions like hiring and resume processing isn't a task current models are equipped to handle naively.
- leeny 3y agoAuthor here. I think our issue is that many recruiting tools are built on top of naive ChatGPT... because most recruiting solutions don't have the training data to fine-tune. So whatever biases are in ChatGPT persist in other products.
- observationist 3y agoRecruiting tools built on top of naive ChatGPT is just a bad idea. Any tool that can have such a large impact on someone's life should be used competently and with all the nuance and care that can be brought to bear on the task. I'm not talking at all about fine tuning, simply building a process with multiple prompts and multiple stages, taking advantage of the things that AI can do well, instead of trying to jam an entire resume down the AI's throat and hoping for the best. My beef with both the Bloomberg article and the response to it is that they're analyzing a poorly thought out and inappropriate use of a technology in a way that is almost guaranteed to cause unintended problems - like measuring how long it takes people to dig holes with a shovel without a handle. It's not a sensible thing to do, and the Bloomberg journos aren't acting in good faith, anyway - they'll continue attacking AI and reaping clicks until they figure out some other way to leech off the AI boom.
- Decker87 3y agoDid you comment on the right article? This seems to have nothing to do with whether the Bloomberg study article is correct or not.
- Rinzler89 3y ago> Assumptions will bite you in the ass. Assumptions bite you in the ass even when you deal with humans who you work with daily. Assuming the LLM can read your mind is laughable. Despite it being all knowing you have to explain things to it like it's a 5 year old to make sure you're always on the same page.
- WalterBright 3y ago> appropriately safeguarded prompts Why do people need to be protected from text an AI bot might emit?
- exe34 3y agoIt's for the reputation of the people passing the text off as their own or the LLM as their agent acting on their behalf.
- WalterBright 3y agoIf people don't vet what they get from AI, it's their fault, not AI's.
- kelseyfrog 3y agoWhose fault it is is irrelevant. What is relevant is consequences. Folks are responsible for consequences of their decisions. Drawing a line between those producing AI output and those consuming AI output is entirely arbitrary. Those producing AI content have a responsibility too. That's just basic human decency.
- WalterBright 3y agoBowdlerized results are less useful/interesting to me. I'm not interested in pablum.
- kelseyfrog 3y agoI respect your honesty, but interestingness won't ever override responsibility.
- exe34 3y agoWe need to get you in charge of writing tabloid headlines!
- deleted 3y ago
- gaws 3y agoThis response is nonsense. Did you even read the OP link or the Bloomberg story?