3 ms·

To be honest, I think in the future we will interview people on their ability to work with an LLM. This would be a separate skill from the other ones we are loo

by foxyv 3y ago

To be honest, I think in the future we will interview people on their ability to work with an LLM. This would be a separate skill from the other ones we are looking for. Maybe even have them do some fact checks on a given prompt and response as well as suggest new prompts that would give better results. There might even be an entire AI based section of an interview. In the end, it's just a new way to "Google" the answer. After all, there isn't much difference between reading off an LLM response and just reading the Wikipedia page after a quick Google search, except for less advertisements.

jacques_chester 3y ago
I agree that this is the likely long term outcome. But for now folks want to think that everyone needs to have memorized every individual screw, nail, nut and bolt in the edifice of computer science.
SOLAR_FIELDS 3y ago
I’ve already been allowed to use it in programming interviews where they’ve said it’s explicitly allowed to use ChatGPT. It’s led to some fun interactions because I use it a lot and as such I’m quite good with it and interviewers are often taken aback by how quickly I’m able to just destroy the question they put out with a good prompt I will say there are still some programming questions you can give that will stump the hell out of ChatGPT. In particular I took one online coding assessment where I used it and there was a question about plotting on a graph with code and calculating areas based on the points plotted that ChatGPT failed miserably at, but someone pretty good with math and geometry would find pretty tractable.
- fragmede 3y ago
  There are ChatGPT resistant questions you can ask. ChatGPT recognizes the question but doesn't actually think about it, so if you give it the river crossing problem (farmer, fox, sheep, and grain need to cross a river) but tell it the boat can take all the items, it won't actually read those details and blithely solve the problem the expected way. Give candidates a problem that's trivially solvable if you actually read the question and see if they try and solve it the ChatGPT way.
  - hirsin 3y ago
    Indeed - with 3.5 at least it does fail the easy mode riddle https://chat.openai.com/share/b3761807-551d-4cfc-b291-6d37ee2de4dd https://chat.openai.com/share/b3761807-551d-4cfc-b291-6d37ee...
    - fragmede 3y ago
      it's a fun problem to explore, and gpt-4 doesn't do any better. swapping in other things doesn't help because it internally recognizes it as the river crossing problem and proceeds to solve it normally. I was able to get it to two shot it with a lot of coaching but yeah, it's a trip.
  - ric2b 3y ago
    The downside is that you're now wasting interview time on the river crossing problem instead of actually relevant questions for the job you're hiring for.
    - fragmede 3y ago
      Don't literally use the river crossing problem, but it's existence implies there is a form of question that ChatGPT will solve for where someone actually reading the prompt can solve trivially but someone using ChatGPT will get stuck on. You're already asking Leetcode questions that are irrelevant for the job you're hiring for. What's the problem with asking one more to test for cheaters?
theamk 3y ago
We didn't start testing people on Google usage when Googling became useful, so I don't see why LLMs would be different. Instead, there would be tasks that can be completed using any tools available - Google, LLM, whatever. And candidates are rated on how well the task is done, and maybe asked a few questions to make sure they made decisions knowingly and not just copied the first answer off the internet. This already exists and is called "take home programming assignment"

by foxyv 3y ago

jacques_chester 3y ago

I agree that this is the likely long term outcome. But for now folks want to think that everyone needs to have memorized every individual screw, nail, nut and bolt in the edifice of computer science.

SOLAR_FIELDS 3y ago

I’ve already been allowed to use it in programming interviews where they’ve said it’s explicitly allowed to use ChatGPT. It’s led to some fun interactions because I use it a lot and as such I’m quite good with it and interviewers are often taken aback by how quickly I’m able to just destroy the question they put out with a good prompt I will say there are still some programming questions you can give that will stump the hell out of ChatGPT. In particular I took one online coding assessment where I used it and there was a question about plotting on a graph with code and calculating areas based on the points plotted that ChatGPT failed miserably at, but someone pretty good with math and geometry would find pretty tractable.

fragmede 3y ago
There are ChatGPT resistant questions you can ask. ChatGPT recognizes the question but doesn't actually think about it, so if you give it the river crossing problem (farmer, fox, sheep, and grain need to cross a river) but tell it the boat can take all the items, it won't actually read those details and blithely solve the problem the expected way. Give candidates a problem that's trivially solvable if you actually read the question and see if they try and solve it the ChatGPT way.
- hirsin 3y ago
  Indeed - with 3.5 at least it does fail the easy mode riddle https://chat.openai.com/share/b3761807-551d-4cfc-b291-6d37ee2de4dd https://chat.openai.com/share/b3761807-551d-4cfc-b291-6d37ee...
  - fragmede 3y ago
    it's a fun problem to explore, and gpt-4 doesn't do any better. swapping in other things doesn't help because it internally recognizes it as the river crossing problem and proceeds to solve it normally. I was able to get it to two shot it with a lot of coaching but yeah, it's a trip.
- ric2b 3y ago
  The downside is that you're now wasting interview time on the river crossing problem instead of actually relevant questions for the job you're hiring for.
  - fragmede 3y ago
    Don't literally use the river crossing problem, but it's existence implies there is a form of question that ChatGPT will solve for where someone actually reading the prompt can solve trivially but someone using ChatGPT will get stuck on. You're already asking Leetcode questions that are irrelevant for the job you're hiring for. What's the problem with asking one more to test for cheaters?

theamk 3y ago

We didn't start testing people on Google usage when Googling became useful, so I don't see why LLMs would be different. Instead, there would be tasks that can be completed using any tools available - Google, LLM, whatever. And candidates are rated on how well the task is done, and maybe asked a few questions to make sure they made decisions knowingly and not just copied the first answer off the internet. This already exists and is called "take home programming assignment"