3 ms·

Yeah I've been doing the same. Have been pretty stunned at people's inability to use AI tools effectively.

3eb7988a1663 2y ago
What does effective use look like? I have attempted messing around with a couple of options, but was always disappointed with the output. How do you properly present a problem to a LLM? Requiring an ongoing conversation feels like a tech priest praying to the machine spirit.
- Guthur 2y ago
  In my opinion that's literally what we're aiming for, whether or not intentionally.
- CSMastermind 2y ago
  Candidates generally use it in one of two ways: either as an advanced autocomplete or like a search engine. They'll type in things like, "C# read JSON from file" As opposed to something like: > I'm working on a software system where ... are represented as arrays of JSON objects with the following properties: > ... > I have a file called ... that contains an array of these objects ... > I have ... installed locally with a new project open and I want to ... How can I do this? No current LLMs can solve any of the problems we give them so pasting in the raw prompt won't be helpful. But the set up deliberately encourages them to do some basic scaffolding, reading in a file, creating corresponding classes, etc. that an LLM can bang out in about 30 seconds but I've seen candidates spend 30 minutes+ writing it all out themselves instead of just getting the LLM to do it for them.
  - neonsunset 2y ago
    GitHub Copilot Edit can do the second version of this. It is pretty good at it too. It sometimes gets things wrong but for your average code (and candidates typing in "C# read JSON from file" are way below average unless they never written in C#), if you give all the files for a specific self-contained part of the program, it can extend/modify/test/etc. it impressively well for an LLM. The difference compared to where we were just 1-2 years ago is staggering. Edit: the above is with Claude-3.5-Sonnet
- codeyperson 2y ago
  High level - having a discussion with the LLM about different approaches and the tradeoffs between each Low level - I'll write up the structure of what I want in the form of a set functions with defined inputs and outputs but without the implementation detail. If I care about any specifics with the functions I'll throw some comments in there. And sometimes I'll define the data structures in advance as well. Once all this is set up it often spits out something that compiles and works first try. And all the context is established so iteration from that point becomes easier.
  - feoren 2y ago
    > High level - having a discussion with the LLM about different approaches and the tradeoffs between each I honestly can't imagine this. If the AI says "However, a downside of approach B is that it takes O(n^2) time instead of the optimal O(nlog(n))", what do you think the odds are that it literally made up both of those facts? Because I'd be surprised if they were any lower than 30%. It's an extremely confident bullshitter, and you're going to use it to talk about engineering tradeoffs!? > Once all this is set up it often spits out something that compiles and works first try I'm sorry, but I'm extremely* doubtful that it actually works in any real sense. The fact that you even use "compiles and works first try" as some sort of metric that the code it's producing shows how easily it could slip in awful braindead bugs without you ever knowing. You run it and it appears to work!? The way to know whether something works -- not first try, but every try -- is to understand every character in the code. If that is your standard -- and it must be -- then isn't the AI just slowing you down?
    - concordDance 2y ago
      It can list tradeoffs and approaches you might have forgotten. Thats the big use case for me.
    - sanarothe 2y ago
      I don't code for a living, and I'm probably worse than a fresh grad would be but I use: "Please don't generate or rewrite code, I just want to discuss the general approach." Bc I don't know any design patterns or idiomatic approach, being able to discuss is amazing. Though quality and consistency of responses is another thing... :)
al_borland 2y ago
When I was last interviewing people (several years ago now), I’d let them use the internet to help them on anything hands on. I was astounded by how bad some people were at using a search engine. Some people wouldn’t even make an attempt.

What does effective use look like? I have attempted messing around with a couple of options, but was always disappointed with the output. How do you properly present a problem to a LLM? Requiring an ongoing conversation feels like a tech priest praying to the machine spirit.

Guthur 2y ago
In my opinion that's literally what we're aiming for, whether or not intentionally.
CSMastermind 2y ago
Candidates generally use it in one of two ways: either as an advanced autocomplete or like a search engine. They'll type in things like, "C# read JSON from file" As opposed to something like: > I'm working on a software system where ... are represented as arrays of JSON objects with the following properties: > ... > I have a file called ... that contains an array of these objects ... > I have ... installed locally with a new project open and I want to ... How can I do this? No current LLMs can solve any of the problems we give them so pasting in the raw prompt won't be helpful. But the set up deliberately encourages them to do some basic scaffolding, reading in a file, creating corresponding classes, etc. that an LLM can bang out in about 30 seconds but I've seen candidates spend 30 minutes+ writing it all out themselves instead of just getting the LLM to do it for them.
- neonsunset 2y ago
  GitHub Copilot Edit can do the second version of this. It is pretty good at it too. It sometimes gets things wrong but for your average code (and candidates typing in "C# read JSON from file" are way below average unless they never written in C#), if you give all the files for a specific self-contained part of the program, it can extend/modify/test/etc. it impressively well for an LLM. The difference compared to where we were just 1-2 years ago is staggering. Edit: the above is with Claude-3.5-Sonnet
codeyperson 2y ago
High level - having a discussion with the LLM about different approaches and the tradeoffs between each Low level - I'll write up the structure of what I want in the form of a set functions with defined inputs and outputs but without the implementation detail. If I care about any specifics with the functions I'll throw some comments in there. And sometimes I'll define the data structures in advance as well. Once all this is set up it often spits out something that compiles and works first try. And all the context is established so iteration from that point becomes easier.
- feoren 2y ago
  > High level - having a discussion with the LLM about different approaches and the tradeoffs between each I honestly can't imagine this. If the AI says "However, a downside of approach B is that it takes O(n^2) time instead of the optimal O(nlog(n))", what do you think the odds are that it literally made up both of those facts? Because I'd be surprised if they were any lower than 30%. It's an extremely confident bullshitter, and you're going to use it to talk about engineering tradeoffs!? > Once all this is set up it often spits out something that compiles and works first try I'm sorry, but I'm extremely* doubtful that it actually works in any real sense. The fact that you even use "compiles and works first try" as some sort of metric that the code it's producing shows how easily it could slip in awful braindead bugs without you ever knowing. You run it and it appears to work!? The way to know whether something works -- not first try, but every try -- is to understand every character in the code. If that is your standard -- and it must be -- then isn't the AI just slowing you down?
  - concordDance 2y ago
    It can list tradeoffs and approaches you might have forgotten. Thats the big use case for me.
  - sanarothe 2y ago
    I don't code for a living, and I'm probably worse than a fresh grad would be but I use: "Please don't generate or rewrite code, I just want to discuss the general approach." Bc I don't know any design patterns or idiomatic approach, being able to discuss is amazing. Though quality and consistency of responses is another thing... :)

al_borland 2y ago

When I was last interviewing people (several years ago now), I’d let them use the internet to help them on anything hands on. I was astounded by how bad some people were at using a search engine. Some people wouldn’t even make an attempt.