8 ms·

> Treat it as a naive but intelligent intern That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know”

> Treat it as a naive but intelligent intern That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know” or “I’m not sure I did it right”. LLMs do none of that, they will take whatever you ask and give a reasonable-sounding output that might be anything between brilliant and nonsense. With an intern, I don’t need to measure how good my prompting is, we’ll usually interact to arrive to a common understanding. With a LLM, I need to put a huge amount of thought into the prompt and have no idea whether the LLM understood what I’m asking and if it’s able to do it.

jappgar 2y ago
They've explicitly been trained/system-prompted to act that way. Because that's what the marketing teams at these AI companies want to sell. It's easy to override this though by asking the LLM to act as if it were less-confident, more hesitant, paranoid etc. You'll be fighting uphill against the alignment(marketing) team the whole time though, so ymmv.
kranuck 2y ago
> have no idea whether the LLM understood what I’m asking That's easy. The answer is it doesn't. It has no understanding of anything it does. > if it’s able to do it This is the hard part.
mercer 2y ago
Can I have some of those sorts of interns?
arthurcolle 2y ago
many many teams are actively building SOTA systems to do this in ways previously unimagined. you can enqueue tasks and do whatever you want. I gotta say as a current gen LLM programmer person, I can completely appreciate how bad they are now - I recently tweeted about how I "swore off" AI tools but like... there are many ways to bootstrap very powerful software or ML systems around or inside these existing models that can blow away existing commercial implementations in surprising ways
- gmerc 2y ago
  “building” is the easy part
  - falcor84 2y ago
    building SOTA systems is the easy part?! Easy compared to what?
    - kristianp 2y ago
      Probably, to get them to work without hallucinating, or without failing a good percentage of the time.
      - falcor84 2y ago
        I wonder what would our world look like if these two expectations that you seem to be taking for granted were applied to our politicians.
        AbstractH24 2y ago
        Are you suggesting people are satisfied with our politicians and aspire for other things to be just as good as them? What if we applied those two expectations to building construction? What if we didn’t?
        falcor84 2y ago
        I think it's always good to aspire for more, but we shouldn't be expecting perfect results in novel areas of technology. Taking up your construction metaphor, LLMs are now where construction was perhaps 3000 years ago; buildings weren't that sturdy, but even if the roofs leaked a bit, I'm sure it beat sleeping outside on a rainy night. We need to continue iterating.
        AbstractH24 2y ago
valval 2y ago
Really it just does what you tell it to. Have you tried telling it “ask me clarifying questions about all the APIs you need to solve this problem”? Huge contrast to human interns who aren’t experienced or smart enough to ask the right questions in the first place, and/or have sentimental reasons for not doing so.
- aktuel 2y ago
  I have to say, having to tell it to ask me clarifying questions DOES make it really look smart!
  - arthurcolle 2y ago
    imagine if you make it keep going without having to reprompt it
    - carlmr 2y ago
      Isn't that the exact point of o1, that it has time to think for itself without reprompting?
      - arthurcolle 2y ago
        yeah but they aren't letting you see the useful chain of thought reasoning that is crucial to train a good model. Everyone will replicate this over next 6 months
        optimalsolver 2y ago
        >Everyone will replicate this over next 6 months Not without a billion dollars worth of compute, they won't.
        arthurcolle 2y ago
        Are you sure its a billion? Helps with estimating the training run
raverbashing 2y ago
> A good intern will ask clarifying questions, tell me “I don’t know” Your expectations are bigger than mine (Though some will get stuck in "clarifying questions" and helplessness and not proceed neither)
- williamdclt 2y ago
  Note that we are talking about a “good” intern here
  - TeMPOraL 2y ago
    Unreasonably good. Beyond fresh junior employee good. Also, that's your standard; 'MPSimmons said to treat the model as "naive but intelligent" intern, not a good one.
- steveBK123 2y ago
  Indeed. My expectation of a good intern is to produce nothing I will put in production, but show aptitude worth hiring them for. It's a 10 week extended interview with lots of social events, team building, tech talks, presentations, etc. Which is why I've liked the LLM analogy of "unlimited free interns".. I just think some people read that the exact opposite way I do (not very useful).
  - Martinussen 2y ago
    If I had to respect the basic human rights of my LLM backends, it would probably be less appealing - but "Unlimited free smart-for-being-braindead zombies" might be a little more useful, at least?
    - steveBK123 2y ago
      Interns, at least on paper, have the optionality of getting better with time in observable obvious ways as they become grad hires, junior engineers, mid engineers etc. So far, 2 years of publicly accessible LLMs have not improved for intern replacement tasks at the rate a top 50% intern would be expected to.
singingfish 2y ago
> LLMs do none of that, they will take whatever you ask and give a reasonable-sounding output that might be anything between brilliant and nonsense. This is exactly why I’ve been objecting so much to the use of the term “hallucination” and maintain that “confabulation” is accurate. People who have spent enough time with acutelypsychotic people, and people experiencing the effects of long term alcohol related brain damage, and trying to tell computers what to do will understand why.
- bartread 2y ago
  I don't know that "confabulation" is right either: it has a couple of other meanings beyond "a fabricated memory believed to be true" and, of course, the other issue is that LLMd don't believe anything. They'll backtrack on even correct information if challenged.
berniedurfee 2y ago
I’m starting to think this is an unsolvable problem with LLMs. The very act of “reasoning” requires one to know that they don’t know something. LLMs are giant word Plinko machines. A million monkeys on a million typewriters. LLMs are not interns. LLMs are assumption machines. None of the million monkeys or the collective million monkeys are “reasoning” or are capable of knowing. LLMs are a neat parlor trick and are super powerful, but are not on the path to AGI. LLMs will change the world, but only in the way that the printing press changed the world. They’re not interns, they’re just tools.
- idiotsecant 2y ago
  I think LLMs are definitely on the path to AGI in the same way that the ball bearing was on the path to the internal combustion engine. I think its quite likely that LLMs will perform important functions within the system of an eventual AGI.
  - swader999 2y ago
    This may be accurate. I wonder if there's enough energy in the world for this endeavour.
    - TeMPOraL 2y ago
      Of course! 1. We've barely scratched the surface of this solution space; the focus only recently started shifting from improving model capabilities to improving training costs. People are looking at more efficient architectures, and lots of money is starting to flow in that direction, so it's a safe bet things will get significantly more efficient. 2. Training is expensive, inference is cheap, copying is free. While inference costs add up with use, they're still less than costs of humans doing the equivalent work, so out of all things AI will impact, I wouldn't worry about energy use specifically.
    - int_19h 2y ago
      Humans don't require immense amounts of energy to function. The reasons why LLMs do is because we are essentially using brute force as the methodology for making them smarter for the lack of better understanding of how this works. But this then gives us a lot of material to study to figure that part out for future iterations of the concept.
ryoshu 2y ago
1000% this. LLMs can't say "I don't know" because they don't actually think. I can coach a junior to get better. LLMs will just act like they know what they are doing and give the wrong results to people who aren't practitioners. Good on OAI calling their model Strawberry because of Internet trolls. Reactive vs proactive.
- jug 2y ago
  This surprises me. I made a simple chat fed with PDF's and using LangChain and it by default said it didn't know if I asked questions outside of the corpus. It was a simple matter of the confidence score getting too low?
- bartread 2y ago
  I get a lot of value out of ChatGPT but I also, fairly frequently, run into issues here. The real danger zones are areas that lie at or just beyond the edges of my own knowledge in a particular area. I'd say that most of my work use of ChatGPT does in fact save me time but, every so often, ChatGPT can still bullshit convincingly enough to waste an hour or two for me. The balance is still in its favour, but you have to keep your wits about you when using it.
  - ryoshu 2y ago
    Agreed, but the problem is if these things replace practitioners (what every MBA wants them to do), it's going to wreck the industry. Or maybe we'll get paid $$$$ to fix the problems they cause. GPT-4 introduced me to window functions in SQL (haven't written raw SQL in over a decade). But I'm experienced enough to look at window functions and compare them to subqueries and run some tests through the query planner to see what happens. That's knowledge that needs to be shared with the next generation of developers. And LLMs can't do that accurately.
    - lupire 2y ago
      Optimizing a query is certainly something the machine (not necessarily the LLM part) can do better than the human, for 99.9% of situations and people. PostgreSQL developers are oposed to query execution hints, because if a human knows a better way to execute a query, the devs want to put that knowledge into the planner.
0xdeadbeefbabe 2y ago
A lot of interns are overconfident though
naasking 2y ago
> That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know” or “I’m not sure I did it right”. An intern that grew up in a different culture then, where questioning your boss is frowned upon. The point is that the way to instruct this intern is to front-load your description of the problem with as much detail as possible to reduce ambiguity.
noisy_boy 2y ago
I feel like it almost always starts well, given the full picture, but then for non-trivial stuff, gets stuck towards the end. The longer the conversation goes, the more wheel-spinning occurs and before you know it, you have spent an hour chasing that last-mile-connectivity. For complex questions, I now only use it to get the broad picture and once the output is good enough to be a foundation, I build the rest of it myself. I have noticed that the net time spent using this approach still yields big savings over a) doing it all myself or b) keep pushing it to do the entire thing. I guess 80/20 etc.
- skybrian 2y ago
  Yes, I’ve seen that too. One reason it will spin its wheels is because it “prefers” patterns in transcripts and will try to continue them. If it gets something wrong several times, it picks up on the “wrong answers” pattern. It’s better not to keep wrong answers in the transcript. Edit the question and try again, or maybe start a new chat.
- mlsu 2y ago
  This is the way. I've had this experience many times: - hey, can you write me a thing that can do "xyz" - sure, here's how we can do "xyz" (gets some small part of the error handling for xyz slightly wrong) - can you add onto this with "abc" - sure. in order to do "abc" we'll need to add "lmn" to our error handling. this also means that you need "ijk" and "qrs" too, and since "lmn" doesn't support "qrs" out of the box, we'll also need a design solution to bridge the two. Let me spend 600 more tokens sketching that out. - what if you just use the language's built in feature here in "xyz"? does't that mean we can do it with just one line of code? - yes, you're absolutely right. I'm sorry for making this over complicated. If you don't hit that kill switch, it just keeps doubling down on absurdly complex/incorrect/hallucinatory stuff. Even one small error early in the chain propagates. That's why I end up very frequently restarting conversations in a new chat or re-write my chat questions to remove bad stuff from the context. Without the ability to do that, it's nearly worthless. It's also why I think we'll be seeing absurdly, wildly wrong chains of thought coming out of o1. Because "thinking" for 20s may well cause it to just go totally off the rails half the time.
yukIttEft 2y ago
Makes me wonder if "I don't know" could be added to LLM: whenever an activation has no clear winner value (layman here), couldn't this indicate low response quality?
- Regic 2y ago
  This exists and does work to some degree, e.g. Detecting hallucinations in large language models using semantic entropy https://www.nature.com/articles/s41586-024-07421-0 https://www.nature.com/articles/s41586-024-07421-0
jacobn 2y ago
Is this a dataset issue more than an LLM issue? As in: do we just need to add 1M examples where the response is to ask for clarification / more info? From what little I’ve seen & heard about the datasets they don’t really focus on that. (Though enough smart people & $$$ have been thrown at this to make me suspect it’s not the data ;)
richerram 2y ago
I think this is the main issue with these tools... what people are expecting of them. We have swallowed the pill that LLMs are supposed to be AGI and all that mumbo jumbo, when they are just great tools and as such one needs to learn to use the tool the way it works and make the best of it, nobody is trying to hammer a nail with a broom and blaming the broom for not being a hammer...
- koe123 2y ago
  I completely agree. To me the discussion here reads a little like: “Hah. See? It cant do everything!”. It makes me wonder if the goal is to convince each other that: yes, indeed, humans are not yet replaced. It’s next token regression, of course it can’t truely introspect. That being said LLMs are amazing tools and o1 is yet another incremental improvement and I welcome it!
jasondigitized 2y ago
Have you ever worked with an intern? They have personalities and expectations that need to be managed. They get sick. The get tired. They want to punch you if you treat them like a 24-7 bird dog. It's so much easier to not let perfect be the enemy of the good and just rapid fire ALL day at a LLM for any and everything I need help with. You can also just not use the LLM. Interns need to be 'fed' work or the ROI ends upside down. Is a LLM as good as a top tier intern. No, but with a LLM I can have 10 pretty good interns by opening 10 tabs.
- sqeaky 2y ago
  I think listing off a set of pros and cons for interns and LLMs misses the point, they seem like categorically different kinds of intelligence.
- ww2supercut 2y ago
  The LLMs are getting better and better at a certain kind of task, but there's a subset of tasks that I'd still much rather have any human than an LLM, today. Even something simple, like "Find me the top 5 highest grossing movies of 2023" it will take a long time before I trust an LLM's answer, without having a human intern verify the output.
Closi 2y ago
> With an intern, I don’t need to measure how good my prompting is, we’ll usually interact to arrive to a common understanding. With interns you absolutely do need to worry about how good your prompting is! You need to give them specific requirements, training, documentation, give them full access to the code base... 'prompting' an intern is called 'management'.
- ddrdrck_ 2y ago
  This might be the best definition I will come across of what it means to be an "IT project manager".

jappgar 2y ago

They've explicitly been trained/system-prompted to act that way. Because that's what the marketing teams at these AI companies want to sell. It's easy to override this though by asking the LLM to act as if it were less-confident, more hesitant, paranoid etc. You'll be fighting uphill against the alignment(marketing) team the whole time though, so ymmv.

kranuck 2y ago

> have no idea whether the LLM understood what I’m asking That's easy. The answer is it doesn't. It has no understanding of anything it does. > if it’s able to do it This is the hard part.

mercer 2y ago

Can I have some of those sorts of interns?

arthurcolle 2y ago

many many teams are actively building SOTA systems to do this in ways previously unimagined. you can enqueue tasks and do whatever you want. I gotta say as a current gen LLM programmer person, I can completely appreciate how bad they are now - I recently tweeted about how I "swore off" AI tools but like... there are many ways to bootstrap very powerful software or ML systems around or inside these existing models that can blow away existing commercial implementations in surprising ways

gmerc 2y ago
“building” is the easy part
- falcor84 2y ago
  building SOTA systems is the easy part?! Easy compared to what?
  - kristianp 2y ago
    Probably, to get them to work without hallucinating, or without failing a good percentage of the time.
    - falcor84 2y ago
      I wonder what would our world look like if these two expectations that you seem to be taking for granted were applied to our politicians.
      - AbstractH24 2y ago
        Are you suggesting people are satisfied with our politicians and aspire for other things to be just as good as them? What if we applied those two expectations to building construction? What if we didn’t?
        falcor84 2y ago
        I think it's always good to aspire for more, but we shouldn't be expecting perfect results in novel areas of technology. Taking up your construction metaphor, LLMs are now where construction was perhaps 3000 years ago; buildings weren't that sturdy, but even if the roofs leaked a bit, I'm sure it beat sleeping outside on a rainy night. We need to continue iterating.
        AbstractH24 2y ago

valval 2y ago

Really it just does what you tell it to. Have you tried telling it “ask me clarifying questions about all the APIs you need to solve this problem”? Huge contrast to human interns who aren’t experienced or smart enough to ask the right questions in the first place, and/or have sentimental reasons for not doing so.

aktuel 2y ago
I have to say, having to tell it to ask me clarifying questions DOES make it really look smart!
- arthurcolle 2y ago
  imagine if you make it keep going without having to reprompt it
  - carlmr 2y ago
    Isn't that the exact point of o1, that it has time to think for itself without reprompting?
    - arthurcolle 2y ago
      yeah but they aren't letting you see the useful chain of thought reasoning that is crucial to train a good model. Everyone will replicate this over next 6 months
      - optimalsolver 2y ago
        >Everyone will replicate this over next 6 months Not without a billion dollars worth of compute, they won't.
        arthurcolle 2y ago
        Are you sure its a billion? Helps with estimating the training run

raverbashing 2y ago

> A good intern will ask clarifying questions, tell me “I don’t know” Your expectations are bigger than mine (Though some will get stuck in "clarifying questions" and helplessness and not proceed neither)

williamdclt 2y ago
Note that we are talking about a “good” intern here
- TeMPOraL 2y ago
  Unreasonably good. Beyond fresh junior employee good. Also, that's your standard; 'MPSimmons said to treat the model as "naive but intelligent" intern, not a good one.
steveBK123 2y ago
Indeed. My expectation of a good intern is to produce nothing I will put in production, but show aptitude worth hiring them for. It's a 10 week extended interview with lots of social events, team building, tech talks, presentations, etc. Which is why I've liked the LLM analogy of "unlimited free interns".. I just think some people read that the exact opposite way I do (not very useful).
- Martinussen 2y ago
  If I had to respect the basic human rights of my LLM backends, it would probably be less appealing - but "Unlimited free smart-for-being-braindead zombies" might be a little more useful, at least?
  - steveBK123 2y ago
    Interns, at least on paper, have the optionality of getting better with time in observable obvious ways as they become grad hires, junior engineers, mid engineers etc. So far, 2 years of publicly accessible LLMs have not improved for intern replacement tasks at the rate a top 50% intern would be expected to.

singingfish 2y ago

> LLMs do none of that, they will take whatever you ask and give a reasonable-sounding output that might be anything between brilliant and nonsense. This is exactly why I’ve been objecting so much to the use of the term “hallucination” and maintain that “confabulation” is accurate. People who have spent enough time with acutelypsychotic people, and people experiencing the effects of long term alcohol related brain damage, and trying to tell computers what to do will understand why.

bartread 2y ago
I don't know that "confabulation" is right either: it has a couple of other meanings beyond "a fabricated memory believed to be true" and, of course, the other issue is that LLMd don't believe anything. They'll backtrack on even correct information if challenged.

berniedurfee 2y ago

I’m starting to think this is an unsolvable problem with LLMs. The very act of “reasoning” requires one to know that they don’t know something. LLMs are giant word Plinko machines. A million monkeys on a million typewriters. LLMs are not interns. LLMs are assumption machines. None of the million monkeys or the collective million monkeys are “reasoning” or are capable of knowing. LLMs are a neat parlor trick and are super powerful, but are not on the path to AGI. LLMs will change the world, but only in the way that the printing press changed the world. They’re not interns, they’re just tools.

idiotsecant 2y ago
I think LLMs are definitely on the path to AGI in the same way that the ball bearing was on the path to the internal combustion engine. I think its quite likely that LLMs will perform important functions within the system of an eventual AGI.
- swader999 2y ago
  This may be accurate. I wonder if there's enough energy in the world for this endeavour.
  - TeMPOraL 2y ago
    Of course! 1. We've barely scratched the surface of this solution space; the focus only recently started shifting from improving model capabilities to improving training costs. People are looking at more efficient architectures, and lots of money is starting to flow in that direction, so it's a safe bet things will get significantly more efficient. 2. Training is expensive, inference is cheap, copying is free. While inference costs add up with use, they're still less than costs of humans doing the equivalent work, so out of all things AI will impact, I wouldn't worry about energy use specifically.
  - int_19h 2y ago
    Humans don't require immense amounts of energy to function. The reasons why LLMs do is because we are essentially using brute force as the methodology for making them smarter for the lack of better understanding of how this works. But this then gives us a lot of material to study to figure that part out for future iterations of the concept.

ryoshu 2y ago

1000% this. LLMs can't say "I don't know" because they don't actually think. I can coach a junior to get better. LLMs will just act like they know what they are doing and give the wrong results to people who aren't practitioners. Good on OAI calling their model Strawberry because of Internet trolls. Reactive vs proactive.

jug 2y ago
This surprises me. I made a simple chat fed with PDF's and using LangChain and it by default said it didn't know if I asked questions outside of the corpus. It was a simple matter of the confidence score getting too low?
bartread 2y ago
I get a lot of value out of ChatGPT but I also, fairly frequently, run into issues here. The real danger zones are areas that lie at or just beyond the edges of my own knowledge in a particular area. I'd say that most of my work use of ChatGPT does in fact save me time but, every so often, ChatGPT can still bullshit convincingly enough to waste an hour or two for me. The balance is still in its favour, but you have to keep your wits about you when using it.
- ryoshu 2y ago
  Agreed, but the problem is if these things replace practitioners (what every MBA wants them to do), it's going to wreck the industry. Or maybe we'll get paid $$$$ to fix the problems they cause. GPT-4 introduced me to window functions in SQL (haven't written raw SQL in over a decade). But I'm experienced enough to look at window functions and compare them to subqueries and run some tests through the query planner to see what happens. That's knowledge that needs to be shared with the next generation of developers. And LLMs can't do that accurately.
  - lupire 2y ago
    Optimizing a query is certainly something the machine (not necessarily the LLM part) can do better than the human, for 99.9% of situations and people. PostgreSQL developers are oposed to query execution hints, because if a human knows a better way to execute a query, the devs want to put that knowledge into the planner.

0xdeadbeefbabe 2y ago

A lot of interns are overconfident though

naasking 2y ago

> That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know” or “I’m not sure I did it right”. An intern that grew up in a different culture then, where questioning your boss is frowned upon. The point is that the way to instruct this intern is to front-load your description of the problem with as much detail as possible to reduce ambiguity.

noisy_boy 2y ago

I feel like it almost always starts well, given the full picture, but then for non-trivial stuff, gets stuck towards the end. The longer the conversation goes, the more wheel-spinning occurs and before you know it, you have spent an hour chasing that last-mile-connectivity. For complex questions, I now only use it to get the broad picture and once the output is good enough to be a foundation, I build the rest of it myself. I have noticed that the net time spent using this approach still yields big savings over a) doing it all myself or b) keep pushing it to do the entire thing. I guess 80/20 etc.

skybrian 2y ago
Yes, I’ve seen that too. One reason it will spin its wheels is because it “prefers” patterns in transcripts and will try to continue them. If it gets something wrong several times, it picks up on the “wrong answers” pattern. It’s better not to keep wrong answers in the transcript. Edit the question and try again, or maybe start a new chat.
mlsu 2y ago
This is the way. I've had this experience many times: - hey, can you write me a thing that can do "xyz" - sure, here's how we can do "xyz" (gets some small part of the error handling for xyz slightly wrong) - can you add onto this with "abc" - sure. in order to do "abc" we'll need to add "lmn" to our error handling. this also means that you need "ijk" and "qrs" too, and since "lmn" doesn't support "qrs" out of the box, we'll also need a design solution to bridge the two. Let me spend 600 more tokens sketching that out. - what if you just use the language's built in feature here in "xyz"? does't that mean we can do it with just one line of code? - yes, you're absolutely right. I'm sorry for making this over complicated. If you don't hit that kill switch, it just keeps doubling down on absurdly complex/incorrect/hallucinatory stuff. Even one small error early in the chain propagates. That's why I end up very frequently restarting conversations in a new chat or re-write my chat questions to remove bad stuff from the context. Without the ability to do that, it's nearly worthless. It's also why I think we'll be seeing absurdly, wildly wrong chains of thought coming out of o1. Because "thinking" for 20s may well cause it to just go totally off the rails half the time.

yukIttEft 2y ago

Makes me wonder if "I don't know" could be added to LLM: whenever an activation has no clear winner value (layman here), couldn't this indicate low response quality?

Regic 2y ago
This exists and does work to some degree, e.g. Detecting hallucinations in large language models using semantic entropy https://www.nature.com/articles/s41586-024-07421-0 https://www.nature.com/articles/s41586-024-07421-0

jacobn 2y ago

Is this a dataset issue more than an LLM issue? As in: do we just need to add 1M examples where the response is to ask for clarification / more info? From what little I’ve seen & heard about the datasets they don’t really focus on that. (Though enough smart people & $$$ have been thrown at this to make me suspect it’s not the data ;)

richerram 2y ago

I think this is the main issue with these tools... what people are expecting of them. We have swallowed the pill that LLMs are supposed to be AGI and all that mumbo jumbo, when they are just great tools and as such one needs to learn to use the tool the way it works and make the best of it, nobody is trying to hammer a nail with a broom and blaming the broom for not being a hammer...

koe123 2y ago
I completely agree. To me the discussion here reads a little like: “Hah. See? It cant do everything!”. It makes me wonder if the goal is to convince each other that: yes, indeed, humans are not yet replaced. It’s next token regression, of course it can’t truely introspect. That being said LLMs are amazing tools and o1 is yet another incremental improvement and I welcome it!

jasondigitized 2y ago

Have you ever worked with an intern? They have personalities and expectations that need to be managed. They get sick. The get tired. They want to punch you if you treat them like a 24-7 bird dog. It's so much easier to not let perfect be the enemy of the good and just rapid fire ALL day at a LLM for any and everything I need help with. You can also just not use the LLM. Interns need to be 'fed' work or the ROI ends upside down. Is a LLM as good as a top tier intern. No, but with a LLM I can have 10 pretty good interns by opening 10 tabs.

sqeaky 2y ago
I think listing off a set of pros and cons for interns and LLMs misses the point, they seem like categorically different kinds of intelligence.
ww2supercut 2y ago
The LLMs are getting better and better at a certain kind of task, but there's a subset of tasks that I'd still much rather have any human than an LLM, today. Even something simple, like "Find me the top 5 highest grossing movies of 2023" it will take a long time before I trust an LLM's answer, without having a human intern verify the output.

Closi 2y ago

> With an intern, I don’t need to measure how good my prompting is, we’ll usually interact to arrive to a common understanding. With interns you absolutely do need to worry about how good your prompting is! You need to give them specific requirements, training, documentation, give them full access to the code base... 'prompting' an intern is called 'management'.

ddrdrck_ 2y ago
This might be the best definition I will come across of what it means to be an "IT project manager".