3 ms·

> a Junior (in ANY subject) has the ability to LEARN so much faster with an AI research assistant I’m not seeing this. And based on what we’re seeing at the un

> a Junior (in ANY subject) has the ability to LEARN so much faster with an AI research assistant I’m not seeing this. And based on what we’re seeing at the university level, I’m not expecting to.

sonofhans 6mo ago
Yes, I agree, the skills are orthogonal. Digital typesetting is vastly quicker than manually putting down metal type, and since you’re exposed to more type you have the opportunity to learn faster. But getting good at typography with digital tools will help you very little if you need to lay out type manually.
- JumpCrisscross 6mo ago
  > getting good at typography with digital tools will help you very little if you need to lay out type manually The analogy is unlimited typing in Gmail won’t make you a better writer or typesetter on its own.
vanuatu 6mo ago
I wonder how much of this is due to poor incentives at the university level? I've seen this work well at a job when there's a feedback loop for juniors that incentivized them to learn with more scope and compensation
- judahmeek 6mo ago
  How did that business evaluate that the juniors were actually mastering concepts they had not known before?
  - vanuatu 6mo ago
    Mentorship & engaged managers
keeda 6mo ago
I think the key word is ability, and I fully agree with that. Using GenAI as a teaching aid can supercharge learning, especially as it makes it very easy to learn by doing. The problem is that people use GenAI to do and hence don't learn. (The preliminary research so far supports this: using AI to do the hard assignments produces poor learning outcomes, but using AI as a tutor, or even just for help with the hard assignments, produces slightly better learning outcomes.) I think what you're seeing is the effect of the incentives of the system. The system uses simplistic numbers like grades as proxies for actual learning, and these grades heavily influence students' job prospects, and so you're simply seeing Goodhart's Law in action. Given how easy current methods of skill assessment are to game with AI, my guess is the entire system has to be overhauled.
- JumpCrisscross 6mo ago
  > using AI as a tutor, or even just for help with the hard assignments, produces slightly better learning outcomes Source? The few people I’ve seen try to do this wind up with a terrible understanding of the material, with large knowledge gaps and one or two fundamental fuckups. In every case, an introductory textbook would have been better. (It would also have been harder.)
  - keeda 6mo ago
    For coding specifically (there are many studies out there by now, but I know of these offhand): https://www.mdpi.com/2076-3417/14/10/4115 https://www.mdpi.com/2076-3417/14/10/4115 -- probably the earliest one of its kind, finds over-reliance degrades critical skills but supplementary use is mostly harmless. https://arxiv.org/html/2601.20245v2 https://arxiv.org/html/2601.20245v2 -- Anthropic's study, same as above except supplementary use (like clarifying concepts) can actually be beneficial. https://scale.stanford.edu/ai/repository/ai-meets-classroom-when-do-large-language-models-harm-learning https://scale.stanford.edu/ai/repository/ai-meets-classroom-... -- "Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to solve practice exercises for them and thus do not invest sufficient own mental effort." Interestingly, they found simply disabling copy-paste on the chatbot interface resulted in better outcomes! Beyond coding, I recently came across this new meta-study; largely positive findings (which it admits may be biased) but does highlight evidence of negative effects of over-reliance: https://www.sciencedirect.com/science/article/pii/S2666920X25001699 https://www.sciencedirect.com/science/article/pii/S2666920X2... (Multiple studies find that the outcome depends on how AI is used. Surprisingly, incorrect guidance / unreliability / hallucinations appear to be a bigger problem than over-reliance! That could also explain poor performance in some cases.) My intuition, supported by these studies, is that as long as students are willing to do the hard cognitive work -- for which there is no substitute, really -- having LLM assistance is a boon. Which makes sense, it's comparable to having a tutor explain difficult concepts. This is why in my mind the real problem is that the incentives to use AI as a crutch are just too strong.

sonofhans 6mo ago

Yes, I agree, the skills are orthogonal. Digital typesetting is vastly quicker than manually putting down metal type, and since you’re exposed to more type you have the opportunity to learn faster. But getting good at typography with digital tools will help you very little if you need to lay out type manually.

JumpCrisscross 6mo ago
> getting good at typography with digital tools will help you very little if you need to lay out type manually The analogy is unlimited typing in Gmail won’t make you a better writer or typesetter on its own.

vanuatu 6mo ago

I wonder how much of this is due to poor incentives at the university level? I've seen this work well at a job when there's a feedback loop for juniors that incentivized them to learn with more scope and compensation

judahmeek 6mo ago
How did that business evaluate that the juniors were actually mastering concepts they had not known before?
- vanuatu 6mo ago
  Mentorship & engaged managers

keeda 6mo ago

I think the key word is ability, and I fully agree with that. Using GenAI as a teaching aid can supercharge learning, especially as it makes it very easy to learn by doing. The problem is that people use GenAI to do and hence don't learn. (The preliminary research so far supports this: using AI to do the hard assignments produces poor learning outcomes, but using AI as a tutor, or even just for help with the hard assignments, produces slightly better learning outcomes.) I think what you're seeing is the effect of the incentives of the system. The system uses simplistic numbers like grades as proxies for actual learning, and these grades heavily influence students' job prospects, and so you're simply seeing Goodhart's Law in action. Given how easy current methods of skill assessment are to game with AI, my guess is the entire system has to be overhauled.

JumpCrisscross 6mo ago
> using AI as a tutor, or even just for help with the hard assignments, produces slightly better learning outcomes Source? The few people I’ve seen try to do this wind up with a terrible understanding of the material, with large knowledge gaps and one or two fundamental fuckups. In every case, an introductory textbook would have been better. (It would also have been harder.)
- keeda 6mo ago
  For coding specifically (there are many studies out there by now, but I know of these offhand): https://www.mdpi.com/2076-3417/14/10/4115 https://www.mdpi.com/2076-3417/14/10/4115 -- probably the earliest one of its kind, finds over-reliance degrades critical skills but supplementary use is mostly harmless. https://arxiv.org/html/2601.20245v2 https://arxiv.org/html/2601.20245v2 -- Anthropic's study, same as above except supplementary use (like clarifying concepts) can actually be beneficial. https://scale.stanford.edu/ai/repository/ai-meets-classroom-when-do-large-language-models-harm-learning https://scale.stanford.edu/ai/repository/ai-meets-classroom-... -- "Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to solve practice exercises for them and thus do not invest sufficient own mental effort." Interestingly, they found simply disabling copy-paste on the chatbot interface resulted in better outcomes! Beyond coding, I recently came across this new meta-study; largely positive findings (which it admits may be biased) but does highlight evidence of negative effects of over-reliance: https://www.sciencedirect.com/science/article/pii/S2666920X25001699 https://www.sciencedirect.com/science/article/pii/S2666920X2... (Multiple studies find that the outcome depends on how AI is used. Surprisingly, incorrect guidance / unreliability / hallucinations appear to be a bigger problem than over-reliance! That could also explain poor performance in some cases.) My intuition, supported by these studies, is that as long as students are willing to do the hard cognitive work -- for which there is no substitute, really -- having LLM assistance is a boon. Which makes sense, it's comparable to having a tutor explain difficult concepts. This is why in my mind the real problem is that the incentives to use AI as a crutch are just too strong.