3 ms·

More than a decade ago I worked as a research programmer on a similar AI tutoring product for hierarchical skills like mathematics at Carnegie Mellon (in conjun

More than a decade ago I worked as a research programmer on a similar AI tutoring product for hierarchical skills like mathematics at Carnegie Mellon (in conjunction with the Pittsburgh Science of Learning Center and Carnegie Learning). The system would prompt students with problems that incorporated dozens of sub-skills, each of which incorporated other sub-skills and so on. If the student gets the problem right, all requisite sub-skills are marked as reasonably strong. If they get it wrong then the statistical model concludes that at least one sub-skill must be weak/faulty/missing, and when selecting subsequent problems the system would incorporate problems with different subsets of the original sub-skills to identify the missing sub-skill(s), with the goal being to spend time reinforcing only those skills that the student has not mastered. I no longer work in that domain, but I recall it being amazingly efficient -- it often only took a dozen questions to assess a student's understanding of thousands of skills.

BenoitP 2y ago
Seems quite close to the 'active learning' ML topic: let the model choose the question that will bring the best information return. And this kind of decomposition using simple knowledge pieces (logical axioms?) is IMHO what we have to do to bring LLMs to senses. These pieces should light up in the intermediate embeddings inside the LLM. It won't really be more intelligent, but it'll model reality better.
owenpalmer 2y ago
Reminds me of using git-bisect
ahmni_dev 2y ago
Agreed that this article is strongly related to Intelligent Tutoring Systems like the Carnegie Tutor. I recently posted a brief literature review video of the field for those interested in the bigger picture around the article: https://youtu.be/SDb9-LGRrCU?si=t6FM3832-_OwwpSz https://youtu.be/SDb9-LGRrCU?si=t6FM3832-_OwwpSz The TLDR of the relevant sections (see video description for relevant timestamps) - Models like ACT-R represent a model of the domain and a model of the student - Knowledge tracing is an ML-heavy field that predicts skills, state, etc (like parent says) - Recommender Systems use that knowledge to intelligently recommend next problems

BenoitP 2y ago

Seems quite close to the 'active learning' ML topic: let the model choose the question that will bring the best information return. And this kind of decomposition using simple knowledge pieces (logical axioms?) is IMHO what we have to do to bring LLMs to senses. These pieces should light up in the intermediate embeddings inside the LLM. It won't really be more intelligent, but it'll model reality better.

owenpalmer 2y ago

Reminds me of using git-bisect

ahmni_dev 2y ago

Agreed that this article is strongly related to Intelligent Tutoring Systems like the Carnegie Tutor. I recently posted a brief literature review video of the field for those interested in the bigger picture around the article: https://youtu.be/SDb9-LGRrCU?si=t6FM3832-_OwwpSz https://youtu.be/SDb9-LGRrCU?si=t6FM3832-_OwwpSz The TLDR of the relevant sections (see video description for relevant timestamps) - Models like ACT-R represent a model of the domain and a model of the student - Knowledge tracing is an ML-heavy field that predicts skills, state, etc (like parent says) - Recommender Systems use that knowledge to intelligently recommend next problems