4 ms·

I briefly worked with a Watson team on a cool idea to map a person's 'knowledge space' (or probable knowledge space given their background) against Watson's kno

I briefly worked with a Watson team on a cool idea to map a person's 'knowledge space' (or probable knowledge space given their background) against Watson's knowledge space and guide them to relevant learning materials and journal articles and the like. The idea was to save people time so they aren't rehashing stuff they know down pat or jumping ahead into material they cannot understand but, instead, find that next step into what they almost know. The idea from there would be to let them specify where they want to go and guide them, step by step, exposure by exposure, to that summit. In a few days, it turned into Just Another News Article Recommendation Engine based on interest and similar profiles with other clients. Yawn.

ironchief 8y ago
I'm interested in the original idea. Can you expand on how your ideal system would function?
- clavalle 8y ago
  Well, the hope was that Watson, having explored and built a connected knowledge graph from various sources, could ask probing, adaptive questions to find out where a person landed. So, say I'm an undergrad at a good university and I tell the system "I'm interested in Computer Science. I am particularly interested in Scientific Computing and would like to get to a graduate level of knowledge." The system might ask "Sort the following operations by their worst-case run-time...". Then, if they do well there, maybe "Which of these two examples of auto-parallelization using Matlab's parfor would fail to parallize the code.." or something like that. Over the course of so many questions, the system would start to paint a more and more reliable picture of where contours of a person's knowledge. This is time-consuming, of course, but over time it would get easier and faster to find contours by using the 'average' of people with similar backgrounds as a starting point. Once a fairly good mapping is done of the person to Watson's cognitive model, Watson would need to trace back to the source(s) of nearby concepts and offer them to the user, which, ideally, are then rated by the user for relevance and perceived difficulty to further refine the person's model and rank the material offered for that particular profile. Now imagine a Grad Student asking a similar question. Or a middle school student. What would those interactions look like? The mappings? The suggestions? Don't get me wrong...mapping a person's knowledge space is a Very Hard Problem. Watson takes a kitchen sink approach that just isn't possible for a human being. And maybe it wouldn't be possible to tease apart the resulting cognitive model into tidy nodes enough to map anything to. These were questions I'd hoped that IBM could help answer. Instead, it was on to the easy, well-understood problem and solution.
  - lozenge 8y ago
    This already exists, it's called adaptive testing. I guess the new part would be modelling somebody's ability, not in one topic, but many related topics.
    - clavalle 8y ago
      Sure. Existing adaptive tests were an inspiration for the idea, of course. But putting together a good adaptive test is, itself, very time-consuming. And the test itself doesn't do more than measure ability. The crux of the idea is to use the adaptive test to suggest learning material from a wide swathe of sources to the individual.
      - robbiemitchell 8y ago
        This is what Knewton is all about. knewton.com
        clavalle 8y ago
        Knewton is very cool, thanks for the link. I'm assuming they break things down into traditional objective units within a standard curriculum and material that is relevant. That's great but it is a very manual process. Remember Yahoo curated keyword recommendations from the 90s? This Knewton approach is more like that. What I'd like to see is more like a Google Search in this analogy...something more flexible and comprehensive.
        robbiemitchell 8y ago
        The basic idea is that you might start with a manual linkage between concepts, derived from linkages between content developed by subject matter experts, but over time it organically morphs into a knowledge graph that describes how certain concepts relate to and build on each other, and for which scenarios of learners. That knowledge graph combined with a concrete goal (mastery level) and time to get there (deadline) can then be used to recommend to a specific learner what material to study or activities to do next. It's theoretically possible to do this even more organically, but you need clearly tagged educational "stuff" (written content, lectures, activities, etc.) and, perhaps more importantly, clear ways to measure outcomes as a result of interacting with that "stuff" (typically quizzes which themselves have been clearly calibrated).
hinkley 8y ago
In a similar vein, I find a machine beating a grand master fairly boring. Show me a machine that can teach someone to become a master (or even a very good amateur) and we can talk. That is, don’t beat the game, beat the opponent, understand why, and then model an adaptation strategy for the opponent (teach).
- dahart 8y ago
  > In a similar vein, I find a machine beating a grand master fairly boring. I know a lot of people felt that way even before it happened. It feels like an accounting problem rather than intelligence. But, it's fun to remember when it was thought by some very smart people that chess required more accounting than was ever possible by a machine, so beating a grand master would have to be a demonstration of intelligence. > don’t beat the game, beat the opponent, understand why, and then model an adaptation strategy for the opponent (teach). Yeah that would be closer. My favorite Turing test, if you will, is whether the AI can tell you you're asking it the wrong question. If Watson got bored of beating grand masters at chess and started refusing to play, maybe then a case could be made it was reasoning.
  - lerpa 8y ago
    > Watson got bored of beating grand masters at chess and started refusing to play That's when you reboot it, if that doesn't fix it good old persuasive maintenance techniques can also come into play
    - dahart 8y ago
      "I'm afraid I can't let you do that, Dave" - HAL
  - opportune 8y ago
    A lot of AI "milestones" were actually caused by improvements in computing itself, IMO. That is, they weren't facilitated by novel algorithms or engineering solutions specific to the milestone itself. They kind of just happened, either because computers got faster (e.g. disk I/O, ram size, processor speeds and caching) or because someone finally decided to throw enough distributed computation at the problem. People knew about the main algorithms that Deep Blue used (alpha-beta minimax with heuristic evaluation) for decades before Deep Blue existed. It was simply a matter of waiting until IBM found marketing value in dedicating a top 300 world super computer to beating Gary Kasparov

ironchief 8y ago

I'm interested in the original idea. Can you expand on how your ideal system would function?

clavalle 8y ago
Well, the hope was that Watson, having explored and built a connected knowledge graph from various sources, could ask probing, adaptive questions to find out where a person landed. So, say I'm an undergrad at a good university and I tell the system "I'm interested in Computer Science. I am particularly interested in Scientific Computing and would like to get to a graduate level of knowledge." The system might ask "Sort the following operations by their worst-case run-time...". Then, if they do well there, maybe "Which of these two examples of auto-parallelization using Matlab's parfor would fail to parallize the code.." or something like that. Over the course of so many questions, the system would start to paint a more and more reliable picture of where contours of a person's knowledge. This is time-consuming, of course, but over time it would get easier and faster to find contours by using the 'average' of people with similar backgrounds as a starting point. Once a fairly good mapping is done of the person to Watson's cognitive model, Watson would need to trace back to the source(s) of nearby concepts and offer them to the user, which, ideally, are then rated by the user for relevance and perceived difficulty to further refine the person's model and rank the material offered for that particular profile. Now imagine a Grad Student asking a similar question. Or a middle school student. What would those interactions look like? The mappings? The suggestions? Don't get me wrong...mapping a person's knowledge space is a Very Hard Problem. Watson takes a kitchen sink approach that just isn't possible for a human being. And maybe it wouldn't be possible to tease apart the resulting cognitive model into tidy nodes enough to map anything to. These were questions I'd hoped that IBM could help answer. Instead, it was on to the easy, well-understood problem and solution.
- lozenge 8y ago
  This already exists, it's called adaptive testing. I guess the new part would be modelling somebody's ability, not in one topic, but many related topics.
  - clavalle 8y ago
    Sure. Existing adaptive tests were an inspiration for the idea, of course. But putting together a good adaptive test is, itself, very time-consuming. And the test itself doesn't do more than measure ability. The crux of the idea is to use the adaptive test to suggest learning material from a wide swathe of sources to the individual.
    - robbiemitchell 8y ago
      This is what Knewton is all about. knewton.com
      - clavalle 8y ago
        Knewton is very cool, thanks for the link. I'm assuming they break things down into traditional objective units within a standard curriculum and material that is relevant. That's great but it is a very manual process. Remember Yahoo curated keyword recommendations from the 90s? This Knewton approach is more like that. What I'd like to see is more like a Google Search in this analogy...something more flexible and comprehensive.
        robbiemitchell 8y ago
        The basic idea is that you might start with a manual linkage between concepts, derived from linkages between content developed by subject matter experts, but over time it organically morphs into a knowledge graph that describes how certain concepts relate to and build on each other, and for which scenarios of learners. That knowledge graph combined with a concrete goal (mastery level) and time to get there (deadline) can then be used to recommend to a specific learner what material to study or activities to do next. It's theoretically possible to do this even more organically, but you need clearly tagged educational "stuff" (written content, lectures, activities, etc.) and, perhaps more importantly, clear ways to measure outcomes as a result of interacting with that "stuff" (typically quizzes which themselves have been clearly calibrated).

hinkley 8y ago

In a similar vein, I find a machine beating a grand master fairly boring. Show me a machine that can teach someone to become a master (or even a very good amateur) and we can talk. That is, don’t beat the game, beat the opponent, understand why, and then model an adaptation strategy for the opponent (teach).

dahart 8y ago
> In a similar vein, I find a machine beating a grand master fairly boring. I know a lot of people felt that way even before it happened. It feels like an accounting problem rather than intelligence. But, it's fun to remember when it was thought by some very smart people that chess required more accounting than was ever possible by a machine, so beating a grand master would have to be a demonstration of intelligence. > don’t beat the game, beat the opponent, understand why, and then model an adaptation strategy for the opponent (teach). Yeah that would be closer. My favorite Turing test, if you will, is whether the AI can tell you you're asking it the wrong question. If Watson got bored of beating grand masters at chess and started refusing to play, maybe then a case could be made it was reasoning.
- lerpa 8y ago
  > Watson got bored of beating grand masters at chess and started refusing to play That's when you reboot it, if that doesn't fix it good old persuasive maintenance techniques can also come into play
  - dahart 8y ago
    "I'm afraid I can't let you do that, Dave" - HAL
- opportune 8y ago
  A lot of AI "milestones" were actually caused by improvements in computing itself, IMO. That is, they weren't facilitated by novel algorithms or engineering solutions specific to the milestone itself. They kind of just happened, either because computers got faster (e.g. disk I/O, ram size, processor speeds and caching) or because someone finally decided to throw enough distributed computation at the problem. People knew about the main algorithms that Deep Blue used (alpha-beta minimax with heuristic evaluation) for decades before Deep Blue existed. It was simply a matter of waiting until IBM found marketing value in dedicating a top 300 world super computer to beating Gary Kasparov