I am in complete agreement that augmented reality ("AR") has the potential to be a game changer for physical tasks. It is the kind of thing that has the potent

I am in complete agreement that augmented reality ("AR") has the potential to be a game changer for physical tasks. It is the kind of thing that has the potential to raise the effective IQ of the user quite substantially. The military has spent a fair amount of time working on something similar to what you describe for vehicular repair. I think you might find the videos off of this page ( http://graphics.cs.columbia.edu/projects/armar/index.htm http://graphics.cs.columbia.edu/projects/armar/index.htm ) and this page ( http://singularityhub.com/2010/01/11/augmented-reality-to-help-military-mechanics-fix-vehicles-video/ http://singularityhub.com/2010/01/11/augmented-reality-to-he... ) very suggestive of the kind of thing you are looking for in medicine. I haven't seen the same level of thing yet for medical systems, though it may exist. I have seen some work on using AR for visualization of imaging (e.g. map the CT scan onto the body I am looking at). I think the sort of direct procedural guidance you describe is probably harder for people than it is for vehicular repair. At the core that is probably just a computational problem though.