You might find the work coming out of MIT's Fluid Interfaces group fascinating. The "Smarter Objects" (http://fluid.media.mit.edu/projects/smarter-objects http

You might find the work coming out of MIT's Fluid Interfaces group fascinating. The "Smarter Objects" (http://fluid.media.mit.edu/projects/smarter-objects http://fluid.media.mit.edu/projects/smarter-objects) project seem to tackle some of the things you're talking about. This article (http://singularityhub.com/2013/05/20/virtual-and-real-objects-meet-and-become-smarter-objects-at-mit/ http://singularityhub.com/2013/05/20/virtual-and-real-object...) explains it a bit more.