I started something similar a while ago, but more like a window manager. It would track the user's movement and recalculate all the window positions based on th

I started something similar a while ago, but more like a window manager. It would track the user's movement and recalculate all the window positions based on the perceived head position - making the monitor more like a real window. This would enable you to peek behind window edges by moving your head to the side or up/down. The main barriers I faced was the processing power required to monitor the user's position accurately - needing a camera with relatively good resolution (to get accurate depth perception) and good frame rate (to reduce latency). Also, what to do when another person enters the scene? (Keep tracking the primary / previous target) Also, using it as a 3d desktop / window manager is probably not worth that much drag on your system - but I could see it being quite useful for cad systems / 3d modelling / prototype development. Also, developing this way could lead to some interesting developments in the UI field - there's already some development here for the disabled, so is not entirely new - but hasn't really made it to the mainstream desktop...