To address your last bit about raycasting: Rendering a scene to another buffer is a good way, although you do need some minor tricks to get information such as

To address your last bit about raycasting: Rendering a scene to another buffer is a good way, although you do need some minor tricks to get information such as position and normal of the intersection, particularly so with high precision. It's worth noting that this approach causes a very costly GPU/CPU sync when hit testing needs to be done and the scene has changed since last time (because glReadPixels() has to be called to pull in the data from VRAM to RAM). This is wasteful if the scene changes every frame, as you're essentially limiting the performance of your GPU to the performance of your CPU. Rendering to a smaller off-screen buffer (thus sacrificing precision) or calling glReadPixels() at most on each Nth frame and relying on temporal coherence to save you from the most unpleasant false positives and false negatives.