I wasn't saying that you'd estimate normals from the point cloud. You'd need to estimate the normals separately and store the world position and world normal a

I wasn't saying that you'd estimate normals from the point cloud. You'd need to estimate the normals separately and store the world position and world normal along with the color. This should be possible as these values can be represented as a color texture, so you should be able to construct something that renders a normal map and depth map from any angle just like this renders the color currently.