This post deserves a long reply. You seem to think that ray tracing as a very limited thing. I guess it stems from the fact that the nomenclature has become a

This post deserves a long reply. You seem to think that ray tracing as a very limited thing. I guess it stems from the fact that the nomenclature has become a bit murky in recent years. Ray tracing often is used to refer to the initial algorithm developed by Whitted. This has pretty much all these limitations you mention, but it is an obsolete method. What usually gets mixed in with ray tracing in marketing and non-expert discussions is also path tracing and its various variations. These methods are capable of solving the rendering equation exactly for a huge range of scenarios. Unfortunately, some variants deal with certain situations better than with others and there is no overall best algorithm. But you can most certainly trace absolutely perfect soft shadows or complex caustics with these methods. Color spills are a walk in the park even for simple unidirectional path tracing. The main limiting factor right now is that these are all Monte Carlo methods and need to draw high numbers of samples for each pixel to converge. This is why fast and accurate denoisers are such a huge deal at the moment. They are another hack, a last one that ideally fades away as the number of rays per second increases. Beyond that, it "just" comes down to modelling surfaces and materials accurately enough to get a photorealistic image. As you mention photon mapping: it has pretty severe limitations, notably around the handling of specular paths and the biased artifacts created by the filter kernel (loght leaks, certain SDS-type paths aren't covered). There are some improvements like (A)PPM which converge to unbiased solutions with infinite runtime. At that point, path tracing based methods become more attractive w.r.t. runtime, memory requirements and implementation complexity.