I think there are engineers working for crawler companies who are paid well enough to figure out how to do this without kneecapping the most well-known noncomme

I think there are engineers working for crawler companies who are paid well enough to figure out how to do this without kneecapping the most well-known noncommercial projects still surviving on the capitalized internet.