I remember writing a dumb parser for robots.txt. I have to agree, robots.txt is simplistic but so non-standard. I wonder why search engines can't just say NO to

I remember writing a dumb parser for robots.txt. I have to agree, robots.txt is simplistic but so non-standard. I wonder why search engines can't just say NO to this. Does search engines today still honor robots.txt? Here's my shameless plug: https://github.com/yeukhon/robots-txt-scanner https://github.com/yeukhon/robots-txt-scanner I still remember writing most of this on Caltrain one morning heading to SF visiting someone I dearly loved.....