Have you actually tried indexing on your own? Most websites whitelist the google crawler and backlist anyone else they detect. If you want to be nice and resp

Have you actually tried indexing on your own? Most websites whitelist the google crawler and backlist anyone else they detect. If you want to be nice and respect robots.txt a lot of those whitelist only Googlebot. This isn't the 90s where anyone can crawl the web without countermeasures.