2 ms·
robots.txt does nothing for helping search engines understand what a page is about, just how to crawl. How do you see this aiding GPT? You're probably thinking
by verdverm 4y ago
robots.txt does nothing for helping search engines understand what a page is about, just how to crawl. How do you see this aiding GPT?
You're probably thinking of JSON-LD, which obviously already exists, but is not some golden truth. Search engines have been using machine learning to understand the content and topics of sites for a long time.
GPT is token prediction, you still want to do all the same things for your website to aid discovery and understanding.