2 ms·
That's a valid robots.txt, but "proper" is entirely dependent on what you want to achieve. If you aren't looking to treat different bots differently, and are lo
by bragr 2y ago
That's a valid robots.txt, but "proper" is entirely dependent on what you want to achieve. If you aren't looking to treat different bots differently, and are looking allow all of your site to be indexed, then that is exactly what you want.