4 ms·
Having an admin or test site publicly accessible is a really bad idea, for exactly this reason. If any search engine comes across a link to this site, it will c
by eklitzke 17y ago
Having an admin or test site publicly accessible is a really bad idea, for exactly this reason. If any search engine comes across a link to this site, it will crawl the site. Even if it doesn't promote the test site above you main site, it can cause a lot of problems:
* googlebot finds a ton of new content, and starts crawling this new content rather quickly, increasing load on your machines
* since multiple domains are detected serving the same content, it could make your site look spammy and decrease your PR
* there's no guarantee that your main site content will show up first in google search results, which is obviously bad
I worked at a site that encountered this problem in a similar manner. The way it worked was the main site was www.example.com, and we had a CDN + squid in front of static.example.com. Only static assets like images were served from the CDN version of the site. However, the code didn't actually enforce this (i.e. it was purely enforced by only linking to static assets). Somehow Google found a link to the cached copy of the site and started crawling everything. Badness ensued.
It sounds like your situation is pretty easy to fix -- enforce that only internal IPs can access the test site (or require a special cookie, etc.). In the previous example that I gave, we had to explicitly whitelist certain servlets as acceptable to be served from the static site (and enforce that in the code).