3 ms·

I hope this throttling includes the VCS web frontends. It used to be that the hg (mercurial) and some other VCS (I forgot which one it was) web frontend were cr

by secure 13y ago

I hope this throttling includes the VCS web frontends. It used to be that the hg (mercurial) and some other VCS (I forgot which one it was) web frontend were crawled mercilessly by someone (unidentified so far AFAIK). That used to put so much load on alioth that the git frontend would refuse to work, which was very annoying for me :). That being said, I hope that either a) the author of the post is exaggerating or b) the alioth admins will improve the throttling to better target abusive users and not legitimate websites such as this one

igouy 13y ago
a) I was not exaggerating b) All the upstream Fusion Forge generated pages, for all 1,000 projects, were being served without any HTML robots meta tags -- which, of course, is asking for <meta name="robots" content="index,follow" /> and duplicate pages were being served from 4 subdomains, and the 1,000 projects also had their own web pages without any HTML robots meta tags... In other words, googlebot was visiting alioth because alioth was repeatedly asking googlebot to visit. Luckily I found a way for the alioth admins to take some control of the Fusion Forge generated pages, that did not require changes to Fusion Forge. Header set X-Robots-Tag "noindex, nofollow" https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag https://developers.google.com/webmasters/control-crawl-index... The alioth admins were willing to ease throttling and see if asking robots not to index and not to crawl Fusion Forge pages reduced the server load. Currently server errors for benchmarksgame subdomain are below 70% and robots.txt below 5% and the benchmarks game pages are indexed.