4 ms·
But the goal is to prevent spam in the first place. I don't think these bots will verify robots.txt to see if the spamming is effective. They just spam anything
by _ikke_ 6y ago
But the goal is to prevent spam in the first place. I don't think these bots will verify robots.txt to see if the spamming is effective. They just spam anything they can get their hands on.
- remram 6y agoThey probably don't have general code that spam any form, it's more likely that they have code specific to GitLab CE instances that knows to post snippets. If GitLab changes their default configuration so that those snippets are no longer indexed by Google, the spammer are likely to stop using that GitLab CE spamming script, after a while.