17 ms·
Anyone thinking of moving to their own Gitlab instance with Gitlab CE-- either stay on Github or prepare to waste your time dealing with user spam bots that pol
by jancsika 6y ago
Anyone thinking of moving to their own Gitlab instance with Gitlab CE-- either stay on Github or prepare to waste your time dealing with user spam bots that pollute your site's search results.
In other words-- if you want the common use case for a FOSS project:
1. publicly viewable main repository with publicly viewable issue tracker
2. requirement to log in to view all snippets, user profiles, perhaps even other repos as enforced by administrator settings (otherwise SEO bots will leverage these features to eat your search results)
3. anyone with an email can sign up to post issues to the main repo's issue tracker
There is no combination of settings in Gitlab CE to achieve this. Any sane approach has to leave out step #2. That means that your Gitlab instance gets hammered with user spam from bots which then get indexed in Google search results for your site.
Worse, Gitlab has no tools to make it easy to remove the user spam (and obviously no tools to prevent it from happening).
Just run a public-facing Gitlab CE instance for a few days. Search for one of the spam snippets you collect, and you'll find results for all the FOSS projects out there running their own Gitlab instances.
I've never seen any solutions offered by Gitlab for this, nor frankly any interest in the myriad bug reports about them addressing this at all.
Edit: typo
- ben0x539 6y agoYou'd think Wikimedia in particular has experience with the issue of spam bots polluting the site's search result.
- oconnor663 6y agoIs it possible to configure a robots.txt file to accomplish #2?
- clscott 6y agoNo, robots.txt is for well behaving bots like bing bot and google bot, not bots that will spam your forums (and Git repo apparently).
- marcinzm 6y agoBut if the issue is SEO bots then robots.txt would block the search engines thus meaning the spam content is of no importance (it's effectively private) and doesn't cause issues for the main sites SEO (nor help the spammers).
- jancsika 6y agoTo be honest, I'm not certain of the purpose of the spam. Some portion of it would end up in the search results, sure. But I don't know if there's some secondary benefit to, say, a casino showing a link coming from my site even if my site has a robots.txt saying that the address for that link isn't to be directly indexed. Is there such a benefit? If not then I'll just set up the robots.txt and observe whether that does indeed solve the problem. But I'd much prefer to just set up the permissions I know I want on my own running instance than spend time making inferences about the reasons bots are abusing my instance's inputs.
- detaro 6y agoIt doesn't stop the spam from being created, but it does stop the spam from ruining your site reputation in search results was the suggestion I guess.
- _ikke_ 6y agoBut the goal is to prevent spam in the first place. I don't think these bots will verify robots.txt to see if the spamming is effective. They just spam anything they can get their hands on.
- remram 6y agoThey probably don't have general code that spam any form, it's more likely that they have code specific to GitLab CE instances that knows to post snippets. If GitLab changes their default configuration so that those snippets are no longer indexed by Google, the spammer are likely to stop using that GitLab CE spamming script, after a while.
- tylersmith 6y agorobots.txt is nothing more than a request. It's essentially a "no dog on lawn" sign in the yard of a vacation home.
- deleted 6y ago[deleted]
- jefftk 6y agoCan you say more about how it's a problem if people can view things without logging in? Naively I would have seen that as a plus.
- marcinzm 6y ago1. Create user account 2. Create spam content (snippets, profile, etc.) 3. Get spam content search indexed 4. Profit
- dtech 6y agoa free user sign up is not going to prevent scraping...
- cortesoft 6y agoI think the idea is that if you can't view issues without logging in, then google won't index your issues (because it can't view them), so you won't get people spamming in order to get into google
- wolco2 6y agoIt also means 95% of people will stop casually viewing issues.
- noizejoy 6y agoOver the years I've frequently seen Google search results showing things that require login to the indexed site. Has that changed?
- marcinzm 6y agoI believe that requires the site giving google’s bot a logged in view rather than something google does themselves.
- wyldfire 6y ago
- 908B64B197 6y agoI honestly can't see why someone would go through the trouble of making sure their instance is correctly configured and available when there are solutions (like GitHub) that just work out of the box.
- paulryanrogers 6y agoControl. Github can change the behavior, pricing, availability, or security of their offerings at any time. If they get hacked then you could suffer. GH is also closed source. For many a self hosted solution is better, despite the costs.
- modoc 6y agoIt's not a lot of trouble. To be fair my GitLab instance isn't setup for large numbers of public contributors. I have a somewhat limited network connection and work on projects that often have large-ish codebases, building Docker images, etc.... and I do 90% of that on my home network (local servers, storage, etc....). So running GitLab locally allows me (and a few other folks) to get all those nice features without relying on the world facing internet connection and without having lots of delays moving large files up and down...
- 908B64B197 6y agoThat's the use-case I can understand (if someone has a large number of machines or a fast internal network behind a VPN). My comment was more aimed at public facing projects that are open and accept contributions from anyone.
- ognarb 6y agoStrange I never saw this behaviour on our Gitlab instance invent.kde.org.
- atomi 6y agoBecause it's Microsoft FUD.
- ran3824692 6y agoinvent.kde.org uses the nonfree google Recaptcha, that prevents it mostly. Not very nice for KDE to make people run nonfree software blob in their browser that gives up their freedom, gives up their privacy to google and trains Google's proprietary machine learning models.
- mattl 6y agoWhere does it use that?
- marcinzm 6y agoYou seem to be using a central login system (https://identity.kde.org/ https://identity.kde.org/) that requires going to a separate website to create an account which presumably is non-standard enough to throw off most bots.
- sytse 6y agoAt GitLab Inc. we have a Trust and Safety team https://about.gitlab.com/handbook/engineering/security/operations/abuse/ https://about.gitlab.com/handbook/engineering/security/opera... that prevents spam. So far that functionality has lived in separate repositories from the core codebase since few people needed it, the cycle time was quicker, and it is an advantage to not have the spammers see the code. If there is strong interest in collaborating on this I'm sure they will be happy to engage. I'll ask them how best to structure this.
- ran3824692 6y agoThere's been a gitlab bug for almost 3 years to stop relying on recaptcha, https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45684 https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45684 Debian, KDE and Gnome have never wanted to make their users run Google's nonfree javascript blob to contribute on their gitlab instance. There's been interest, Gitlab has done very little about it. Edit: other bugs about this can be found here https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46548 https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46548
- sytse 6y agoI think the code of this problem is that it is hard to identify if a user is a bot or a human. I've not seen any elegant free solutions to this.
- ran3824692 6y agoThat is not the core of the problem. Spammers are humans, and sometimes they will solve recaptchas in large quantities to get their spam through. Its about having a multipronged approach for administrators to stay ahead of them. For some examples of free solutions see https://www.mediawiki.org/wiki/Manual:Combating_spam https://www.mediawiki.org/wiki/Manual:Combating_spam. It's even possible to connect spamassassin to forms. Gitlab needs tools and automation that detects and rolls back spam, bans users, knobs to tune restrictions and rate limits based on how spammers are acting. Gitlab inc just hasn't seemed to care much to help people trying to use Gitlab and keep their software freedom.
- zeeZ 6y agoIn general settings, you can check "Public" under "Restricted visibility levels". According to the blurb, "selected levels cannot be used by non-admin users for groups, projects or snippets". Is that not what you want with #2?
- jancsika 6y agoIt's what I want for #2, but it has the unfortunate side-effect of restricting visibility for my main public repo. My #1 goal above is for people to be able to clone from my main repo without logging in.
- zeeZ 6y agoI see. And as soon as you make the repo public you end up with public issues again, unless you restrict them to project members... And a workaround of auto syncing just the code to a public repo where issues and stuff is disabled isn't available natively in CE.
- phikai 6y agoHi! I'm the PM at GitLab who works on Snippets, so thanks for providing this feedback. We do have Recaptcha support which can be configured - are you seeing these kinds of issues with that enabled/configured? One item that is on the roadmap that is coming and may be of interest is `Optional Admin Approval for local user sign up` - https://gitlab.com/groups/gitlab-org/-/epics/4491 https://gitlab.com/groups/gitlab-org/-/epics/4491. I'm not in the group working on that, but it does appear to be coming soon and would limit the ability of newly created accounts from doing anything until they're approved.
- protoduction 6y agoHi phikai, I built a privacy friendly alternative to ReCaptcha called FriendlyCaptcha [1], is there a possibility to see this integrated as a more user friendly alternative? Happy to chat (e-mail in profile) [1] https://friendlycaptcha.com/ https://friendlycaptcha.com/
- sytse 6y agoThat looks cool! Can someone create an issue to add support for this to GitLab? And maybe we can consider switching GitLab.com to this as well.
- robotmay 6y agoI'm personally interested in this too so I've created one :D https://gitlab.com/gitlab-org/gitlab/-/issues/273480 https://gitlab.com/gitlab-org/gitlab/-/issues/273480
- sytse 6y agoThanks for creating this! I think adding support for this in GitLab is a no-brainer. After that we can consider enabling it for GitLab.com
- barnabask 6y agoMan this needs more attention, cool project. I see you tried to submit to HN a couple of times and didn't get traction, that's too bad. Don't give up!
- abbe98 6y agoIs this mainly a concern for the Gitlab issue tracker? Wikimedia will continue to use Phabricator for issue tracking, Gitlab CE will only be used for CI and code review/hosting...
- ran3824692 6y agoNo. Spammers will create repos and user profiles and snippets and anything they can with spam in them.
- abbe98 6y agoI would imagine authentication being done through Wikimedia's existing LDAP or Mediawiki solution and I hope that features that already exists in Phabricator(such as snippets) will be disabled.
- gramakri 6y agoThe spam is infuriating (not GitLab's fault, of course). Atleast, on our instance at https://git.cloudron.io https://git.cloudron.io, we got massive snippet spam. After we disabled snippets, we got massive spam on the issue tracker (!). The way we "fixed" is by turning on mandatory 2FA for all users. As a general lesson, what we learnt is these are not bots. These are real humans working in some poor country manually creating accounts (always gmail accounts) and pasting all sorts of random text. Some of these people even setup 2FA and open issues with junk text, it's amazing. Unfortunately, GitLab from what I can tell cannot make issues read-only to non project members (i.e I only want project members to open issues, others can just read and watch issues). Currently, our forum spam (https://forum.cloudron.io https://forum.cloudron.io) is way more than GitLab spam. On the forum, we even have Captcha enabled (something we despise) but even that doesn't help when there are real humans at work.
- csdreamer7 6y agoWhy are they posting random text in Gitlab?
- riffic 6y agogetting that sweet sweet seo backlink juice
- gkop 6y agoIsn’t that why we add rel=nofollow to low friction user submitted links on our platforms?
- ivank 6y agoGoogle changed the interpretation of those a year ago. https://webmasters.googleblog.com/2019/09/evolving-nofollow-new-ways-to-identify.html https://webmasters.googleblog.com/2019/09/evolving-nofollow-...
- 6y ago
- wdb 6y agoPersonally, I also think, GitHub's Checks API, and it Github bots. Gives a much better experience compared to GitLab. On a daily basis I am confused about how diff's are being rendered in GitLab merge requests, as it has a weird way to render ${blah} in strings in lines. Also when you want to check if an issue exists in GitLab own repository, you always end up in some jungle of redirected tickets. Just now I got redirected to three different tickets , as they switch projects or something. Really annoying.
- boleary-gl 6y agoCan you share an example or screenshot of what you mean around the rendering issue in merge requests? As for the issue redirects - it does stink. It's an artifact of our move to a single code base for CE and EE [1]. A lot of issues have long-standing SEO so the "old" issue often comes up in a Google search. [1] https://about.gitlab.com/blog/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/ https://about.gitlab.com/blog/2019/08/23/a-single-codebase-f...
- wdb 6y agoYes, for example, in today's pull request, I added a new file and now in my Typescript file it renders things like this, which I find confusing: https://imgur.com/a/C82hcMK https://imgur.com/a/C82hcMK
- tyteen4a03 6y agoI wonder how effective QuestyCaptcha would be on GitLab.
- bawolff 6y agoGenerally speaking questycaptcha works great if you fly under the radar enough that nobody is putting any effort in, and its just automated bots. It tends to fall apart the more high profile you are.
- eznzt 6y agoCan't you just put the gitlab instance behind an nginx proxy to achieve this? Like, if you are requesting ^/user/$, check for a cookie; if invalid, return 403
- kodah 6y agoI would encourage folks to look at Gitea.io. I run that on Kubernetes alongside Drone and it basically replicates all the most important parts of GitHub.
- trynewideas 6y agoI'll be curious to see whether they even use GitLab user auth. For Gerrit (and Phabricator), Wikimedia already requires contributors to have a dev account on Wikimedia's LDAP system: https://wikitech.wikimedia.org/wiki/Help:Create_a_Wikimedia_developer_account https://wikitech.wikimedia.org/wiki/Help:Create_a_Wikimedia_...
- mgbmtl 6y agoIf you can, place your Gitlab CE instance behind an LDAP server. Have another site handle signups. (admitedly, setting up something with LDAP is often a massive pain. I duct-tape around it by using LdapJS on top of a CMS) I've had a handful of projects where human spammers will bother to create an account and jump through the loops, but in the 2-3 years of running a Gitlab instance, which has 1300 users, I only had 2-3 incidents (we keep an eye on recent projects, snippets, etc).
- chillfox 6y agoThe GitLab LDAP config is pretty easy.