7 ms·
I'm ready to believe the most charitable explanation for this: the new code search (which I have found to be exceptionally good) has a lot more going on under t
by simonw 3y ago
I'm ready to believe the most charitable explanation for this: the new code search (which I have found to be exceptionally good) has a lot more going on under the hood than a normal search engine, which makes it a lot more resource-intensive - limiting it to signed-in accounts saves a huge amount of server resources that would otherwise be used to serve crawlers.
My guess is that the trade-off here is genuinely a question of if they spend 3-4x (I'm guessing, but I think this is likely a low-ball estimate) on their search infrastructure, v.s. having people angry at them for requiring login to use the feature.
I actually built my own code search tool for use with GitHub repos, but I've mostly stopped using that because the new GitHub code search is so useful by default: https://simonwillison.net/2020/Nov/28/datasette-ripgrep/ https://simonwillison.net/2020/Nov/28/datasette-ripgrep/
- Fatnino 3y agoThey were already rate limiting unauthenticated users. So are expected to believe that a hostile user who has zillion of ip addresses (to get around the rate limits) won't also be able to make a handful of accounts?
- ldoughty 3y agoMight also help with thwarting easy access to tokens/keys... especially for those that are not currently in their filtered list. While a bad actor can certainly scrape/clone the same data... Given repos with thousands of files... Scanning by scraping vs searching vs cloning.... Search is certainly the cheapest for the attacker while the most expensive for GitHub
- Thorrez 3y agoThat could be mostly solved by requiring login only for searches across all repos. Searches in a single repo could be anonymous. In fact, that's exactly how it was until in June GitHub started requiring login for searches even in just a single repo.
- 1980phipsi 3y agoIf the new code search is more resource intensive, then why not have a slimmed down, less resource intensive version for when users aren't logged in?
- supriyo-biswas 3y agoThis may involve running two different sets of (for example, ElasticSearch) clusters, which doesn’t justify itself from a business perspective, especially given that non-logged in users can still clone code and use local tooling for search.
- isodev 3y agoThat's interesting but the technical reasons are really besides the point. The end result is that it's no longer possible to participate in "GitHub hosted open source" without actually being a member of GitHub.
- Justsignedup 3y agoright, but being a member isn't a pay-wall. This feels more like an academic argument than a practical one.
- isodev 3y agoBecoming a member means sharing your personal information with GitHub and abiding by their terms of service. It also implies that they will then track and profile locations where you sign in, technologies you use to interact with their services etc. That's a lot to give for "I want to browse a piece of code someone made for free".
- underdeserver 3y agoAnd it's what they get in return for hosting and providing indexing and a git server. I'm not saying it's a deal that makes sense for everyone. But it's certainly a barter trade and is well within their prerogative to offer.
- johnfn 3y agoThe only bit of "personal information" you need to provide is an email, and you're free to use a burner. > That's a lot to give for "I want to browse a piece of code someone made for free". You can still do this. All repositories are accessible without signing in.
- isodev 3y agoFacebook recently estimated that knowing about a user costs close to €11/month. That's a lot of money for hosting and indexing git repositories... I guess we need an alternative.
- 3y ago
- Thorrez 3y agoCheck out the Chromium code search. It has regex support, and doesn't require login. Although it is smaller than all of GitHub. Disclosure: I work at Google. https://source.chromium.org/chromium https://source.chromium.org/chromium
- ajsnigrutin 3y agoThe most realistic one is, that people will get fed up and register an account, and github can brag about how many new users they brought in. The other side is, that existing users will get pissed at github, because they can't even search anymore without logging in, and sometimes that's a pain (not their pc, public pc, incognito tab, the time needed to do the 2fa, etc.). Github can still keep the cheap, fast basic search for users not logged in, but they didn't.
- rudolph9 3y agoI’m not saying you’re wrong, if anything it makes their decision even more ridiculous, but GitHub has completely saturated the market. Other than pissing people off what do they really benefit from this?
- usea 3y ago> GitHub has completely saturated the market That's an assumption which github may not agree with.
- manvillej 3y agothe most realistic one is to prevent other scrapers from building their own tools on top of github code search. by putting login, they can throttle or block it entirely.
- txdv 3y agoo wow, the KPI of user acquisition should be through the roof with login required
- simonw 3y agoKeeping the old search running for logged out users has substantial costs too: 1: You have to maintain two full independent search and indexing implementations 2: User confusion. Why are the search results different depending on if I'm logged in or logged out?
- spoiler 3y ago
- SnowflakeOnIce 3y agoAside from performance concerns, there have been many occurrences of bad actors using code search to find hardcoded credentials, and then using that to gain unintended access to additional systems. I suspect the change has more to do with security concerns (and having an audit trail) than performance.
- prepend 3y agoI’d like to see these many occurrences. I think the solution to hard coded credentials isn’t making search harder (security through obscurity == bad) but the other things GitHub is doing to detect and mitigate hard coded credentials. Search is still possible using google and other methods, so any theoretical gains from forcing login are dangerous to count on as vulnerable projects are still vulnerable. If anything, I think the solution to projects with hard coded credentials are to make the credentials easier to find and exploit so they fixed more quickly after being created. The most dangerous are ones they are hard to find so someone uses them for long periods of time without detection.
- SnowflakeOnIce 3y ago> I’d like to see these many occurrences. Some high-profile cases where credentials were leaked on public GitHub: Uber in 2014 and 2021 [1, 2] and Twitch in 2021. > Search is still possible using google and other methods Yes, you can search with Google or other sources. But the thing is, pretty much only GitHub has easy access to all the code present there, readily available to search within minutes of pushing. You could try mirroring GitHub yourself, but you'd need enormous disk space and bandwidth, and would quickly hit rate limits. You also wouldn't be able to do fast full regex search like you can with GitHub's search, as you don't have their search infrastructure. Hackers are aware of this and do make use of GitHub's search to identify possible leaked credentials. I recently experimented with uploading a dummy AWS credential pair to a public Git repo, and saw that numerous IPs started trying those credentials less than 5 minutes after I pushed. > any theoretical gains from forcing login are dangerous to count on as vulnerable projects are still vulnerable Indeed, requiring login is not going to _solve_ the problem. But it could lessen the impact of leaked credentials at large scale, by making it more difficult for automated systems to harvest them. Perhaps more significantly, requiring login could give better audit trails in incidence response situations, as the logs would indicate which accounts were searching for secrets. > I think the solution to projects with hard coded credentials are to make the credentials easier to find and exploit so they fixed more quickly after being created Yes, this can help! There are several other companies that specialize in secret detection. I've also written Nosey Parker, a fast regex-based detection tool that has higher-precision rules than similar tools [4]. GitHub also has its own offering in Advanced Security to address this problem. [1] https://www.reuters.com/article/uk-uber-tech-lyft-hacking-exclusive/exclusive-in-lawsuit-over-hacking-uber-probes-ip-address-assigned-to-lyft-exec-sources-idUKKCN0S20D020151008/ https://www.reuters.com/article/uk-uber-tech-lyft-hacking-ex... [2] https://www.securonix.com/blog/securonix-threat-research-uber-hack-software-code-repository-vcs-leaked-credential-usage-detection/ https://www.securonix.com/blog/securonix-threat-research-ube... [3] https://news.ycombinator.com/item?id=28770590 https://news.ycombinator.com/item?id=28770590 [4] https://github.com/praetorian-inc/noseyparker https://github.com/praetorian-inc/noseyparker
- babyshake 3y agoIt's likely to avoid crawling, especially to train competitors to GitHub Copilot or other LLMs.
- supriyo-biswas 3y agoWhy can’t you train on the git repository itself? Clones are not behind a login.
- rudolph9 3y agoThis seems like the most probable reason of any I’ve read so far.
- benjaminwootton 3y agoI see lots of reasons being proposed, but my guess is that they want or need to avoid content being scraped by AI related companies and researchers.
- prepend 3y agoAs someone who contributes a decent amount of OSS code on GitHub, I want my code scraped by AI related companies. If GitHub starts making opinionated decisions about how to “protect” my code, then I won’t like that.
- Tommstein 3y ago> If GitHub starts making opinionated decisions about how to “protect” my code . . . They already did. > . . . then I won’t like that. There is no universe in the multiverse where Microsoft or one of its subsidiaries gives a single flying fuck. Something about those who forget history being doomed to repeat it . . . .
- Mistletoe 3y agoI just assume this to be the case for all future decisions like this. Surely the Reddit API changes were at least partly from this? Reddit is one of the richest sources of (pretty) authentic human interaction that humanity has ever made. Every AI company and startup was going to mercilessly mine this forever. I'm sure Reddit would much rather sell (our) interactions instead.
- codedokode 3y agoProbably code search was misused by bots to search secrets.
- rudolph9 3y agoYeah but that’s been the case since search existed. It seems unlikely that somehow now that’s become an urgent problem or requiring login will fix anything.
- zxt_tzx 3y agois there any reason why can't GitHub put their code search behind something like Cloudflare Turnstile? if HaveIBeenPwned can do this, why not GitHub? https://www.troyhunt.com/fighting-api-bots-with-cloudflares-invisible-turnstile/ https://www.troyhunt.com/fighting-api-bots-with-cloudflares-...
- Retr0id 3y agoI suppose maintaining two systems adds overhead too, but they could've let you use "basic search" without login, with the improvements login-gated.
- __MatrixMan__ 3y agoYou'd think, with all that work going on under the hood, that they'd be able to support searches for repos by multiple paths: > Give me repos with project.toml and a flake.nix at the root The interface supports it, but searches come back empty. I end up leaving scripts running while I sleep which walk search results and narrow them by just checking to see if the files are there. If you're out there Microsoft, please fix your search so I can stop being a bad citizen.
- citruscomputing 3y agoThe new search is still so useless that I just maintain a full local clone of our org's code. Someone was requesting perms to run a script to search across the org today -- it would've taken 10+ minutes to run while respecting undocumented secondary rate limits (which there's no way to check via headers, by the way). I rewrote it to use ripgrep locally, and it's down to 20 seconds.
- zxt_tzx 3y agoWhy can't the crawler problem be stopped by something like CAPTCHA or Cloudflare Turnstile?
- iepathos 3y agoAs far as their code search under the hood it's just Elasticsearch, nothing special. Github's explanation sounds like mostly bullshit. Forced authentication of public endpoints is not an appropriate solution for the issue they're claiming. People building bots who actually care about this endpoint can just have their bots login with free accounts. This doesn't actually prevent load on their servers in a meaningful way.
- slaymaker1907 3y agoWhen they’re logged in, they are much easier to throttle or even ban compared to IP hopping to dodge IP throttling/bans.
- 867-5309 3y ago>bots login with free accounts those accounts still need to be created, which is hard to automate also, now GH can just ban more easily detectable accounts that scrape, vs anonymous visitors
- WorldMaker 3y agoGitHub's code search is no longer using Elasticsearch, it is using an entirely in house (Rust-based) search engine nicknamed Blackbird: https://github.blog/2023-02-06-the-technology-behind-githubs-new-code-search/ https://github.blog/2023-02-06-the-technology-behind-githubs...
- renegat0x0 3y agoWe have been running Internet without requiring user accounts. Search was possible. Has something changed? Has github been having real problems with bots? No?
- duped 3y agoIf that's the case, they should really fall back on a grep of the codebase because whatever is "under the hood" sucks. It routinely returns no results for exact string matches in code. These days I clone the repo and use ripgrep instead of codesearch because it's worse than useless, it's wrong.
- K0IN 3y agowhy cant they just use the old approach for anyone and the good stuff for loggedin users tho.
- superkuh 3y agoIf that's true then they should still provide the old non-heavy search for non-logged in users.
- executesorder66 3y agoIf they are doing this intentionally, then fuck Microsoft. Bunch of assholes as always. If this is mere incompetence, then again I'm not surprised that they fumbled such a simple task. Grep is literally 50 years old. They don't even have to write any code to do text search, they could use free software if they love it so much. This isn't some amateur startup with a CRUD app and 3 devs. This is one of the largest and wealthiest tech companies in the world, and they can't even get text search to work without a login? What a bunch of losers. If there is a 3rd explanation, I'd love to hear it.
- ivank 3y agohttps://github.blog/2023-02-06-the-technology-behind-githubs-new-code-search/ https://github.blog/2023-02-06-the-technology-behind-githubs...
- executesorder66 3y agoWhere in that post does it say that the user not being logged in is making it difficult/impossible for them to return search results?
- Tommstein 3y ago> If they are doing this intentionally, then fuck Microsoft. Bunch of assholes as always. They explicitly said that they "had" to require being logged in, so, yeah: https://github.com/orgs/community/discussions/77046#discussioncomment-7681591 https://github.com/orgs/community/discussions/77046#discussi... . Although they do assure us, however, that they're totally sorry for the inconvenience.
- executesorder66 3y agoYou are correct, but I was just covering all the possibilities since I never trust what they say either.