4 ms·

Do they explain the recent change to require login for code search?

marginalia_nu 3y ago
Very likely because public search APIs are a huge target for bot activity. For reference, about 99.5% of the search queries that goes toward my search engine is demonstrably bot traffic (excluding the traffic that goes to the free and public API endpoint I offer). A capable search functionality isn't exactly cheap to run, so requiring a login is the obvious way to be able to rate limit searches, which is otherwise impossible against a botnet.
tedunangst 3y ago
Gotta pump those MAUs.
- Paul-Craft 3y ago
  Bingo. Some product manager is probably getting a fat bonus for that.
- blinded 3y ago
  Its also a perf / ddos thing. If anyone from anywhere can run expensive searches it could degrade performance. A little log in won't hurt.
  - tedunangst 3y ago
    Now I just clone the repo and run grep myself. Anything I can do to help lighten the load.
    - blinded 3y ago
      Same, but silver searcher or fuzzy search ftw :)
      - dumbo-octopus 3y ago
        FWIW, github.dev has a search feature that automates this flow using the fastest available subroutines for each step, all you do is a repo full text search with the same UI as normal VS Code. In my testing it searches about as fast as AG, and much faster than Grep. RipGrep beats it in a direct search time comparison, but the overhead of cloning the files is dominant in large repos. GitHub.dev searches via a .tar.gz archive stored in IndexedDB, which is much faster than cloning even at depth 1 and building up the files on disk (~50% speedup on my testing in very large repos, ~90k files).
        blinded 3y ago
        Thanks for the pointer.
ramraj07 3y ago
I was about to type “why would the engineers who upgraded search have a say on the login behavior” then I answered it myself by realizing a far more powerful search feature means they spend more compute per query. Thus it actually makes sense they use login requirement to limit the search to users who are adding actual value to their bottom line. Again you could disagree but it is a decision that can be rooted in sound logic.
- throwaway7679 3y ago
  The same sound logic could have been used to put the old search being a login. That also consumed some amount of compute per query after all. Actually there's no reason to serve logged out users with anything except a login page. Why spend any resources on users who aren't adding value? Or, maybe this is all value-destroying nonsense that only makes the service worse. In any case, the performance of the new search system is supposed to be better, not worse. [1] [1] https://github.blog/2023-05-08-github-code-search-is-generally-available/ https://github.blog/2023-05-08-github-code-search-is-general...
  - n_plus_1_acc 3y ago
    The rest of github (viewing code and issues, downloading releases and zips, etc) is easily cacheable, minimizing the cost.
  - eru 3y ago
    > Why spend any resources on users who aren't adding value? What makes you think that logged out users aren't adding any value? The behaviour of github here is consistent with them assuming that logged out users provide a small but non-zero amount of value for them. The link you provided seems to suggest that the code search is supposed to be faster for users, but I can't see anything in it that suggests that it's less taxing on the backend?
    - throwaway7679 3y ago
      I may not have made it clear enough, but my first line was meant to be absurd and obviously wrong. I think a service should focus on how it provides value to users, not the other way around. People use it because of what it provides to them, not because they desire to support the company's bottom line. GitHub built a remarkably valuable product, which attracted 100M users and $1B in revenue. It has tons of public features that never required login. They could have put search, or any number of other things, behind a login at any point in the last 15 year. But they didn't, because that's not actually a good thing to do. Doing that can only ever make the feature less usable. What drives them to do it now is unclear. Perhaps just wanting to juice their login numbers or whatever. But, performing little bait-and-switches on random features isn't going to make people use them more or spend more money. It's going to make them seek hedges against the destruction of the features they enjoy. As for my link, the search result speed for users is directly related to cost per query, unless you believe that all of this is fake and the new search system is secretly just a bunch of expensive new hardware.
dontupvoteme 3y ago
Cynic in means says it's two main things: 1. Effectively serves as a walled garden for copilot to make sure that competitors can't use their data (your code) in coding assistants, knowledge discovery, etc. 2. Ensure that competitors who are using github are maximally mineable so they can easily discover and implement their solutions to eat their lunch (much like OpenAI's strategy here) This is covered up by the excuse of performance, but it's fundamentally no different than the loginwalls we saw go up around twitter and reddit ... except this time done by a big industry player who LARPs as part of the open source community but is positioned to cannibalize it from within.
- maple3142 3y ago
  Shouldn't the competitors just clone the repository locally and train their model on it instead of relying search api which probably cost more computation resources?
  - dontupvoteme 3y ago
    You could, yes, and probably many are doing this, but you now have to git pull on all of those if you want to, say, know which LLM libraries are currently trending, or how quickly PopularLib v0.2 is being used in codebases related to Y, etc. IMO It's much less about the legacy code (there exist already terabyte size datasets that take in a lot of things on github) and MUCH more about how up-to-date your LLM/AI is with new repos, "best practices" (or most common practices), etc. Plus you often get "LLM Code poisoning" from older training data as it attempts to use functionality which has experienced breaking changes versus the current stable release. Current is King. Also there's the whole goldmine of github discussions, issues, etc that a repo just.. doesn't have. Right now you can still easily index those (though iirc they sometimes ban datacenter IPs), but they may also fall victim to the loginwall.

Very likely because public search APIs are a huge target for bot activity. For reference, about 99.5% of the search queries that goes toward my search engine is demonstrably bot traffic (excluding the traffic that goes to the free and public API endpoint I offer). A capable search functionality isn't exactly cheap to run, so requiring a login is the obvious way to be able to rate limit searches, which is otherwise impossible against a botnet.

tedunangst 3y ago

Gotta pump those MAUs.

Paul-Craft 3y ago
Bingo. Some product manager is probably getting a fat bonus for that.
blinded 3y ago
Its also a perf / ddos thing. If anyone from anywhere can run expensive searches it could degrade performance. A little log in won't hurt.
- tedunangst 3y ago
  Now I just clone the repo and run grep myself. Anything I can do to help lighten the load.
  - blinded 3y ago
    Same, but silver searcher or fuzzy search ftw :)
    - dumbo-octopus 3y ago
      FWIW, github.dev has a search feature that automates this flow using the fastest available subroutines for each step, all you do is a repo full text search with the same UI as normal VS Code. In my testing it searches about as fast as AG, and much faster than Grep. RipGrep beats it in a direct search time comparison, but the overhead of cloning the files is dominant in large repos. GitHub.dev searches via a .tar.gz archive stored in IndexedDB, which is much faster than cloning even at depth 1 and building up the files on disk (~50% speedup on my testing in very large repos, ~90k files).
      - blinded 3y ago
        Thanks for the pointer.

ramraj07 3y ago

I was about to type “why would the engineers who upgraded search have a say on the login behavior” then I answered it myself by realizing a far more powerful search feature means they spend more compute per query. Thus it actually makes sense they use login requirement to limit the search to users who are adding actual value to their bottom line. Again you could disagree but it is a decision that can be rooted in sound logic.

throwaway7679 3y ago
The same sound logic could have been used to put the old search being a login. That also consumed some amount of compute per query after all. Actually there's no reason to serve logged out users with anything except a login page. Why spend any resources on users who aren't adding value? Or, maybe this is all value-destroying nonsense that only makes the service worse. In any case, the performance of the new search system is supposed to be better, not worse. [1] [1] https://github.blog/2023-05-08-github-code-search-is-generally-available/ https://github.blog/2023-05-08-github-code-search-is-general...
- n_plus_1_acc 3y ago
  The rest of github (viewing code and issues, downloading releases and zips, etc) is easily cacheable, minimizing the cost.
- eru 3y ago
  > Why spend any resources on users who aren't adding value? What makes you think that logged out users aren't adding any value? The behaviour of github here is consistent with them assuming that logged out users provide a small but non-zero amount of value for them. The link you provided seems to suggest that the code search is supposed to be faster for users, but I can't see anything in it that suggests that it's less taxing on the backend?
  - throwaway7679 3y ago
    I may not have made it clear enough, but my first line was meant to be absurd and obviously wrong. I think a service should focus on how it provides value to users, not the other way around. People use it because of what it provides to them, not because they desire to support the company's bottom line. GitHub built a remarkably valuable product, which attracted 100M users and $1B in revenue. It has tons of public features that never required login. They could have put search, or any number of other things, behind a login at any point in the last 15 year. But they didn't, because that's not actually a good thing to do. Doing that can only ever make the feature less usable. What drives them to do it now is unclear. Perhaps just wanting to juice their login numbers or whatever. But, performing little bait-and-switches on random features isn't going to make people use them more or spend more money. It's going to make them seek hedges against the destruction of the features they enjoy. As for my link, the search result speed for users is directly related to cost per query, unless you believe that all of this is fake and the new search system is secretly just a bunch of expensive new hardware.

dontupvoteme 3y ago

Cynic in means says it's two main things: 1. Effectively serves as a walled garden for copilot to make sure that competitors can't use their data (your code) in coding assistants, knowledge discovery, etc. 2. Ensure that competitors who are using github are maximally mineable so they can easily discover and implement their solutions to eat their lunch (much like OpenAI's strategy here) This is covered up by the excuse of performance, but it's fundamentally no different than the loginwalls we saw go up around twitter and reddit ... except this time done by a big industry player who LARPs as part of the open source community but is positioned to cannibalize it from within.

maple3142 3y ago
Shouldn't the competitors just clone the repository locally and train their model on it instead of relying search api which probably cost more computation resources?
- dontupvoteme 3y ago
  You could, yes, and probably many are doing this, but you now have to git pull on all of those if you want to, say, know which LLM libraries are currently trending, or how quickly PopularLib v0.2 is being used in codebases related to Y, etc. IMO It's much less about the legacy code (there exist already terabyte size datasets that take in a lot of things on github) and MUCH more about how up-to-date your LLM/AI is with new repos, "best practices" (or most common practices), etc. Plus you often get "LLM Code poisoning" from older training data as it attempts to use functionality which has experienced breaking changes versus the current stable release. Current is King. Also there's the whole goldmine of github discussions, issues, etc that a repo just.. doesn't have. Right now you can still easily index those (though iirc they sometimes ban datacenter IPs), but they may also fall victim to the loginwall.