7 ms·
Lessons from building GitHub code search [video]
- IshKebab 3y agoObligatory https://grep.app/ https://grep.app/
- sccxy 3y agoI wonder if he got a job at Github or not https://news.ycombinator.com/item?id=22396824 https://news.ycombinator.com/item?id=22396824
- panzerboiler 3y agoI don't know if it is related to the changes to the code search (I use github exclusively as guest), but in the last year github got slower and heavier to the point that even scrolling a page showing a file of 200 lines is very painful. One with more than 2000 lines crashes the tab. I don't have the most powerful machines at my disposal, but what I have should be more than powerful enough to browse a repository. My intuition points at the syntax highligther and the file browser/go-to-symbol as possible causes. Anybody knows of some magic setting I can use to have a better experience? Right now I am forced to avoid using it as much as possible, and clone the repositories to browse them locally.
- jamietanna 3y agoThere was also a recent rollout of React, of which I've seen a number of complaints about performance related to the timing. Personally, my Firefox Nightly on Android will regularly crash (along with the whole System UI) when opening / trying to type into a GitHub page
- selckin 3y agoIn the actions overview, just the spinner icon (svg w/ css animation) takes 25-40% cpu and 10% gpu in my chrome. Enough to keep my fingers very warm on the laptop
- 3np 3y agoI'm looking into options for syncing issues and PRs for local browsing as well due to the frontend performance regressions in GitHub. There's a hanging draft for it in Gitea (inb4 yes also Forgejo). https://github.com/go-gitea/gitea/pull/20311 https://github.com/go-gitea/gitea/pull/20311 Maybe this is the push I need to actually start using taskwarrior... https://github.com/GothenburgBitFactory/bugwarrior https://github.com/GothenburgBitFactory/bugwarrior
- inferiorhuman 3y agoThe magic setting seems to be don't use Github, unfortunately. There were a couple GH discussion threads about the dreadful UI rollout, but someone complained one too many times about (code) search returning the wrong results so they've since been locked. For me it's the issue search that seems to consistently return the wrong results, and the text input widgets that are glitchy as hell in Firefox. Performance-wise it's worth keeping in mind that Github is now rendering all the text client side. If you go through the discussion threads you can see where the GH devs didn't know how to account for multibyte codepoints and how that broke their non-native rendering with things like emoji and Chinese characters.
- naikrovek 3y agoall these posts about how it's so obvious that search doesn't work, but no examples that demonstrate it from anyone.
- inferiorhuman 3y agoYou're certainly welcome to go find the discussion threads on Github where people were posting use cases and screenshots. I already found my solution: don't use Github.
- Kuinox 3y agoOh yeah, they use react now, and have to use smart tricks so you can actually select/search highlighted code. The code is also probably added as you scroll too, using a virtual list.
- FridgeSeal 3y agoGitHub has definitely gotten worse. Stale data, PR’s and such don’t update properly, page loads are glacial, half the time it’s doing some loading bar of its own-no doubt as part of some trying-to-be-clever SPA thing, and it’s awful.
- infotogivenm 3y agoI’ve noticed the code reader is worthless on even slightly out of date browsers now, and even on newer browsers it tends to choke and stutter on large files. Sad :( it used to be the best
- bkircher 3y agoExcellent talk! Sad seeing Strange Loop go…
- easton 3y agoDoes anyone know of a good (preferably US based) replacement? It had a unique vibe.
- jakub_g 3y agoSince we talk GH code search, a frequent issue raised on community forums is that some repos stopped being indexed, and yield 0 results. (Happened also to repos in my bigcorp). Has there been a systemic fix for that issue, other than asking GH support teams to reindex the repo?
- welder 3y agoThey list things like "Fast" and "Regular Expressions", but what about correct results? I often use GitHub web search and it never finds terms which I know for a fact exist. After it fails to return results, I have to instead manually navigate to the file containing the search term and yes it's there.
- lettergram 3y agoYup I was about to write something about how I quite literally clone & grep code bases after GitHub search fails probably half the time.
- jve 3y agoFor repository wide search 1. Press dot "." 2. CTRL + SHIFT + F 3. Enjoy results powered by rg (ripgrep)
- welder 3y agoI just use sourcegraph it's faster than cloning usually.
- prepend 3y agoSource graph is nice but completely overpriced. I have 500 people who want to search, maybe 1000. I’m not paying $100/user/year forever when I can just clone all thousand repos onto a drive and let people grep it. Grep is not as good, but the cost is like $500/year for infinite users vs $50m/year with a marginal cost for new users. Also, my general comment on pricing applies here. Copilot is $100/year. So I’m supposed to think being able to search my code is the same price as being able to write the code. Also, they are kind of in trouble because this will exactly be replaced once I just fork over $500/user/year for superchatgpt and it will definitely cover code search.
- quartz 3y agoThe new UX does a really bad job of helping you realize when you're logged out and search results are being suppressed. It very much implies "zero results" when in fact it means to say "zero results, except in code, where there might be results, but you need to log in first to find out".
- AlexErrant 3y agoI also enjoyed the Treesitter talk from 5 years ago by Max Brunsfeld https://www.youtube.com/watch?v=Jes3bD6P0To https://www.youtube.com/watch?v=Jes3bD6P0To I'm currently building a query language whose grammar is very much inspired by Github's search syntax. I'm using Lezer, which is a GLR like Treesitter, so this talk learned me some parser generators (I've no formal CS education). Here's my grammar, a playground, and an example search query if anyone wants to play with it https://github.com/AlexErrant/Pentive/blob/main/app/src/query.grammar https://github.com/AlexErrant/Pentive/blob/main/app/src/quer... https://littletools.app/lezer https://littletools.app/lezer -(a) spider-man -a b -c -"(quote\"d) str" OR "l o l" OR a b c ((a "c") b) tag:what -deck:"x y" I just converted the syntax tree to an abstract syntax tree so I can do De Morgan's law transformations. Literally the only time in my professional life where I feel like I'm solving a leetcode-style problem.
- 100k 3y agoTree-sitter is very cool! We use it to power semantic analysis for GitHub code search.
- msephton 3y agoDo they explain the recent change to require login for code search?
- marginalia_nu 3y agoVery likely because public search APIs are a huge target for bot activity. For reference, about 99.5% of the search queries that goes toward my search engine is demonstrably bot traffic (excluding the traffic that goes to the free and public API endpoint I offer). A capable search functionality isn't exactly cheap to run, so requiring a login is the obvious way to be able to rate limit searches, which is otherwise impossible against a botnet.
- tedunangst 3y agoGotta pump those MAUs.
- Paul-Craft 3y agoBingo. Some product manager is probably getting a fat bonus for that.
- blinded 3y agoIts also a perf / ddos thing. If anyone from anywhere can run expensive searches it could degrade performance. A little log in won't hurt.
- tedunangst 3y agoNow I just clone the repo and run grep myself. Anything I can do to help lighten the load.
- blinded 3y agoSame, but silver searcher or fuzzy search ftw :)
- 3y ago
- ccleve 3y ago[dead]
- nliang86 3y agoHere's a detailed text outline with key frames for those who don't have time to watch the 36 minute video: https://www.videogist.co/videos/lessons-from-building-github-code-search-by-luke-francl-strange-loop-2023-1823 https://www.videogist.co/videos/lessons-from-building-github...
- ngshiheng 3y agothis is awesome. i think it'd be super cool if it can read/summarize the comments too
- paradox460 3y agoNot sure if it can, but Kagi can: • GitHub's previous code search was slow, limited, and did not support searching Forks due to indexing challenges. A new system called Blackbird was built from scratch to address these issues. • Indexing code poses unique challenges compared to natural language documents, such as handling file changes in version control systems and deduplicating shared code across repositories. • The talk discussed techniques used in Blackbird like trigram tokenization, delta compression, caching, and dynamic shard assignment to improve indexing speed and efficiency at scale. • Architectural decisions like separating indexing from querying and using message queues helped Blackbird scale independently without competing for resources. • Data structures like geometric XOR filters were developed to efficiently estimate differences between codebases and enable features like delta compression. • Iteration speed was improved by making the system easier to change through frequent index version increments without migrations. • Resource usage was optimized through techniques such as document deduplication, caching, and compaction to reduce indexing costs. • Blackbird's design allowed it to efficiently support over 100 million code repositories while the previous system struggled at millions. • Building custom solutions from scratch can be worthwhile when leveraging data structure to outperform generic tools for a domain. • Anticipating and addressing scaling challenges at each magnitude is important to ensure a system remains performant as it grows over time.
- jve 3y agoThose don't look like video comments. Which are: fantastic/well done/wonderful/great talk etc.
- bakugo 3y agoDo they mention why the new search still doesn't provide sorting options? I found them very useful in the previous version.
- nayuki 3y agoSeconded. I liked the previous GitHub code search where I sort by recency to pick up new mentions of keywords. Now the new search is useless to me.
- 100k 3y ago(I'm the speaker.) This is frequently requested, and we've tried to answer it in GitHub's feedback forums. The reason it's not implemented is it's quite complicated and there are other things we'd rather work on (sorting by recently was mostly used by scrapers so it's not high yield for us, unfortunately). It's complicated due to some consequences of things I discuss in the talk. First of all, the old system didn't really implement sorting for "recently changed" files. It implemented sorting for "recently indexed" documents. This was a decent proxy when we only rebuilt the index every two years. But as I discuss in the talk, we rebuild the index all the time now, so it would be pretty worthless as a sorting option. Another reason is due to the content deduplication I also discussed in the talk. When a blob is shared between repositories, what time do you store? Finally, it's complicated because of how Git works. If we did implement this, we'd like it to reflect when the path changed in the branch, which is what people mean by "when did this file last change". But Git doesn't have an efficient way to retrieve that information, so we'd have to build a system to keep track of it. In short, it's hard to do right and easy to abuse. :-(
- NovaX 3y agoI found sorting by recency to be helpful to see what mistakes users would make. There is not much feedback in open-source until after an api is released, at which point it is usually too late to fix problems. Seeing what users did, what frustrated them, and providing feedback helped make a better library. That's typically what internal teams get to do on a large corpus, like google3, and github is equally excellent for these insights. An alert on recently indexed content that matches keyword subscriptions, ala google alerts, would be an excellent alternative for that use-case.
- 100k 3y agoOh hey, that's my talk! Thanks for submitting it. It was a huge honor to present my team's work at Strange Loop.
- skilled 3y agoHi Luke, Could you please give an update on whether or not GitHub is still considering adding “sort by recent” to search? —— E: I just saw you answered that already. It’s a dearly missed feature.
- 100k 3y agoPlease see my other comment about why this is difficult: https://news.ycombinator.com/item?id=38638214 https://news.ycombinator.com/item?id=38638214
- deleted 3y ago[deleted]
- rmccue 3y agoI’d just like to say: thank you! GitHub’s previous search was not great, and when the new version launched it was a massive leap forward, where it’s now part of my daily workflow. Before this I thought good search at this scale might just be an intractable problem. :)
- mickael-kerjean 3y agoMeh before Microsoft acquisition, you could get an API key for any service you want by just making a search on github, not sure how many people knew about it, it was probably a dirty secret but I used to crawl tons of stuff by just rotating API keys found from github, none of that is possible anymore. On the plus side I don't count how many reports I've done to companies who did leak not only their username/password but also all the cool proxy you could use to go inside their network. The weirdest one of them was a guy working in security at thales which is supposed to handle security sensitive stuff for governments leaking all that information as he was working on a side project involving poker during business hours ...
- lopkeny12ko 3y agoOne of the (among many) reasons I stopped using GH code search is that the default ranking algorithm is extremely poor. Search relevance is awful, especially when it comes to surfacing forks. 90+% of the time I'm executing a code search, I'm looking for example uses of a library API in open source code. But most of the time, Github code search just surfaces pages upon pages of the same exact code snippet across the origin repository and hundreds of forks. For example, one search I executed earlier today: https://github.com/search?q=load%28%22%40rules_foreign_cc%2F%2Fforeign_cc%3Adefs.bzl%22%2C+%22ninja%22%29&type=code https://github.com/search?q=load%28%22%40rules_foreign_cc%2F...
- dsissitka 3y agoYou might like grep.app. It looks like it filtered out all of the noise in this case: https://grep.app/search?q=load%28%22%40rules_foreign_cc//foreign_cc%3Adefs.bzl%22%2C%20%22ninja%22%29 https://grep.app/search?q=load%28%22%40rules_foreign_cc//for...
- 100k 3y agoI work on this, and I agree we should do more to improve ranking. Exact duplicates are supressed, but often forks have different versions of some files, so they come up in the results. If you don't want to see forks, you can exclude them. Here's your same search, but converted to a regex and not including forks. I only get 3 results. https://github.com/search?q=%2Fload%5C%28%22%40rules_foreign_cc%5C%2F%5C%2Fforeign_cc%3Adefs.bzl%22%2C+%22ninja%22%5C%29%2F+NOT+is%3Afork&type=code https://github.com/search?q=%2Fload%5C%28%22%40rules_foreign...
- inferiorhuman 3y agoAre you at all embarrassed at all by how buggy and intentionally unusable Github has become?
- frankjr 3y agoNot including forks should be the default IMO. There can be a button in the Advanced section of the sidebar to include them.
- fluxem 3y agoIt boggles my mind that github search is case-insensitive and doesn't search the whole words. For a search term getInfo it will return: GetInfoCell Don't get me started about how it's harder than it needs to be to limit the search to your own repos
- charcircuit 3y agoUse sourcegraph to search instead. It doesn't even require you to be logged in.
- quickthrower2 3y agoOne of my favourite tools, I use it almost daily at org level to find related code.