37 ms·
Show HN: Search code in GitHub repos using regular expressions
- simonw 7y agoImpressive! Really fast, full featured code search across a huge corpus. 1. How did you build the index? Did you use a GitHub dump of some sort? How often do you refresh it? 2. Is it Elasticsearch or similar or a completely custom engine? 3. What kind of RAM/CPU are you using to power it? 4. Any plans to open source the code or commercialize the technology? I could absolutely imagine paying for a private code search engine like this to run against a large internal company codebase spread across many repositories.
- danfox 7y agoThanks! It's built on top of Solr. It fetches the repos from GitHub - it should pick up any updates to repos within a few days. It's running on a couple servers with 20 cores each, which is not really enough for the traffic it's getting right now.
- heipei 7y agoI'd be curious how you built the step from regex to ElasticSearch. My guess would be an n-gram (3-gram) index in ElasticSearch and then translating the regexes to that, but just curious if you built that custom or used something off-the-shelf. Love the site!
- speedplane 7y ago> I'd be curious how you built the step from regex to ElasticSearch. My guess would be an n-gram (3-gram) index in ElasticSearch and then translating the regexes to that, but just curious if you built that custom or used something off-the-shelf. Love the site! I'm pretty sure Elasticsearch supports regex search, it's just that it's horrendously slow and can blow up the system.
- fanf2 7y agoI wonder how this compares to Debian Code Search (https://codesearch.debian.net/about https://codesearch.debian.net/about) and Russ Cox’s code search tools (https://swtch.com/~rsc/regexp/regexp4.html https://swtch.com/~rsc/regexp/regexp4.html). Obviously the source material is different (Debian packages vs GitHub repos) and grep.app also uses re2, but that is all I can see from a look at the “about” blurb.
- sciurus 7y agoAnother related tool is https://searchfox.org/ https://searchfox.org/ https://github.com/bgrins/searchfox https://github.com/bgrins/searchfox
- aodj 7y agoYou have no idea how often I've wanted something like this for GitHub. Thanks so much!
- enriquto 7y agoCurious that I found many "secret forks" of my stuff, but none of my repos is directly indexed.
- justanotheratom 7y agoCan you elaborate how you found them?
- enriquto 7y agoI looked for strings that I am sure only appear in my code, and I found several copies of them, but not mine.
- polyphonicist 7y agoCan you provide detailed steps to reproduce? What strings did you search? Two examples of repos that appeared in the results? What is the link to your repo that did not appear in the results? Details like this would help the OP to track down the exact cause of why it has indexed the forks but not the original repo.
- enriquto 7y agoThe authors are quite explicit that this site only includes a fraction of all github repos. Thus, this is not a "bug" that needs to be corrected. In my case, I am not talking about forks but about people who copied my files into their repositories (with proper attribution and respecting the license). I just searched for my surname and was happily surprised to see it in major projects like ffmpeg, pytorch, bytedeco, scikit and opencv.
- glouwbug 7y agoAmazin, why Microsoft hasn't built this for GitHub yet is beyond me. Can it grep on individual repos?
- funklute 7y agoWhy would you want to use this tool to grep individual repos? If you know the repo you're interested in, you can just clone it and then grep it locally...?
- m3kw9 7y agoSo you don’t have to clone it and grep it locally
- big_chungus 7y agoSome things can take a while to clone. On the top end, repos like blink, webkit, and gecko can take half an hour or more.
- leni536 7y agoEven with --depth=1?
- tehlike 7y agoI like to grep a code pattern through out all repos. I use gstreamer, and sometimes i just don't know how to use it to do a specific thing. So i search substrings to find out usage patterns by other people.
- hcm 7y agoI built https://grephub.com https://grephub.com for that. It doesn’t maintain an index so it’s not super snappy, but it’s good enough / better than you’d expect in many cases!
- lol768 7y agoHow did you pick the 500k repositories to index out of the 28 million or so which are public?
- danfox 7y agoIt was based on the number of stars/forks and the size of the repository.
- atxbcp 7y agoThere must be something else or something wrong, because you indexed one of my small repo (~100 stars, ~20 forks, ~20Mb) and not the bigger ones (~500 stars, ~100/150 forks, ~150Mb)
- giovannibonetti 7y agoMaybe he is limiting it to repositories of 50 MB or less, for example.
- patrickdevivo 7y agoor possibly there's a "time decay" element where more recently "popular" repos are prioritized, not just based on absolute start/fork count
- tempay 7y agoLooking around at repositories I'm familiar with this seems to be the case.
- ferenczy 7y agoI would say this needs a list of indexed repos and mainly an explanation of how it exactly works to be usable (how's the index build and how often it's refreshed, what types of files are being indexed, etc.). Otherwise, there's no much value in searching in an unknown data, is it? Anyway, to not only criticize, good job! It's definitely one of GitHub's missing features. And I can imagine it's not an easy job to build something like that. But as I wrote, it really has to be well explained to be actually usable.
- clarry 7y ago> there's no much value in searching in an unknown data, is it? So you know exactly how Google's index works? I think "best effort", whatever it is, is useful even if I don't know the specifics of what it captures or misses. As long as it returns useful results.
- TACIXAT 7y agoI do not have a great example to try on my phone, but are results deduplicated? That's my big peeve with GitHub search is getting 5 pages of the same forked repo.
- danfox 7y agoThere isn't any deduplication, although that will hopefully be less of an issue at this point since there's a limited number of repositories in the index.
- hoorayimhelping 7y agoto grep specific repos locally, I use a tool called Hound, https://github.com/hound-search/hound https://github.com/hound-search/hound developed by a couple of engineers at Etsy while I was there, but never released officially.
- deleted 7y ago[deleted]
- jasoncwarner 7y agoThis is awesome! @danfox, sent you an email though commenting here too. I'm the CTO @ GitHub. Would love to talk to you about this and other things we are building in this area at GitHub. Feel free to email direct to jason at github.com
- sixwing 7y agohah. you beat me to it, Jason. @danfox, i'm always down to talk code search as well - rand@github.com
- latenightcoding 7y agogithub's code search is notoriously bad, feels like a huge missed opportunity. Nice to see you guys reaching out to other people working in this area.
- lucasverra 7y agoI'm not a dev and what i like to to do is go searching for code (a la exact match) to replace whatever variable or text should be changed. Github search in repo kinda worked at some moment, then not. Then i had to download repo in my local; run VS code (updating first), search there, modify, push. I wish i could do this on Ghub web GUI
- erikpukinskis 7y agoThe fact that you can’t search for file names is the funniest part to me.
- nickthemagicman 7y agoYou can but only in the repo itself not on a site wide scale.
- cole-h 7y agoReally? I searched for `filename:home.nix` (which brought me to https://github.com/search?utf8=%E2%9C%93&q=filename%3Ahome.nix https://github.com/search?utf8=%E2%9C%93&q=filename%3Ahome.n...). That seems site-wide to me, unless I'm misunderstanding you.
- whatever1 7y agoWhy regex still exists? It is unintuitive, requires mastering an obscure syntax, it is very hard to debug, and very difficult to explain to others how it works. It feels like we are trying to write intermediate code by ourselves, while we should have a human readable language that generates regex.
- frabert 7y agoDo it! You will find that it's very easy, but the result will either be extermely verbose or just like regex. Since most regexes (at least for me) are meant as one-time-use, the extra verboseness has no added benefit. If you have complex needs, you should probably be using something other that regex, anyways.
- thanatos_dem 7y agoExtremely verbose is right. Here's one such approach in java that I found last year - https://github.com/sgreben/regex-builder https://github.com/sgreben/regex-builder. Yeah, regex can be a bit clunky at times and has a steeper learning curve, but they're pretty industry standard at this point, and portable across languages with a few caveats.
- GrantZvolsky 7y agoThe investment into learning regexes is worth it if you write or read enough of them. They become the human readable language you speak of, eventually. The question is where the threshold lies.
- tyingq 7y ago"Why regex still exists?" Is there an alternative that is clearly superior?
- samatman 7y agoYour mileage may vary, but to my taste, the lpeg flavor of Parsing Expression Grammars is clearly superior. It uses operator overloading to build patterns from component parts. I don't think anything can replace the terseness of regex for command line use, or vim searching, cases like that. But for a program, give me lpeg every time.
- sonicxxg 7y agoTypical HN comment. I could swear, if someone came here to share a breakthrough cure to HIV, some random nobody would still find a way to say something like "Hey, how come you haven't tested it against a population of female midgets suffering from cancer?".
- dang 7y agoThere's no need for personal attack. We ban accounts that do that, so please don't. Cherry-picking one post from a statistical cloud and calling it typical is dodgy. Even the distribution in this thread doesn't match your description. Actually, even the comment you're picking on doesn't match your description. We detached this subthread from https://news.ycombinator.com/item?id=22397156 https://news.ycombinator.com/item?id=22397156.
- deleted 7y ago[deleted]
- nickjj 7y agoHey Dan, if you ever wanted to come on my podcast to talk about your tech stack (how your site is developed / deployed, lessons learned, etc.), I'd love to have you on. That podcast is at: https://runninginproduction.com/ https://runninginproduction.com/, drop me a line at nick.janetakis@gmail.com if you're interested.
- sqs 7y agoThis is really cool. What are you using it for? Usage examples, debugging, etc.? I'm the CEO at Sourcegraph (universal code search for companies to use on their internal code). Our product is really optimized for searching a company's internal code right now, but soon we'll start working on offering much better search for public and open-source code as well. If you'd like to help out or just chat, please reach out! sqs@sourcegraph.com
- edwinyzh 7y agoSorry, but his code search covers far more languages than yours the last time I tried yours :)
- akavel 7y agoDoesn't sourcegraph allow to just search regex over any files in a repo? This is textual search, so how are languages relevant to it? I didn't seem to have problems with that
- edwinyzh 7y agoSorry, maybe I have confused SourceGraph with https://searchcode.com https://searchcode.com, but last time I tried, it supports only most widely used languages such as Java, Python and so on, but not the language I use (Delphi/Object Pascal). I'm sorry if I'm wrong.
- hartator 7y agoExcellent work! I am the CEO at SerpApi. If you need a job, shot me an at julien _at_ serpapi.com.
- j1elo 7y agoGitHub confirmed to me that their search is not able to find in substrings; this is annoying because if you want to find all affected code among all possibly involved repositories, before a change, you need to clone them and grep locally. In the end this means you need to clone absolutely everything you work with, because otherwise you might miss changing that one repo you didn't think of: https://stackoverflow.com/questions/43891605/search-partial-words-in-github-organizations-code https://stackoverflow.com/questions/43891605/search-partial-... I've used Sourcegraph and it was cool; will have a look at this new tool too. But, GitHub pretty please add plain food old grep abilities to your search!
- oefrha 7y agoA tangent, my biggest gripe with GitHub code search (within a repo) off the top of my head is the inability to blacklist directories or only search whitelisted directories. Often times I want to look up the implementation of a function, and bam, three pages of results from tests.
- Noctem 7y agoI'm glad I'm not the only one. It's very common that I'll be searching for a keyword that only appears in the actual code a handful of times but hundreds of times in tests. GitHub's search is practically useless in those cases. I almost always just resort to cloning and searching with ripgrep, which can be annoying if I have no other reason to have the codebase on my machine or it's just a one-off.
- cynicalreason 7y agoyeap .. having this issue as well, trying to easily find where a method is defined in JS/TS I'd so much want to be able to exclude `*.(spec|test).(js|ts|jsx|tsx)`
- cddotdotslash 7y agoThe interface for this is really clean and nice - did you use a theme or framework?
- danfox 7y agoThanks! It's using Elastic's Search UI (https://github.com/elastic/search-ui https://github.com/elastic/search-ui) and Ant Design (https://github.com/ant-design/ant-design https://github.com/ant-design/ant-design).
- Existenceblinks 7y ago^(.)'(.)'(.)$ I got a tooltip say: Error: JSON.parse: unexpected character at line 1 column 1 of the JSON data Update:: Oh ^(.)"(.)"(.)$ works and fast.
- danfox 7y agoI think that error was just because the server was overloaded - sorry about that.
- thanatos_dem 7y agoNext post from danfox - “how to get 3 job offers in 3 hours”. Already has been publicly contacted by: - GitHub CTO - SerpApi CEO - SourceGraph CEO Search is hot right now!
- Existenceblinks 7y agoI'm surprised as well, think why big tech companies didn't have this awesome search already.
- thanatos_dem 7y agoIf this were to be offered by an actual company (a first party solution), there are some features that'd be expected that make the problem space a lot harder. Here's an "intro to search" article that's a good read, and I'll use it to highlight some of the things that'd be different in a first party solution - https://medium.com/startup-grind/what-every-software-engineer-should-know-about-search-27d1df99f80d https://medium.com/startup-grind/what-every-software-enginee... (See the "Theory: the search problem" section) Size: This is only indexing ~500k public repos. A first party solution would be expected to index all of it, public and private. Indexing speed: This can take up to a few days to index. A first party solution would be expected to have a much lower index latency - seconds to minutes. Query language: This can (and does) have its own simple query language. A first party solution would need to have support embedded into and not break backwards compatibility with the current query language. Context-dependence: A first party solution would be expected to index private repos as well, and now the query context (logged in user) becomes another variable in an already multi-variate problem space. Latency: Gets harder with scale, and a first party solution would likely provide a SLA/SLO around latency. Access control: Same issue as context-dependence, with private repos being included. There's also unknown but likely considerations around compliance and internationalization, which are quite tricky problems. Note - I don't mean for this to be critical of the author at all. This is an awesome and useful tool, with a fantastic UX. I just want to make it clear that search at scale is a lot harder than it seems at first glance, especially as the feature requirements increase.
- 7y ago
- dang 7y agoI still miss Google Code Search, which was a great way to find examples of anything I wanted to learn about in programming and usually answered my questions better than anything else, including Stack Overflow. Has it really been 8 years... https://news.ycombinator.com/item?id=3112029 https://news.ycombinator.com/item?id=3112029 If this tool can fill that hole in my world, I'll be stoked. I've bookmarked it.
- londons_explore 7y agoGoogle code search still exists as long as you want to search Chromium source code. [1]: https://cs.chromium.org/ https://cs.chromium.org/
- londons_explore 7y agoThe main difference it has IMO is it indexes a symbolic code graph extracted from halfway through the compilation process. That means when you search, it knows which functions are frequently called. For example, the LOG() macro is defined in hundreds of places, but the one in logging.h is the one everyone calls, so that's the one that comes top of the results. It also keeps track of back references, so you can search "who calls any function in this file", which is very hard to do with any other search system. Major disadvantages are it only indexes one build config, so if you're debugging android code in a multi-platform project and the indexing was done on the windows version, you won't find much (apart from dumb text based search which it does in addition). The difficulty of compiling every project to build a decent index would make this approach hard on a GitHub scale - all it takes is one missing header file from a dependency not in the repo and the build fails and the whole project can't be indexed. Also, have fun with things like JavaScript which are so dynamic you have to solve the halting problem to know which bit of code calls which other.
- jakear 7y agoAny plans to include backrefs? I'd like to see how many examples of /(\w+) && \1\./ are out there in .js/.ts compared to /(\w+)\?\./
- tyingq 7y agoThe about blurb mentions it uses RE2. So backreferences aren't likely. See https://github.com/google/re2/issues/101 https://github.com/google/re2/issues/101
- jakear 7y agoRipgrep is based on RE2 and supports backrefs. Wonder why they didn't use that.
- burntsushi 7y agoNot quite. ripgrep uses Rust's regex engine, not RE2. Rust's regex engine is descended from RE2, but there is no code sharing. Rust's regex engine does not support backreferences. RE2 does not either. ripgrep does however have a -P/--pcre2 flag which causes it to use PCRE2 instead of Rust's regex engine. PCRE2 supports backreferences and other things, like look-around. (ripgrep also has an --auto-hybrid-regex flag, which will automatically enable PCRE2 for you if you write a regex with backreferences or look-around.) The reason not to use an engine like PCRE2 for a project like this is because it would be trivially exposed to ReDoS: https://en.wikipedia.org/wiki/ReDoS https://en.wikipedia.org/wiki/ReDoS
- jakear 7y agoThanks for the clarification. As an aside, how difficult do you think it would be to compile ripgrep to wasm? In VS Code we use ripgrep for full-workspace search and Node's regex library for in-memory searches, but this leads to discrepancies and issues such as catastrophic backtracking in the in-memory search.
- 7y ago
- danielecook 7y agoWow. This is incredibly helpful. You can use it to see how someone may have used a function with named parameters: my_function(label=x, option_1=2) my_function.*option_1 # search
- SlowRobotAhead 7y agoThat was my first thought. I’ll have to wait until tomorrow to try it, but I have one super rarely used function ima rare package I’d love to see how other people are using.
- inetknght 7y agoI was going to say that I didn't want javascript on this. But it's actually pretty #neat. It's all tidied up into a single app without any dependencies. This rocks and, so far, seems way way WAY better than Github's own search tool.
- yuz 7y agoDo you index the default branch of every repo? Or do you just index the master branch?
- danfox 7y agoIt indexes the default branch of each repo.
- yuz 7y agoCool. Keep up :) definitely gonna share with my co-workers. Can't wait for filename filters which would make this the perfect solution
- danfox 7y agoThanks :) If you type into the path filter box, that'll match against the full path for each file, so you can use that to filter on a filename.
- deleted 7y ago[deleted]
- dbielik 7y agoCheck out https://NerdyData.com https://NerdyData.com :-)
- appleflaxen 7y agoThis seems unrelated. I hope u/dang sees your comment history; you are basically just spamming nerdydata.com
- patrickdevivo 7y agoThis is really cool! Awesome work. I assume you've seen https://sourcegraph.com/ https://sourcegraph.com/ as well? This to me seems much clearer and a bit more intuitive (though I've only spent a little time in sourcegraph). Really really cool. Does it also search code comments?
- edwinyzh 7y agolast time I tried sourcegraph doesn't cover the language I use, so it's useless to me.
- akavel 7y agoFor regex?? how's language relevant?
- edwinyzh 7y agoSorry, maybe I have confused SourceGraph with https://searchcode.com https://searchcode.com, but last time I tried, it supports only most widely used languages such as Java, Python and so on, but not the language I use (Delphi/Object Pascal)
- deleted 7y ago[deleted]
- welder 7y agoCan I search only additions/deletions? Recently when searching GitHub I wanted to find if anyone had replaced the usage of a deprecated method with the new one, because the docs for that library don't mention the non-deprecated method name.
- dabei 7y agoIt’s interesting how it took so many years for such an obviously useful tool to emerge. I guess hosting this is finally getting cheap enough.
- edwinyzh 7y agoI've been wondering the same thing for many years. And I don't know why Google killed Code Search
- franciscop 7y agoThis is amazing! One thing that allows me to do, which I wasn't before, is to do a search for the repos that use some of my open source. While there were some tools for this, they fail sort for older projects where using a library meant copy/paste it into your project, which is not reported in the CDN stats, npm installs or github "uses". Now I can run a search with a bit of code that is only present in my library and find reliably those who copy/pasted it. While I publish my code under the MIT, this would also be very useful for those publishing under the GPL to detect bad actors.
- AdrianEGraphene 7y agoFeels like magic to me! Lets me easily see who's working on similar topics. Thanks!
- edwinyzh 7y agoCan you share your search string? Thanks.
- edwinyzh 7y agoAwesome! To me it looks like the come back of "Google Code Search" which I've been missing for many years!
- edwinyzh 7y ago@danfox, Without revealing your tech/business secretes, I wonder if you can share some tips about building such a search app :)
- KhoomeiK 7y agoHow is it that fast?
- sn4pp 7y agoSeems to be good for stuff like api_key="[a-z0-9]+" Ty
- bananaeater 7y ago"We didn't find any matching results."
- rafi_kamal 7y agoYou need to enable regular expression.
- blackandblue 7y agothank you so much for doing this! i hope it continues to open more doors of opportunities to you! primo, this is a crazy snappy proof that shows that github search can be done. next, the UI is amazing. and finally, all my queries worked! i am now going to remove "github search sucks" from my to-be-published rants because this post demonstrates that 1. people care 2. github was already working on it.
- chasers 7y agoHow do you handle expensive regex statements?
- bilekas 7y agoThis is cool, reminds me of the vulnerability search too. https://shhgit.darkport.co.uk/ https://shhgit.darkport.co.uk/
- OutsmartDan 7y agoThis is one of the fastest, most responsive searches i've ever used. Great work!
- mtnGoat 7y agothis is awesome stuff, thank you! great work!
- doubleorseven 7y agoMy last name(Ament) is really rare where I come from, so I've used the tool to find other people with the same last name. Was not disappoint. Thank you!
- thrownaway954 7y agomight be a good idea to have some sort of clickable "demo" search or "try these" example on the frontend page to show off the capabilities of this.
- mrkramer 7y agoVery similar to https://news.ycombinator.com/item?id=18565239 https://news.ycombinator.com/item?id=18565239 Backend for codegrep was Play framework + Elasticsearch and you could search by programming languages. Screenshot: http://archive.is/0mFML http://archive.is/0mFML
- stagas 7y agoI wish there was something this fast, but for searching error outputs instead (along with discussions/solutions).
- w-m 7y agoAmazing feat! Something I found when testing the regexp: the highlights seem to be off sometimes. When grepping for '<.*?@gmail.com>' (sorry, just the first thing that came to mind to try out the regexp), the second highlight in the first result seems to be in the wrong location: https://grep.app/search?q=%3C.%2A%3F%40gmail.com%3E®exp=true https://grep.app/search?q=%3C.%2A%3F%40gmail.com%3E®exp=t... https://imgur.com/a/VyUXhcF https://imgur.com/a/VyUXhcF
- tekkk 7y agoSuperb work. You built a better code search than Github (well with some of its features missing sure) with a lot less resources. Shows how stagnated the progress in big companies is after a service is deemed "good enough". Good for you kicking them in their butts to lead the way. Hope you get out of this something else too than HN karma. Really like the minimalistic design, not too designy but still easy on my eyes. Just the way I want it to let me focus on the task at hand
- habit20 7y agoHello world