6 ms·
Grep.app – Search across a half million Git repos
- IncRnd 4y agoPlease don't check your keys or passwords into github. This was my first search to show AWS API Keys. [1] and my second to show google API keys. [2] Unfortunately, there are many more of these types of searches - and far too many results. [1] https://grep.app/search?q=AKIA%5B0-9A-Z%5D%7B16%7D®exp=true https://grep.app/search?q=AKIA%5B0-9A-Z%5D%7B16%7D®exp=tr... [2] https://grep.app/search?q=AIza%5B0-9A-Za-z%5C%5C-_%5D%7B35%7D®exp=true https://grep.app/search?q=AIza%5B0-9A-Za-z%5C%5C-_%5D%7B35%7...
- capableweb 4y agoAs far as I can tell, all the git repos come from GitHub, and GitHub automatically sends you a warning if you commit access keys of various kinds (and I seem to remember they even de-activate some, maybe AWS keys, automatically? Not 100% sure about this). Most of the keys shown are unlikely to be working, especially the ones older than just a couple of days (either owner notice, or someone else was faster than you, got access and regenerated new keys for themselves)
- IncRnd 4y ago> or someone else was faster than you, got access and regenerated new keys for themselves There are many people who do these types of searches - regularly. If there is any key material checked into gh, consider it immediately compromised, even if the repo is removed.
- capableweb 4y agoTons of automated scraping for these sort of things as well, with the whole flow of "check if key is working, check permissions, if authorized, generate new key and deactivate old one".
- IncRnd 4y agoYes. There are indeed several mitigations in place. However, my point was that within 5 seconds of looking at this tool I was able to uncover hundreds of potential issues. There are many other regexs I have on my list. Please don't fall into the trap of "we have anti-virus, so we can't get attacked."
- capableweb 4y agoSorry if I was unclear, my previous comment was about attackers doing that in order to compromise infrastructure, not that companies are doing it in order to defend themselves. But companies probably do it to defend themselves as well, but I'm not as familiar with that as with the other side of things.
- greysteil 4y agoGitHub PM here, just confirming that that's correct (re: us sending detected secrets to the relevant service provider, who take action automatically). There's more detail here: https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning#about-secret-scanning-alerts-for-partners https://docs.github.com/en/code-security/secret-scanning/abo....
- iambateman 4y agoThis is probably one of the single most important security measures on the whole internet. Thanks for keeping an eye out.
- abraham 4y agoMost of the Google API keys look like they are client side keys and should be configured to only work on specific origins or apps.
- TYMorningCoffee 4y agoYour search has a lot of false positives like AKIAIOSFODNN7EXAMPLE
- madeofpalk 4y agoMost of these actually look like fake keys to mock data in unit tests.
- teaearlgraycold 4y agoHacker noises - "I'm in"
- kristopolous 4y ago----- BEGIN RSA NOT SO PRIVATE KEY -----
- arp242 4y agoMany of these are tests, false positives (e.g. base64 images or "Migration1599570560FixSlovakiaDisplayedAsSloveniaTest"), or examples from documentation. Can't find a single match for an actual API key.
- codalan 4y agoThese are probably invalid by now. https://grep.app/search?current=2&q=api_key%20%3D%20%22%5Ba-zA-Z0-9%5D%2B%22®exp=true https://grep.app/search?current=2&q=api_key%20%3D%20%22%5Ba-... As others have posted, Github implemented an alerting system for keys/passwords/etc. that have been checked into the codebase. My guess is most of these will be invalidated by now, either by the developer requesting a new key (and not publishing it), or the provider invalidating it for them.
- zikohh 4y agohttps://sourcegraph.com/search?q=context:global+AKIA%5B0-9A-Z%5D%7B16%7D&patternType=regexp&sm=1&groupBy=repo https://sourcegraph.com/search?q=context:global+AKIA%5B0-9A-... Sourcegraph is much better than this site.
- asicsp 4y agoPrevious discussion: "Show HN: Search code in GitHub repos using regular expressions" https://news.ycombinator.com/item?id=22396824 https://news.ycombinator.com/item?id=22396824 (614 points | Feb 23, 2020 | 155 comments)
- barbinbrad 4y agoAt first glance, this is great!
- khaledh 4y agoPrevious discussions: - https://news.ycombinator.com/item?id=28424845 https://news.ycombinator.com/item?id=28424845 (September 5, 2021 — 54 points, 6 comments) - https://news.ycombinator.com/item?id=22396824 https://news.ycombinator.com/item?id=22396824 (February 23, 2020 — 614 points, 155 comments)
- NoZebra120vClip 4y agoThis is a great idea and I commend it and I hope others find it useful. However, I have a rather narrow use-case for GitHub searches, and in my initial experiments, this tool comes up 100% empty on common searches I use every day for work. Most of my GitHub searches don't actually revolve around actual code-code. I search for text strings in prose that turn up in code searches. So the file types may be text or MarkDown instead. Perhaps grep.app only indexes and presents actual code-code and filters out all the prose I'd be interested in. Sad, because there are actually a few instances where my searches involve code with many intricate special characters, and GitHub's current code search ignores all those.
- deleted 4y ago[deleted]
- voytec 4y agoGitHub can be searched directly without any JS bloat.
- bcjordan 4y agoI'd be curious to hear how the ingestion/search pipeline works — the searches are incredibly fast!
- Scaevolus 4y agoIt uses Solr: https://news.ycombinator.com/item?id=22397392 https://news.ycombinator.com/item?id=22397392
- birriel 4y agoMissed opportunity to call it "Grit."
- deleted 4y ago[deleted]
- dang 4y agoRecent and probably related: The technology behind GitHub’s new code search - https://news.ycombinator.com/item?id=34680903 https://news.ycombinator.com/item?id=34680903 - Feb 2023 (168 comments)
- deleted 4y ago[deleted]
- ptannor 4y agoHaha I can share my first use case - searching for repos I'm involved in, kind of like searching for myself on Google Scholar
- moneywoes 4y agoHow Is this different than GitHubs search?
- 100k 4y agoHi, I work on GitHub code search. Grep.app is based on Solr and indexes about 500,000 public repositories. As far as I know, the author has not shared their tokenization strategy, but based on the the results I expect they are indexing trigrams to perform regular expression matching. The current GitHub code search is based on Elasticsearch and indexes more than 100M repositories. Its tokenization is based on whitespace, case changes (like CamelCase) and punctuation (like kebab-case) and strips out non-letter characters like < or { which is why it can't do exact match search. Our new code search, currently in beta and indexing about 45 million repositories, uses an search engine we built in house that indexes content using a technique we call sparse ngrams. This allows us to execute searches faster than a trigrams index, while also being smaller than a positional trigram index. My teammate discussed some of the technology behind it in our blog post that was published yesterday: https://github.blog/2023-02-06-the-technology-behind-githubs-new-code-search/ https://github.blog/2023-02-06-the-technology-behind-githubs...
- xigoi 4y agoSearch across a half million GitHub repos. It doesn't support any other git hosts than the proprietary one.
- sebastianconcpt 4y agoYet it doesn't have a clue about the existance of one of my 17 star GitHub public repos