15 ms·

Google Open Source Code Search

• kediz 7y ago
  Code search is a great tool! It really helps with productivity! But sometimes it is very easy to go down the rabbit hole. I wonder if they can open source code search itself.
  • Eridrus 7y ago
    I don't know the state of it, but Kythe is open source: https://kythe.io/ https://kythe.io/ But in reality you probably want something more like SourceGraph which packages everything up nicely so that you don't need to worry about it, or something more specialized.
    • thedance 7y ago
      Bringing up kythe from scratch is very daunting. First thing I did when I left google, of course, but still really hard.
      • __float 7y ago
        What did you use for a UI?
        • thedance 7y ago
          That's one of the main problems. I started with the demo frontend that comes in the box and just hacked on that. I'm no UI developer by any stretch.
        • dmoy 7y ago
          Try TreeTide
  • hanwenn 7y ago
    cs.opensource.google runs on Google's search infrastructure, so it's unlikely to be open sourced. https://github.com/google/zoekt https://github.com/google/zoekt is open source, but lacks cross-referencing, and has a more spartan UI.
• deleted 7y ago
  [deleted]
• phillco 7y ago
  Misread as "Google Open Sources Code Search" :'( cs.opensource.google is amazing
  • hoten 7y ago
    yeah, title is misleading. https://cs.chromium.org/ https://cs.chromium.org/ is the next best thing.
  • arunaugustine 7y ago
    I found this on github in Google's org : https://github.com/google/codesearch https://github.com/google/codesearch
    • thedance 7y ago
      That "codesearch" is only superficially related to this one. The main feature of _this_ codesearch that makes it so useful is the cross references to callers, callees, and overrides. Ye olde codesearch has more in common with things like livegrep.
      • apta 7y ago
        What's the internal codesearch tool written in?
      • dmoy 7y ago
        > The main feature of _this_ codesearch that makes it so useful is the cross references to callers, callees, and overrides. Ye olde codesearch has more in common with things like livegrep. However this part of internal codesearch is the one part that is actually (partially) open sourced: kythe.io
  • escardin 7y ago
    There's this https://github.com/google/zoekt https://github.com/google/zoekt. It's pretty light on features, but dang if it isn't fast and precise.
• kannmig 7y ago
  Seems like this is based on Google's own internal code search tooling, something most engineers at Google rely on for every day code-level work. I personally can't even begin to imagine how I'd navigate the gigantic codebase without it. (I work at Google)
  • microdrum 7y ago
    You'd use Sourcegraph, probably.
    • mistrial9 7y ago
      what is the constant phone-home activity on that opaque container they send as SourceGraph.. It is occassionally the case that devs have too-fast machines, so their code isn't seen on ordinary equipment. With SourceGraph and other inner-network-devs tools, the amount of chatty traffic and build dependancies seems seriously off-putting, trending to useless with ordinary network.
      • deleted 7y ago
        [deleted]
      • voz_ 7y ago
        This seems like a bad attitude. Perhaps you could constructively ask for a sourcegraph-lite that does less, in return for less deps / networking complexity?
        • slimsag 7y ago
          I am a dev at Sourcegraph, I'd be very open to any feedback. You can firewall off Sourcegraph 100% for complete confidence, and aside from the first admin's email address (so we can notify them of any security updates) we only send back aggregated anonymous usage statistics which we are extremely transparent about: https://docs.sourcegraph.com/admin/pings https://docs.sourcegraph.com/admin/pings We sell developer tools, not user data.
• CydeWeys 7y ago
  Nice, my team's project is on there! (Nomulus)
• vikinghckr 7y ago
  Googler here. We have the same Code Search tool internally, this is honestly one of my favorite things about working at Google. Great to see this open sourced.
  • sophiebits 7y ago
    It hasn’t been open sourced, right?
    • kyrra 7y ago
      Correct. This is just exposing indexed versions of some of our larger open source projects with this code search. Chromium also has its code indexed by the older version of this tool: https://cs.chromium.org/ https://cs.chromium.org/
      • jedimastert 7y ago
        > Chromium also has its code indexed by the older version of this tool: https://cs.chromium.org/ https://cs.chromium.org/ Chromium recently switched over to a new version of code search.
        • krackers 7y ago
          They seem to have reduced the information density and killed readability as well with the "material ui" redesign. The old code search UI was perfect, with enough contrast to allow you to quickly grep through xrefs to locate relevant entries. The page itself was lightweight as well. Compare that to the redesign where each xref jump has me staring at a spinner for half a second, the xref bar has no visual separation between type, filename, and code snippet, all buttons visually indistinguishable with a blue on white color scheme, the "layers" dropdown is replaced with a mishmash of buttons scattered across the layout, etc. I really hope they're not forcing this abhorrent redesign on their developers as well.
        • jbroman 7y ago
          Specifically, the new version is at https://source.chromium.org/ https://source.chromium.org/, and the Android counterpart is https://cs.android.com/ https://cs.android.com/.
• rochak 7y ago
  I hope I get to work for Google sometime.
  • vikinghckr 7y ago
    Just apply. Google is actively hiring all year round.
    • rochak 7y ago
      I am a grad student right now with 2 years of industry experience. Google still prefers people who are extremely good at Data Structures and Algorithms. I like doing them, but not so much to just grind them for the sake of getting into Google. I like to learn how to design big systems and grinding Data Structures and Algorithms seems like a waste of time.
      • CydeWeys 7y ago
        I put in "only" 40 hours of refreshing on data structures and algorithms, and doing some practice coding problems, in the weeks leading up to my interview. And I got the job. Frankly, it's been the best hourly return on investment of anything I've done in my life up to this point, by far. Assuming I wouldn't have gotten the job otherwise (which seems reasonable), each of those hours spent studying has proven to be worth several tens of thousands of dollars. I'm not exaggerating; I just did the math. Maybe the interviewing process is broken or sub-optimal or whatever, but it is what it is, and if you can get through it by doing some additional studying, then it's absolutely worth it. Google is a good place to work on designing big systems, so if that's your interest, consider just putting in the work.
        • rochak 7y ago
          This is a solid advice. Thanks! I will try to dedicate a portion of my day to brusing up Data Structures and Algorithms and maybe, eventually, I will get good enough to crack the interview.
  • jedimastert 7y ago
• gerash 7y ago
  It looks like they haven't integrated the kythe cross reference DB as the symbols aren't clickable
  • kwh5336 7y ago
    A few projects have configured kythe for at least one language. See bazel, go, gvisor, kythe, and tensorflow.
• sqs 7y ago
  Sourcegraph CEO here. This is the same underlying code search offered for a while by Google Cloud Source Repositories for private code, and it’s cool to see this usable for Google’s own open-source code, too. If you want to get universal code search for your own (private) code on any/all code hosts, Sourcegraph is easy to set up internally (self-hosted Docker install) at https://docs.sourcegraph.com/ https://docs.sourcegraph.com/. Or you can get code search for all OSS projects at https://sourcegraph.com/search https://sourcegraph.com/search. More general info at https://about.sourcegraph.com https://about.sourcegraph.com. Lots of Xooglers and current Googlers use Sourcegraph, too. Just mentioning Sourcegraph because I’ve seen several other folks mention us in the comments (thanks!).
  • monadic2 7y ago
    What’s the code licensed under? It’s not clear from your site at first glance.
    • sqs 7y ago
      It's open core (Apache 2 + some non-OSS parts for enterprise features). All of the code is public and we develop in the open at https://github.com/sourcegraph/sourcegraph https://github.com/sourcegraph/sourcegraph.
      • dragonsh 7y ago
        Sourcegraph only support git repository so it's not very useful for enterprise with mercurial, svn or other distributed version control systems. There is another open source application for code search opengrok [1] (it's completely open source unlike sourcegraph and supports multiple version controls beside git). Take a look. It's easy to install and operate on bare metal, cloud and containers, instead of convoluted sourcegraph way of kubernetes or docker. [1] https://github.com/oracle/opengrok https://github.com/oracle/opengrok
        • aurelianito 7y ago
          You always can bridge to git from svn and mercurial. It is almost seamless, and after generating the git repository everything will work.
• repomono 7y ago
  We are building similar experiences for internal repos. Demo: https://demo.repomono.com/cs/view.php https://demo.repomono.com/cs/view.php Code is here: https://github.com/repomono/cs https://github.com/repomono/cs
• malkia 7y ago
  Really loved this interface (also cs.chromium.org) while I worked at Google. It was easy for me to orient myself, find what uses this and that, where it's being used, and then it had whole "debugging" facility: You select your binary on borg (think kubernetes/docker), and it'll fetch from the binary with which CL (think like perforce "CL") it was built, and/or additional cherrypicked CL's, then it'll somehow go back in-time and represent how the source code looked then. later one can (I tried it in Java, but I believe it's available for other languages too), you can inject statements right around the begining of function (a way of breakpoint), and that statement can be something like - let's log how this function was called - you were able to reference nearby statements. This could be set from the command-line, and took a bit mastery (and was bit afraid first time using it, or more like had chilling effect on me), but then my task (with 10 or 11 instances) reported these log lines, and I was able to see them in the browser. (I have no experience with GCP, or the public face of Google Cloud, so I don't know what's available there), but this was freakin cool.
• typon 7y ago
  Question for Googlers or others: What do you think is the most well-written piece of software produced by Google? I would like to study how the world's best engineers write code. (Preferably C++, as it's the language i'm most familiar with)
  • thedance 7y ago
    The one that people at Google who are the keepers of C++ code quality standards maintain themselves is Abseil. https://cs.opensource.google/kythe/kythe/+/master:external/com_google_absl/absl/ https://cs.opensource.google/kythe/kythe/+/master:external/c...
    • alexhutcheson 7y ago
      By necessity, Abseil is full of dark template magic that would very rarely be used elsewhere in the codebase. That's the point - it encapsulates a lot of useful abstractions and allows them to be used without the client code author thinking about the guts of the abstraction. But it makes it pretty unusual relative to typical Google C++.
      • thedance 7y ago
        True for much of it, but if you look at something like cord.h, it's almost free of template programs. Google C++ application code isn't all that spiffy, to be honest. I would say most of the code is dedicated to stuff that nobody outside of Google is going to care about. I think the base libraries are more interesting.
  • deadmutex 7y ago
    > study how the world's best engineers write code Note that you'd only be seeing the final result, not the whole process by studying source code. Also, I'd say definition of good code varies by domain.
    • samatman 7y ago
      Given how version control works, source code is the area of human endeavor for which your first sentence is the least true.
• jonathanoliver 7y ago
  I'm just gonna leave this here: https://killedbygoogle.com/ https://killedbygoogle.com/
  • wahern 7y ago
    Notably, they killed Google Code Search: https://en.wikipedia.org/wiki/Google_Code_Search https://en.wikipedia.org/wiki/Google_Code_Search IIRC, Google Code Search was the impetus for creating the RE2 library.
  • mav3rick 7y ago
    How is this contributing to the discussion. Do you feel cool now after a snarky comment ?
    • carapace 7y ago
      Do you?
      • mav3rick 7y ago
        I am pointing out what he did. And yes I do feel cool for pointing out something egregious.
    • CamperBob2 7y ago
      It's reasonable to warn less-experienced people about Google's historical behavior. People build their businesses on Google products and services like this, not realizing the 90%+ mortality rate over 5 years or so.
      • mav3rick 7y ago
        It's reasonable for a big company to try and fail. When they don't try anything 'they're not innovative'. When they do..'oh look they failed'.
        • 7y ago
• snek 7y ago
  Working with chromium/v8, I can honestly say google's code search infra is one of the most valuable resources available. I really hope they open source the backend at some point.
  • coderdd 7y ago
    The backend is open sourced, it is Kythe.io. It supports go, c++, java out of the box, for some definition of out of the box. Maybe even typescript. Also cross-references protobufs work generated code of you make the stars align ;) As for UI, treetide/underhood I mention elsewhere is the only open option now. But Kythe comes with command line utils and an API you can query directly as well. What is missing from the open source is a production-ready parallel serving table builder. There is one in golang which uses Apache Beam, but last time I checked the go workers are not well supported on the Flink runner. It didn't even work properly on the GCP runner. Hope this would change.
• Omnipresent 7y ago
  sidebar question - Anyone know how they've made the interaction/animation on this page [1] ? Feel like it is a great way to show lot of info in a concise way. [1] - https://opensource.google/projects/explore/featured https://opensource.google/projects/explore/featured
  • evere 7y ago
    Agreed, it is a very nice little interaction! It seems like they're animating the bubbles around a circle while randomly fluctuating the speed and radius at which they rotate. Clicking on a bubble centers it by setting the rotation radius to `0` and expanding the size. Would be interested to know how they expand the bubbles as your cursor moves closer.
• deleted 7y ago
  [deleted]
• ComputerGuru 7y ago
  Google used to have code search before (or at least far better than) anyone else did. Then they killed it.
  • akavel 7y ago
    Yeah, I'm super confused why suddenly now they provide a public code search service again. Uh, capricious overlords :/
  • zerd 7y ago
    I misread an thought it said Google Open Sources Code Search, hoping they open sourced https://en.wikipedia.org/wiki/Google_Code_Search https://en.wikipedia.org/wiki/Google_Code_Search
• dragonsh 7y ago
  If you want to do code search on your private git, mercurial, svn, cvs and other repositories try a fully open source opengrok (https://github.com/oracle/opengrok https://github.com/oracle/opengrok). It’s easy to self install and use, with good documentation with added bonus very fast.
• coderdd 7y ago
  Shameless plug: https://github.com/TreeTide/underhood https://github.com/TreeTide/underhood is a work in progress UI over Kythe indices (the same indices that power Code Search). If you already know how to index, this is a completely open source alternative, likely with less bells and whistles. I worked at Google and miss Code search. But I have lots of ideas as well how one can go beyond the status quo for code reading and debugging. Join if interested.
• MichaelMoser123 7y ago
  so far it doesn't seem to index a lot of stuff. I searched from some terms out of my kubernetes/openshift dependencies and it didn't find them. Is this correct?
  • deleted 7y ago
    [deleted]
• duckmysick 7y ago
  I really hate that some of the elements on the page are translated into a different language, seemingly based on my IP. When did it become acceptable to ignore my browser or my system language settings? The same thing happens on other Google services (like Google Groups), but I noticed this trend on other websites too.
  • neolithic 7y ago
    I find this extremely frustrating as well, across the various Google services I use.
  • Tenoke 7y ago
    I get this extremely annoying behaviour from many sites (sometimes without even an easy option to switch), including Google Search but not here.
    • duckmysick 7y ago
      For me, on the main page it's mostly tooltips (More Elements in the navbar, Help in the searchbar) and the blue Show Project link. It's worse on the project pages where the description is in English, but the entire table (including dates) is localized.
      • jsjddbbwj 7y ago
        Same here. Seems like a bug.
  • nolok 7y ago
    Have you tried changing your accept language ? IE, it is not based strictly on your IP, but also on what your browser asks for. I'm based in France, with a french IP, but my browser language header is set as ACCEPT-LANGUAGE en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7 And this page is fully in english.
    • duckmysick 7y ago
• Hitton 7y ago
  Nice. I'm grateful for this being posted on HN, because discoverability of that page seems to be zero (I couldn't find any link to it from opensource.google. It doesn't even have page title so googling it would be more complicated too.
• deleted 7y ago
  [deleted]
• gitgud 7y ago
  Sorry to be critical, but can someone explain to me the benefits of having code search this powerful? Surely code is easier to explore in an IDE which understands the context and dependencies of the project... This just seems like a glorified "find"
  • SkyBelow 7y ago
    I'm guessing the benefit is the ability to not have to bring up an IDE and download the code in cases where that isn't a quick option.
  • pmarin 7y ago
    It's handy for finding examples of real world use of some api. I usually use https://codesearch.debian.net/ https://codesearch.debian.net/
  • marcyb5st 7y ago
    If you think about Google's codebase size, an IDE wouldn't cut it. You could load and analyze dependencies/imports as you go, but that would make for a terrible user experience (think about IntelliJ indexing task every time you want to check the definition of something). Also, Code Search has baked in a lot of goodies. History layer, cross-references, call sites, ... and it's snappy. Moreover, is really well integrated with all the other internal tools used for coverage, code analysis, issue tracking, web text editor, ... . I think an IDE (like IntelliJ IDEA) can't reach that level of integration with several other systems unless you fully buy into the ecosystem a company like JetBrain proposes you (their issue tracker, their code review tool, ...). So, summarizing, it's a tool made by Googlers for Googlers' needs and it's amazing using it every day for all the above reasons.
  • smadge 7y ago
    You can search for non explicit dependencies. e.g. if you're removing a command line flag in a C++ binary. You can search for all uses of that flag for all users of your binary to make sure it is safe to remove.
• ptman 7y ago
  Check out Debian Code Search https://codesearch.debian.net/ https://codesearch.debian.net/
• excerionsforte 7y ago
  First impression is that it enables discoverability of code across the open sourced Google projects, but trying to find this page even on Google search is not a thing yet. Is that intentional?
• revertts 7y ago
  How strange - OpenDNS/Cisco Umbrella seems to flag the domain and gives me a 403 Forbidden.
• sproketboy 7y ago
  Did Not Connect: Potential Security Issue Firefox detected a potential security threat and did not continue to cs.opensource.google because this website requires a secure connection.
• whadar 7y ago
  For Java and JavaScript there's also codota.com
• rerx 7y ago
  This is very useful to read and search TensorFlow source code. It definitely beats Github for me.
• robinshen 7y ago
  Glad to see google open sourced this. I also implemented code search in my open source project OneDev (https://github.com/theonedev/onedev https://github.com/theonedev/onedev). To try it, please visit https://code.onedev.io/projects/android-framework-base/ https://code.onedev.io/projects/android-framework-base/. Press "t" for quick symbol search, and "v" for advanced search with regular expression support. You may also hover mouse over a symbol to find its declaration and occurrences.
• dunk010 7y ago
  Is this a frontend for Grok - the thing that Steve Yegge built?
• oscargrouch 7y ago
  Nice to see GN there. I wish more people knew about it. For me is as powerful as Bazel, but without the need for a JVM and all the insanity that comes with it in a desktop/dev environment. The syntax is great, powerful (insane customization) and together with Ninja theres nothing like it. Its in C++ and even being as powerful as Bazel, its a light, standalone library that can handle a huge amount of source code, dependencies, tools and configurations.
  • londons_explore 7y ago
    Having tried to battle GN configs... I don't agree. I was working on a big source tree and got frustrated that it kept rebuilding files that hasn't changed just because I switched git branches to look at one file, and then suddenly "Yay, another 18 hour full rebuild!". I tried to fix it and found there is no option to ignore file timestamps, and some guy has tried to patch it to do that[1]... But the patch requires putting an option in GN files which seems to break them wherever I put it... I tried to patch GN, but it wouldn't ever seem to pass that option through... Ended up patching Ninja to always have the option on, but then random other operations broke (like simple file copies). A day wasted, and problem not solved. Maybe my use case isn't common, or a bad workman blames his tools, but for me at least it wasn't a nice experience. [1]: https://github.com/ninja-build/ninja/issues/1459 https://github.com/ninja-build/ninja/issues/1459
• enitihas 7y ago
  Does anyone know the pros and cons of this vs something like Elastic search?
• vkaku 7y ago
  This name is misleading. The domain is okay. It should probably be 'Google Code Search'. I would have expected Google to come up with a search engine for all Open Source code otherwise.