4 ms·

> code search in combination with a monorepo and the somewhat hermetic universe which is Google's code base provides immediate access to all uses of a library.

> code search in combination with a monorepo and the somewhat hermetic universe which is Google's code base provides immediate access to all uses of a library. You can see what worked well and what didn't, and it enables effective refactorings. That also means when writing a client, you can quickly figure out from other client code how stuff is supposed to be used. ... If anything like this were possible for open source (indexing code that depends on a library/API, across subrepos in any version control system, in a way that gets near complete coverage), it would enable similar possibilities of systematic improvement. gets me thinking — this is totally doable for open source code. someone just needs to build some giant indexes across git repos + package repos (maven/npm/pip/etc)... Then write plugins for VScode/intellij, and when developing you can right-click a symbol and “Find uses in open source code” Does something like this already exist?

rockwotj 6y ago
I don't think so. When you consider that you'd have to index every library and every version of it. That's a huge index. Google has the "live at HEAD" [1] mentality so in reality you don't need as much. I don't remember if xrefs for old versions of the code we're retained. 1: https://abseil.io/about/philosophy#upgrade-support https://abseil.io/about/philosophy#upgrade-support
- cobbzilla 6y ago
  It would be huge but storage is cheap, you could build in phases based on popularity. Maybe paid subscriptions for the full index? But yeah, it could get expensive to keep the indexes current as things grow.
  - rockwotj 6y ago
    Yeah but you also have to understand the dependency management systems for every open source project on GitHub to get the right dependency, which is a sisyphean task. Google has the advantage there is only one way to build stuff.
    - cobbzilla 6y ago
      There are only so many package managers out there. Most languages have a "preferred" one which makes things even easier. Stuff built with ad-hoc systems might be more difficult, but I don't think that's the norm.
timetc 6y ago
A while ago I came across Nx (https://nx.dev/ https://nx.dev/) from Nrwl, which sounds a lot like this. It's open source and has a lot of tooling for managing monorepo's. I think they even worked with Google for it. Altough I think the main focus was JS projects (as far as I know).
vageli 6y ago
> gets me thinking — this is totally doable for open source code. someone just needs to build some giant indexes across git repos + package repos (maven/npm/pip/etc)... Sounds a lot like sourcegraph.
breck 6y ago
GitHub is moving in this direction (they now show interpackage dependencies at the package level, and intra dependencies at the symbol level. However, neither works great—maybe 30%-80% coverage?, and far from "in a way that gets near complete coverage" that the parent stated about the benefit of Google's monorepo. It may be a case where you need 90 or 99 or even higher coverage before it becomes a game changer. But I think GitHub is headed there quickly.
- maxekman 6y ago
  Yes, definitely. And what they already have is often very useful, I find myself spontaneously traversing code bases by going to definitions and references. Quite an impressive (and non-intrusive) addition from a UI perspective.
- dcreager 6y ago
  Hi there! I manage the GitHub team responsible for the Code Nav part of this. Re 30-80% coverage, can you elaborate on the biggest missing piece for you? Is it the number of languages that are covered, or too many false positives in the results we show? (Or something else?) For the latter, it's a known limitation in that we're only doing "fuzzy" or "c-tags-like" Code Nav — defs and refs are only matched by their textual unqualified symbol names. For some languages that's not a big deal, because symbol names tend to be unique. But for a language like Go, you might have dozens (or hundreds!) of methods named "Parse", and we don't currently distinguish those in our Jump to Def UI. That's something we're actively working on, though! One benefit to the current fuzzy approach is that it's incremental (we only have to analyze changed files in a commit), and does not depend on tapping into a build or CI process. That makes it much faster — we have a long-running indexer service live and hot, and typically have new commits indexed and live in the UI within 1-5 seconds of receiving a push. And it requires no configuration on the part of the repo owner — no need to tell us how to build your project, or to configure an Actions workflow to generate the data. That makes it easier for us to support entire ecosystems when we roll out support for a new language. We've also tried to make it fairly easy for external communities to add support for their languages — this is all driven by the open-source tree-sitter parsing framework. This file, for instance, is where the symbol extraction rules for Go are defined: https://github.com/tree-sitter/tree-sitter-go/blob/master/queries/tags.scm https://github.com/tree-sitter/tree-sitter-go/blob/master/qu... We've been working on a new approach to generate more precise symbol data, while keeping those incremental and zero-config properties. It's not ready for public launch, yet, but it's close! For those who want to learn more, I gave a talk at FOSDEM 2020 (back in the Before Times) going into more detail on some of these constraints and design decisions. (Though please note that this talk references a legacy way of defining symbol extraction rules, which relied on writing code in our open-source Semantic project.) https://dcreager.net/talks/2020-fosdem/ https://dcreager.net/talks/2020-fosdem/
coredog64 6y ago
There are a couple of search engines that do this. Sometimes when I get stuck on a underdocumented Python library, the big G will take me to one via a search on a method or constant. I can then see the code from other open source libraries/projects. I don’t have any links saved, since they feel kinda sketch (ads all over the place). But it has been very helpful, unlike searching within GitHub for a string that I know is in the repo.
beliu 6y ago
Author of the post here. We're actively working on this at Sourcegraph. Up until recently, we've focused on working well on large private codebases, but looking forward, we have two big efforts that will make this a reality on sourcegraph.com: 1. Vastly expanding the size of our global search index to cover every public repository on github.com, gitlab.com, and bitbucket.org. 2. Enabling LSIF indexing (https://lsif.dev https://lsif.dev) in every major language for compiler-accurate code navigation (go-to-def, find-refs) across repository/dependency boundaries. The latter is already working on a subset of languages for private Sourcegraph instances, and we want to scale it to the entire open-source world. We think a single search box that covers all the visible code in the world and allows you to seamlessly walk the reference graph is super super powerful and someday will be a thing that most developers use every day.
- loosetypes 6y ago
  Don’t get me wrong, I think this is very cool, but is there a way to opt out aside from making a repo private? Do OSS licenses distinguish between corporate usage of repository source as code vs as data.
  - jkaplowitz 6y ago
    Licenses that comply with the Open Source Definition or any of the similar definition from the free-as-in-freedom software community inherently must allow this as a legal use. That said, the community often takes authors' wishes into account. Out of curiosity, why would you want to restrict this? It's not like Sourcegraph has any exclusivity to the concept or the ability to implement it.
- jkaplowitz 6y ago
  Neat. Are you considering adding other large source repositories such as the Debian / Ubuntu / Fedora / RHEL / CentOS source package archives? At least the main section of Debian should offer reasonably high assurances that using the sources in that way is legal.

rockwotj 6y ago

I don't think so. When you consider that you'd have to index every library and every version of it. That's a huge index. Google has the "live at HEAD" [1] mentality so in reality you don't need as much. I don't remember if xrefs for old versions of the code we're retained. 1: https://abseil.io/about/philosophy#upgrade-support https://abseil.io/about/philosophy#upgrade-support

cobbzilla 6y ago
It would be huge but storage is cheap, you could build in phases based on popularity. Maybe paid subscriptions for the full index? But yeah, it could get expensive to keep the indexes current as things grow.
- rockwotj 6y ago
  Yeah but you also have to understand the dependency management systems for every open source project on GitHub to get the right dependency, which is a sisyphean task. Google has the advantage there is only one way to build stuff.
  - cobbzilla 6y ago
    There are only so many package managers out there. Most languages have a "preferred" one which makes things even easier. Stuff built with ad-hoc systems might be more difficult, but I don't think that's the norm.

timetc 6y ago

A while ago I came across Nx (https://nx.dev/ https://nx.dev/) from Nrwl, which sounds a lot like this. It's open source and has a lot of tooling for managing monorepo's. I think they even worked with Google for it. Altough I think the main focus was JS projects (as far as I know).

vageli 6y ago

> gets me thinking — this is totally doable for open source code. someone just needs to build some giant indexes across git repos + package repos (maven/npm/pip/etc)... Sounds a lot like sourcegraph.

breck 6y ago

GitHub is moving in this direction (they now show interpackage dependencies at the package level, and intra dependencies at the symbol level. However, neither works great—maybe 30%-80% coverage?, and far from "in a way that gets near complete coverage" that the parent stated about the benefit of Google's monorepo. It may be a case where you need 90 or 99 or even higher coverage before it becomes a game changer. But I think GitHub is headed there quickly.

maxekman 6y ago
Yes, definitely. And what they already have is often very useful, I find myself spontaneously traversing code bases by going to definitions and references. Quite an impressive (and non-intrusive) addition from a UI perspective.
dcreager 6y ago
Hi there! I manage the GitHub team responsible for the Code Nav part of this. Re 30-80% coverage, can you elaborate on the biggest missing piece for you? Is it the number of languages that are covered, or too many false positives in the results we show? (Or something else?) For the latter, it's a known limitation in that we're only doing "fuzzy" or "c-tags-like" Code Nav — defs and refs are only matched by their textual unqualified symbol names. For some languages that's not a big deal, because symbol names tend to be unique. But for a language like Go, you might have dozens (or hundreds!) of methods named "Parse", and we don't currently distinguish those in our Jump to Def UI. That's something we're actively working on, though! One benefit to the current fuzzy approach is that it's incremental (we only have to analyze changed files in a commit), and does not depend on tapping into a build or CI process. That makes it much faster — we have a long-running indexer service live and hot, and typically have new commits indexed and live in the UI within 1-5 seconds of receiving a push. And it requires no configuration on the part of the repo owner — no need to tell us how to build your project, or to configure an Actions workflow to generate the data. That makes it easier for us to support entire ecosystems when we roll out support for a new language. We've also tried to make it fairly easy for external communities to add support for their languages — this is all driven by the open-source tree-sitter parsing framework. This file, for instance, is where the symbol extraction rules for Go are defined: https://github.com/tree-sitter/tree-sitter-go/blob/master/queries/tags.scm https://github.com/tree-sitter/tree-sitter-go/blob/master/qu... We've been working on a new approach to generate more precise symbol data, while keeping those incremental and zero-config properties. It's not ready for public launch, yet, but it's close! For those who want to learn more, I gave a talk at FOSDEM 2020 (back in the Before Times) going into more detail on some of these constraints and design decisions. (Though please note that this talk references a legacy way of defining symbol extraction rules, which relied on writing code in our open-source Semantic project.) https://dcreager.net/talks/2020-fosdem/ https://dcreager.net/talks/2020-fosdem/

coredog64 6y ago

There are a couple of search engines that do this. Sometimes when I get stuck on a underdocumented Python library, the big G will take me to one via a search on a method or constant. I can then see the code from other open source libraries/projects. I don’t have any links saved, since they feel kinda sketch (ads all over the place). But it has been very helpful, unlike searching within GitHub for a string that I know is in the repo.

beliu 6y ago

Author of the post here. We're actively working on this at Sourcegraph. Up until recently, we've focused on working well on large private codebases, but looking forward, we have two big efforts that will make this a reality on sourcegraph.com: 1. Vastly expanding the size of our global search index to cover every public repository on github.com, gitlab.com, and bitbucket.org. 2. Enabling LSIF indexing (https://lsif.dev https://lsif.dev) in every major language for compiler-accurate code navigation (go-to-def, find-refs) across repository/dependency boundaries. The latter is already working on a subset of languages for private Sourcegraph instances, and we want to scale it to the entire open-source world. We think a single search box that covers all the visible code in the world and allows you to seamlessly walk the reference graph is super super powerful and someday will be a thing that most developers use every day.

loosetypes 6y ago
Don’t get me wrong, I think this is very cool, but is there a way to opt out aside from making a repo private? Do OSS licenses distinguish between corporate usage of repository source as code vs as data.
- jkaplowitz 6y ago
  Licenses that comply with the Open Source Definition or any of the similar definition from the free-as-in-freedom software community inherently must allow this as a legal use. That said, the community often takes authors' wishes into account. Out of curiosity, why would you want to restrict this? It's not like Sourcegraph has any exclusivity to the concept or the ability to implement it.
jkaplowitz 6y ago
Neat. Are you considering adding other large source repositories such as the Debian / Ubuntu / Fedora / RHEL / CentOS source package archives? At least the main section of Debian should offer reasonably high assurances that using the sources in that way is legal.