3 ms·

Xoogler here. Internally as Google, you have a substantially power powerful version of Codesearch, which is semantic, i.e. it relies on actual binary artifacts

by Ayyar 8y ago

Xoogler here. Internally as Google, you have a substantially power powerful version of Codesearch, which is semantic, i.e. it relies on actual binary artifacts produced by builds to index code. All of the code base uses Blaze ( open sourced as Bazel) to express the rules. The version of code search here is cute, but it's still not a full blown structured semantic code search, which I really miss. Searching for and browsing code on Github feels underwhelming in comparison, because you wan't really find or explore code the way you think of it. You can see a demo of semantic Codesearch tech externalized here at: https://blog.bazel.build/2017/12/14/introducing-bazel-code-search.html#searching-the-bazel-codebase https://blog.bazel.build/2017/12/14/introducing-bazel-code-s... http://cs.chromium.org http://cs.chromium.org For example if you click on a header or a symbol, you can get proper cross references to a symbol. https://cs.chromium.org/chromium/src/pdf/document_loader_impl.cc?dr&g=0&l=43 https://cs.chromium.org/chromium/src/pdf/document_loader_imp... I really wish they add structured code search to Cloud Repos, even if it comes with some heavy restrictions such you must use Bazel as a build system. It would be a killer feature, and to my knowledge there is no equivalent to such a product. IDEs offer some similar overlapping functionality, though other code search engines still operate in terms of strings rather than actually real binary symbols ( I think Sourcegraph is trying here but that still does not seem to use some kind of accurate and structured information like you have in the code, it still seems not use build rules).

toeenredbluo 8y ago
How do you deal with things like system headers then? Force everyone to check in their OS (bye BSDs)? Assume everyone runs the same OS? How do you deal with compiler provided internals, where someone might be using clang which has behavior X, but gcc / icc / msvc doesn't? These problems are more prominent in C/C++, but are everywhere... Not easy... Google can probably check in everything for themselves... Probably not so easy for other companies? Maybe? Can Google solve this without sinking too much resources into it?
- Ayyar 8y ago
  My understanding of this is the Google approach is to have all builds in the cloud using Forge i.e. infrastructure that lets you cloud builds. This means you can do hermetic builds, and thousands of other engineers can reuse the build artifacts you produce. And in the particular context of tools such as code searching, your build artifacts update the code searching index pretty quickly post committing a change.
  - toeenredbluo 8y ago
    My point wasn't how do you build these things. It's how you index them.
sqs 8y ago
> I think Sourcegraph is trying here but that still does not seem to use some kind of accurate and structured information like you have in the code, it still seems not use build rules Sourcegraph does use language servers (same as your editor would use) to get precise definitions and references. For example ("find references" on some open-source code): Go: https://sourcegraph.com/github.com/theupdateframework/notary/-/blob/server/storage/tuf_store.go#L37:74&tab=references https://sourcegraph.com/github.com/theupdateframework/notary... TypeScript: https://sourcegraph.com/github.com/ReactiveX/rxjs/-/blob/src/internal/Observable.ts#L19:39&tab=references https://sourcegraph.com/github.com/ReactiveX/rxjs/-/blob/src... It understands many build tools, but not all. Go, JavaScript, TypeScript, Java, Python, and PHP support is all pretty solid. If you use the most common build tools for your language, it usually works well. It degrades un-gracefully if you use custom build tools or scripts, unfortunately, but we're working on making it support more build tools and customization. Sourcegraph development is open source; if you notice any problems, let us know in the issue tracker at https://github.com/sourcegraph/sourcegraph https://github.com/sourcegraph/sourcegraph.
- Ayyar 8y ago
  Thank you for the reply, appreciate you reading the comments so thoroughly. My top piece of feedback is considering some form of a strict mode, where your vision is not best effort support support for some legacy tool for a language, but fabulously amazing knock your socks off good support for users who take the time to invest in making their build friendly with your search index. It might seem tempting to go after the longest common denominator audience here, but personally I think it's far more valuable to show users how code search can be amazing if they do their part to invest in it, vs. code search being "kind of sort of useful" if you have a fallback. And maybe my Xoogler experiences are showing here, though Bazel / Kythe are in my opinion the perfect vehicles to realize that vision.
  - sqs 8y ago
    Very good points, thanks! We will be exposing a lot more configurability and extensibility in Sourcegraph in the next release (3.0-preview, and 3.0 more generally). I will add Kythe support to our roadmap and will think about how we can offer a strict mode soon. (We will need to offer that when Sourcegraph supports "write" functionality like automated refactoring, but it would be great to have it sooner.)
ehsankia 8y ago
It's honestly one of the greatest advantages of working in google3. GitHub search is honestly pitiful next to it.

by Ayyar 8y ago

toeenredbluo 8y ago

How do you deal with things like system headers then? Force everyone to check in their OS (bye BSDs)? Assume everyone runs the same OS? How do you deal with compiler provided internals, where someone might be using clang which has behavior X, but gcc / icc / msvc doesn't? These problems are more prominent in C/C++, but are everywhere... Not easy... Google can probably check in everything for themselves... Probably not so easy for other companies? Maybe? Can Google solve this without sinking too much resources into it?

Ayyar 8y ago
My understanding of this is the Google approach is to have all builds in the cloud using Forge i.e. infrastructure that lets you cloud builds. This means you can do hermetic builds, and thousands of other engineers can reuse the build artifacts you produce. And in the particular context of tools such as code searching, your build artifacts update the code searching index pretty quickly post committing a change.
- toeenredbluo 8y ago
  My point wasn't how do you build these things. It's how you index them.

sqs 8y ago

> I think Sourcegraph is trying here but that still does not seem to use some kind of accurate and structured information like you have in the code, it still seems not use build rules Sourcegraph does use language servers (same as your editor would use) to get precise definitions and references. For example ("find references" on some open-source code): Go: https://sourcegraph.com/github.com/theupdateframework/notary/-/blob/server/storage/tuf_store.go#L37:74&tab=references https://sourcegraph.com/github.com/theupdateframework/notary... TypeScript: https://sourcegraph.com/github.com/ReactiveX/rxjs/-/blob/src/internal/Observable.ts#L19:39&tab=references https://sourcegraph.com/github.com/ReactiveX/rxjs/-/blob/src... It understands many build tools, but not all. Go, JavaScript, TypeScript, Java, Python, and PHP support is all pretty solid. If you use the most common build tools for your language, it usually works well. It degrades un-gracefully if you use custom build tools or scripts, unfortunately, but we're working on making it support more build tools and customization. Sourcegraph development is open source; if you notice any problems, let us know in the issue tracker at https://github.com/sourcegraph/sourcegraph https://github.com/sourcegraph/sourcegraph.

Ayyar 8y ago
Thank you for the reply, appreciate you reading the comments so thoroughly. My top piece of feedback is considering some form of a strict mode, where your vision is not best effort support support for some legacy tool for a language, but fabulously amazing knock your socks off good support for users who take the time to invest in making their build friendly with your search index. It might seem tempting to go after the longest common denominator audience here, but personally I think it's far more valuable to show users how code search can be amazing if they do their part to invest in it, vs. code search being "kind of sort of useful" if you have a fallback. And maybe my Xoogler experiences are showing here, though Bazel / Kythe are in my opinion the perfect vehicles to realize that vision.
- sqs 8y ago
  Very good points, thanks! We will be exposing a lot more configurability and extensibility in Sourcegraph in the next release (3.0-preview, and 3.0 more generally). I will add Kythe support to our roadmap and will think about how we can offer a strict mode soon. (We will need to offer that when Sourcegraph supports "write" functionality like automated refactoring, but it would be great to have it sooner.)

ehsankia 8y ago

It's honestly one of the greatest advantages of working in google3. GitHub search is honestly pitiful next to it.