3 ms·
The kind of cross-reference SourceGraph provides is a must have for me after a few years of using a tool like it to understand big projects (I don't use SourceG
by emmanueloga_ 8y ago
The kind of cross-reference SourceGraph provides is a must have for me after a few years of using a tool like it to understand big projects (I don't use SourceGraph, I use Grok/CodeSearch).
Google open sourced a version of Grok some time ago (https://kythe.io/ https://kythe.io/). [1]
A quick search also found this `OpenGrok` thing which I assume must be yet another generic indexer [2].
I wonder if there are other popular tools out there providing generic cross-referencing of code bases?
<SourceGraph>
I was trying to find the equivalent of this Kythe page (how to write an indexer)[6] on SourceGraph docs, after 20 minutes I was still confused. I wish there was more clarity on the different pieces of the system. At first I thought SourceGraph could create a cross reference index from an LSP server, but I guess makes more sense it is the order way around? (SG can talk to text editors via LSP).
It must be a lot of work to make integrate any single language with both SG or Kythe and LSP, perhaps there's some tools or methodologies to make this easier?
<Tangent>
I feel like language designers don't always think about building their implementations with ease of integration as a goal. I remember wanting to play with doing some AST transformations with both TypeScript and Dart (independently) and finding both languages made the task so hard and tedious. Mandatory remark: this is so easy with Lisp derivatives :-)
Things that a good language+ecosystem should have, IMO:
* Easy to retrieve and manipulate CST (for things like writing refactoring and querying tools, writing code formatters/pretty printers, etc). Many languages provide the syntax tree say, without comments, so it is hard to write a pretty printer.
* Documentation generator with cross-reference navigation.
* Play along nicely with generic build tools like Ninja [3].
* Good debugger and value-pretty-printer support (so I can just drop a breakpoint and print anything easily). And some sort of API to use the debugger programmatically or connect to a remote process.
* An LSP implementation [4], although it seems pretty much every popular language is getting one these days!
1: https://en.wikipedia.org/wiki/Google_Kythe https://en.wikipedia.org/wiki/Google_Kythe
2: http://oracle.github.io/opengrok/ http://oracle.github.io/opengrok/
3: https://ninja-build.org/ https://ninja-build.org/
4: https://langserver.org/ https://langserver.org/
5: https://about.sourcegraph.com/docs/code-intelligence/ https://about.sourcegraph.com/docs/code-intelligence/
6: https://kythe.io/docs/schema/writing-an-indexer.html https://kythe.io/docs/schema/writing-an-indexer.html
<edit: line breaks>
- tmerr 8y agoSome comparisons based on my reading so far: Adding a language: Kythe: a language implementation provides some structured graph SourceGraph: a language implementation is an LSP server Backend logic: Kythe: has common logic for querying for relationships SourceGraph: simply proxies LSP requests through to the appropriate language server. Terminology: Kythe: An indexer is a language implementation SourceGraph: The index means something completely different, it's what keeps track of git repositories. SourceGraph Architecture: https://github.com/sourcegraph/sourcegraph/blob/master/docs/architecture.md https://github.com/sourcegraph/sourcegraph/blob/master/docs/... Adding lang server to SourceGraph (probably somewhat stale) https://github.com/sourcegraph/sourcegraph/blob/master/xlang/xlang.notes.md#hooking-up-a-buildlanguage-server-to-sourcegraph https://github.com/sourcegraph/sourcegraph/blob/master/xlang...
- keegancsmith 8y agoWe (I work at Sourcegraph) actually started off with a similar model to kythe: https://srclib.org/ https://srclib.org/ This project existed before kythe was published, but was based on the ideas talked about pre-dating kythe. If I remember correctly Steve Yegge had some blogpost about solving the MxN problem, which turned into him doing grok at Google. We ended up switching away from that model for a few reasons: - At the time it was very costly to essentially index the whole OSS world, when most commit indexes would never be read. - It was slow to index a codebase for a commit, and most of the work was wasted since a developer would often only look at a handful of files. - Getting incremental indexing working usually required pretty deep integration into the build tool, so was a lot of work per language which didn't scale. - A lot of tools using an "indexed" model end up only indexing the master/trunk branch, and you don't get your code intelligence features for PRs/etc. At some point LSP came onto the scene. We were early adopters making a bet that this would take off. It has, and the list of community created LSP servers is large: https://langserver.org/ https://langserver.org/ It also allowed us to switch from an upfront indexing model, to a model which just encodes user intent. So the underlying LSP server can be as lazy as it wants to be with respect to how it responds to the user == increased perf and reduced resources. Things like cross-repo references doesn't come with LSP. Many LSP servers assume the user sets the build up correctly. There are also quite a few more assumptions LSP authors make which don't easily translate into an automated server environment. So we have done quite a bit of work to smooth that over / contributed some LSP servers for popular languages. Seems like you are digging in so you might have more questions. I'll try keep track of this thread, but also feel free to email me keegan at sourcegraph.com or file issues/questions on our repo.