3 ms·

I think that there should be some sort of standardized, language-agnostic metadata format for semantically indexing a codebase. It could include e.g type inform

by dymk 4y ago

I think that there should be some sort of standardized, language-agnostic metadata format for semantically indexing a codebase. It could include e.g type information for expressions or declared variables (for languages that infer types), and an index of symbols and how they're connected. This metadata file would be generated by a language-specific tool. For instance, Cargo could generate it for Rust projects, ctags/cmake for C / C++, Sorbet for Ruby. Then a service like Github / Gitlab / your own homegrown code viewer could provide things like "Show type", "Jump to source" without ever needing to build a language-specific parser or interpreter (which seems arbitrarily difficult, most build systems provide escape hatches, so you can't assume much about project structure). Basically, LSP as a static file in a standard format that tools can read to understand a codebase, without needing to model the language's semantics.

maxov 4y ago
SemanticDB (https://scalameta.org/docs/semanticdb/guide.html https://scalameta.org/docs/semanticdb/guide.html) is a protobuf-based file format that does almost exactly this for JVM languages, primarily Scala (I was a contributor a while back). It is used to build an intelligent online code browser, as the backend for a language server, and to do intelligent refactorings. I think a language-agnostic semantic metadata format is a good idea, but requires a lot of compromise. ctags partially does this, but only to a very coarse level (mostly definitions and references). I think some ctags implementations also define 'extension fields' that could be used to give type information, but I don't know how/if these are used in practice. SemanticDB is extremely fine-grained, but highly specialized to JVM languages and type systems that are designed to work with the JVM. Finding a common set of semantic features that can be used across languages and type systems that is fine-grained enough to be more useful than ctags sounds very difficult to me.
- dymk 4y ago
  I think simple things like "go to reference" or "show type" would be sufficient for 95% of usecases. But if you split languages up into a few different categories (maybe along the lines of Algol-like vs Lisp-like), and were flexible with extensions, I'd imagine we'd see some common patterns emerging, and clients would take advantage of that. Best effort is probably good enough to greatly improve the ergonomics of search.
slimsag 4y ago
This is pretty much exactly what we've built at Sourcegraph. Microsoft had introduced (but pretty much abandoned before it even started) LSIF, a static index format for LSP servers which encodes in detail all possible LSP requests/responses, effectively. We took that torch and carried it forward, building the spiritual successor called SCIP[0]. It's language agnostic, we have indexers for quite a few languages already, and we genuinely intend for it to be vendor neutral / a proper OSS project[1]. [0] https://about.sourcegraph.com/blog/announcing-scip https://about.sourcegraph.com/blog/announcing-scip [1] https://github.com/sourcegraph/scip https://github.com/sourcegraph/scip
- dymk 4y ago
  This is pretty much exactly what I'd imagined, very cool. Is the plan to open source all SCIP indexers?
  - typesanitizer 4y ago
    All of our SCIP indexers are open-source: scip-java (for Java, Kotlin and Scala), scip-typescript (for TypeScript and JavaScript), scip-python, scip-ruby, scip-go and scip-clang (for C and C++). There are also some community-maintained OSS SCIP indexers. - rust-analyzer: https://sourcegraph.com/github.com/rust-lang/rust-analyzer/-/blob/crates/rust-analyzer/src/cli/scip.rs https://sourcegraph.com/github.com/rust-lang/rust-analyzer/-... - scip-zig: https://github.com/zigtools/scip-zig https://github.com/zigtools/scip-zig

by dymk 4y ago

maxov 4y ago

SemanticDB (https://scalameta.org/docs/semanticdb/guide.html https://scalameta.org/docs/semanticdb/guide.html) is a protobuf-based file format that does almost exactly this for JVM languages, primarily Scala (I was a contributor a while back). It is used to build an intelligent online code browser, as the backend for a language server, and to do intelligent refactorings. I think a language-agnostic semantic metadata format is a good idea, but requires a lot of compromise. ctags partially does this, but only to a very coarse level (mostly definitions and references). I think some ctags implementations also define 'extension fields' that could be used to give type information, but I don't know how/if these are used in practice. SemanticDB is extremely fine-grained, but highly specialized to JVM languages and type systems that are designed to work with the JVM. Finding a common set of semantic features that can be used across languages and type systems that is fine-grained enough to be more useful than ctags sounds very difficult to me.

dymk 4y ago
I think simple things like "go to reference" or "show type" would be sufficient for 95% of usecases. But if you split languages up into a few different categories (maybe along the lines of Algol-like vs Lisp-like), and were flexible with extensions, I'd imagine we'd see some common patterns emerging, and clients would take advantage of that. Best effort is probably good enough to greatly improve the ergonomics of search.

slimsag 4y ago

This is pretty much exactly what we've built at Sourcegraph. Microsoft had introduced (but pretty much abandoned before it even started) LSIF, a static index format for LSP servers which encodes in detail all possible LSP requests/responses, effectively. We took that torch and carried it forward, building the spiritual successor called SCIP[0]. It's language agnostic, we have indexers for quite a few languages already, and we genuinely intend for it to be vendor neutral / a proper OSS project[1]. [0] https://about.sourcegraph.com/blog/announcing-scip https://about.sourcegraph.com/blog/announcing-scip [1] https://github.com/sourcegraph/scip https://github.com/sourcegraph/scip

dymk 4y ago
This is pretty much exactly what I'd imagined, very cool. Is the plan to open source all SCIP indexers?
- typesanitizer 4y ago
  All of our SCIP indexers are open-source: scip-java (for Java, Kotlin and Scala), scip-typescript (for TypeScript and JavaScript), scip-python, scip-ruby, scip-go and scip-clang (for C and C++). There are also some community-maintained OSS SCIP indexers. - rust-analyzer: https://sourcegraph.com/github.com/rust-lang/rust-analyzer/-/blob/crates/rust-analyzer/src/cli/scip.rs https://sourcegraph.com/github.com/rust-lang/rust-analyzer/-... - scip-zig: https://github.com/zigtools/scip-zig https://github.com/zigtools/scip-zig