9 ms·

Glean – System for collecting, deriving and querying facts about source code

213 points by dons 5y ago

z3t4 5y ago
This seems very interesting, would love to see more alternatives to TreeSitter and microsoft LSP - what makes those hard to use is lack of examples and tutorials. So I hope tbere will be examples and tutorials. For example: How do you find all variables in scope when the text cursor is on line x and col y in /file/path/file.js
log101 5y ago
I didn't understand what it does
- sealeck 5y ago
  It allows you to use a query language (think SQL) to analyze source code.
  - tyingq 5y ago
    The docs seem to get to queries here: https://glean.software/docs/angle/guide https://glean.software/docs/angle/guide
soonnow 5y ago
I had a look at the site and it seems to be parsing source code in multiple languages and storing the parsed "syntax trees" into a database for querying. I would love to know what the usecase for this tool is aside from maybe being a source for presentations? (We have 5 million if statements). How can this be used to improve code quality or any other aspect of the code lifecycle? Or is it solving problems in a completely different problem area?
- lazamar 5y ago
  Glean is focused on storing and querying data about the code. The idea is that you have your own program to collect that data, then you use Glean to store that compactly and to have snappy queries. You would create entries like "this is a declaration of X", "this is a use of X". Then you can query things like "give me all uses of X" in sub-millisecond time. You hook that up to an LSP server then you get almost zero-cost find-references, jump-to-definition, etc. The snappy queries also mean it becomes possible to perform whole codebase (and cross-language) analysis. That is, answering questions like "what code is not referenced from this root?", "does this Haskell function use anything that calls malloc?" (analysis through the ffi barrier). One can also attach all kinds of information from different sources to code entities, not only things derived from the source itself. You add things like run-time costs, frequency of use, common errors, etc, and an LSP server could make all of it available right in your editor. For very large or complex codebases, where it is just too expensive or too complicated to calculate this information locally a system like this becomes very useful.
  - soonnow 5y ago
    > For very large or complex codebases, where it is just too expensive or too complicated to calculate this information locally a system like this becomes very useful. Thanks I guess I get it now. But to enable this functionality you'd need to have some form of frontend or integration into the existing build lifecycle? Or IDE integration I guess.
  - scns 5y ago
    Oh wow, mindblowing stuff. Glad to see tech like this being open sourced, fuels the imagination about possible future scenarios. Do you use it on the Linux Kernel?
    - minxomat 5y ago
      A comparable, powerful system (CodeQL) was used recently on the kernel[1] and Chrome. You can learn more about it here: https://codeql.github.com/docs/codeql-overview/about-codeql/ https://codeql.github.com/docs/codeql-overview/about-codeql/ (disclosure: I work at GH on CQL) [1] https://pwning.systems/posts/sequoia-variant-analysis/ https://pwning.systems/posts/sequoia-variant-analysis/
      - X6S1x6Okd1st 5y ago
        Oof on the terms & conditions: https://securitylab.github.com/tools/codeql/license/ https://securitylab.github.com/tools/codeql/license/
  - gigatexal 5y ago
    Thank you for this summary I was unsure of how this is really useful. That before step is missing I think.
dons 5y ago
We use this to power things like find-references or jump-to-def, "symbol search" and autocomplete, or more complicated code queries and analysis (even across languages). Imagine rich LSPs without a local checkout, web-based code queries, or seeding fuzzers and static analyzers with entry points in code. Our focus has been on very large scale, multi-language code indexing, and then low latency (e.g. hundreds of micros) query times, to drive highly interactive developer workflows.
- the_duke 5y ago
  This is really cool. Seems like there are only indexers for Flow and Hack though. Will there be more indexers built by Facebook, or will it rely on community contributions?
  - dons 5y ago
    A bit of both I think.
  - simonmar 5y ago
    There will be more indexers: we have Python, C++/Objective C, Rust, Java and Haskell. It's just a case of getting them ready to open source. You can see the schemas for most of these already in the repo: https://github.com/facebookincubator/Glean/tree/main/glean/schema/source https://github.com/facebookincubator/Glean/tree/main/glean/s...
- soonnow 5y ago
  Does that mean you are using the shell or how is it used to enable these functionalities?
  - dons 5y ago
    Most clients hit the Glean server via the network (thrift/JSON) and then mostly via language bindings to the Glean query language, Angle. The shell is more for debugging/exploration. Imagine an IDE plugin that queries Glean over the network for symbol information about the current file, then shows that on hover. That sort of thing.
booleandilemma 5y ago
The very first page of the site should have examples of what you can do with it.
- _tom_ 5y ago
  It’s such a shame to get this much traffic, and have it almost all bounce, because there’s no description of WHY someone would care.
enjikaka 5y ago
Ew, Facebook.
doddsiedodds 5y ago
An excellent talk by Simon Marlow on Glean here: https://youtu.be/-OPN7QPsYKE https://youtu.be/-OPN7QPsYKE
- simonmar 5y ago
  I should point out that Glean has evolved quite a bit since that talk!
coderdd 5y ago
Great to see this space moving! Any pointers on diff vs Kythe? Kythe has a mostly fixed schema, for one. One of the pain points using Kythe is wiring up the indexer to the build system. Would Glean indexers be easier to wire up for the common cases? Other is the index post-processing, which is not very scalable in the open source version (due to go-beam having rough Flunk support, for example). Third, how does it link up references across compilation units? Is it heuristic, or relies on unique keys from indexers matching? Or across languages?
- simonmar 5y ago
  Kythe has one schema, whereas with Glean each language has its own schema with arbitrary amounts of language-specific detail. You can get a language-agnostic view by defining an abstraction layer as a schema. Our current (work in progress) language-agnostic layer is called "codemarkup" https://github.com/facebookincubator/Glean/blob/main/glean/schema/source/codemarkup.angle https://github.com/facebookincubator/Glean/blob/main/glean/s... For wiring up the indexer, there are various methods, it tends to depend very much on the language and the build system. For Flow for example, Glean output is just built into the typechecker, you just run it with some flags to spit out the Glean data. For C++, you need to get the compiler flags from the build system to pass to the Clang frontend. For Java the indexer is a compiler plugin; for Python it's built on libCST. Some indexers send their data directly to a Glean server, others generate files of JSON that get sent using a separate command-line tool. References use different methods depending on the language. For Flow for example there is a fact for an import that matches up with a fact for the export in the other file. For C++ there are facts that connect declarations with definitions, and references with declarations.
  - Game_Ender 5y ago
    Is there an example of using the C++ indexer? I saw hack and JS on your site but missed C++ (Python would also be amazing!).
    - simonmar 5y ago
      We want to open-source the C++ and Python indexers but they're not ready yet - we have to separate them from internal build-system-specific bits.
erlich 5y ago
I can't believe Facebook hasn't canned Flowtype yet and moved to TypeScript. They will have to do it eventually.
- muglug 5y ago
  I'm not sure you understand the scale at which Facebook operates. They don't have to do anything. As long as billions of people keep using Facebook they can maintain their own static analysis tooling for Javascript for as long as they want.
  - scns 5y ago
    You do have a point, a rewrite on that scale would be a colossal waste of manyears/$$. Your delivery could be nicer though.
- ctvo 5y ago
  Doesn't look like they're stopping their use of Hack either. Eventually is a long time so you're right.
- wingspan 5y ago
  The problem is that TypeScript does not scale to the size of the giant monorepo at Facebook, with hundreds of thousands, if not millions of files. Since they aren't organized into packages, it is just one giant flat namespace (any JS file can import any other JS file by the filename). It is pretty amazing to change a core file and see type errors across the entire codebase in a few seconds. The main way to scale in TypeScript is Project References, which don't work when you haven't separated your code into packages. (Worked at Facebook until June 2021).
simonw 5y ago
Feature request: a live demo! I would love to try out the web interface described at https://glean.software/docs/trying https://glean.software/docs/trying without pulling down a 7GB Docker image first.
- deleted 5y ago
  [deleted]
- jamessb 5y ago
  Even just a short video of someone using the web interface would be helpful.
da39a3ee 5y ago
I was recently looking for a library that takes a few lines of source code as input, and predicts the programming language as output. That seems like a very tractable machine learning problem, yet all I could find was a single python library which looks nice, but doesn't have much adoption, and requires installing the entirety of tensorflow despite the fact that users just want a trained model and a predict() function. Why doesn't a popular library like this exist?
- jamessb 5y ago
  GitHub's linguist library can be used to identify the programming language of a single file (edit: or of a whole project): https://github.com/github/linguist#single-file https://github.com/github/linguist#single-file
  - da39a3ee 5y ago
    Thanks! My searches completely failed to find that. I can’t use it as a ruby library, but perhaps I can pull out the heuristics.yml and the naive bayes classifier weights to use in another language.
aabaker99 5y ago
Cool! I would love to play around with this. How do I write a schema and indexer for my favorite programming language that isn't currently (and won't be) supported with official releases? For Schemas, [1] says to modify (or base new ones off) these: https://github.com/facebookincubator/Glean/tree/main/glean/schema/source https://github.com/facebookincubator/Glean/tree/main/glean/s... For Indexers, it's a little less clear but it looks like I need to write my own type checker? [1] https://glean.software/docs/schema/workflow https://glean.software/docs/schema/workflow
marcodiego 5y ago
Is it a modern cscope?
rognjen 5y ago
Meta: should the ?open tracking part of the URL be removed?
ctvo 5y ago
Great job with this. What's your roadmap for releasing some of the tooling for editor integration? Really, the question is should I build something or wait a few weeks?
Grimm1 5y ago
Very cool! How does this differ algorithmically from the trigram based search that everything uses from google code search from like 20 years ago? And continuing off of that theme in practical terms how does it stand up against zoekt? I’m curious because zoekt is kind of slow when it comes to ingesting large amounts of code like all of the publicly available code on GitHub The few people using that commercially have basically had to spend a lot of time rewriting parts of it to make their goal of public codesearch for all attainable. I and a few people I know are pretty convinced that there are better and easier ways / technologies to make that happen.
ExtraE 5y ago
What, uh, is this? This is a space that I’m not familiar with and the linked site doesn’t make it super clear.
balddenimhero 5y ago
Datalog-ish query languages sure is a fun area to be working in. Such DSLs exist for various domains and, like Semmle's codeQL or the more academic Soufflé, Glean focuses on the domain of programming languages. Glean seems to still be work in progress, e.g. no support for recursive queries yet, but I wonder where they're heading. I'll certainly keep an eye on the project but I wonder how exactly Glean aims to -- or maybe it already does -- improve upon the alternatives? From the talk linked in another comment I guess the distinctive feature may be the planned integration with IDEs. Correct me if I'm wrong. Other contenders provide great querying technology but there is indeed no strong focus on making such tech really convenient and integrated yet.
- dons 5y ago
  I think the point in the space Glean hits well is efficiency/latency (enough to power real time editing, like in IDE autocomplete or navigation), while having a schema and query language generic enough to do multiple languages and code-like things. You can accurately query JavaScript or Rust or PHP or Python or C++ with a common interface, which is a bit nuts :D
ing33k 5y ago
7GB docker image !
justinmchase 5y ago
But whats an example of a fact? Looks cool but I have no idea what its for.
conductor 5y ago
To prevent any confusion, this is a different product than Mozilla's Glean [0][1]. [0] https://docs.telemetry.mozilla.org/concepts/glean/glean.html https://docs.telemetry.mozilla.org/concepts/glean/glean.html [1] https://github.com/mozilla/glean/ https://github.com/mozilla/glean/
- deleted 5y ago
  [deleted]
- senden9 5y ago
  I was also confused first if (Mozilla) glean gained some out-of-scope features.
tclancy 5y ago
Getting a 401 when trying `docker pull ghcr.io/facebookincubator/glean/demo:latest` -- is that true for anyone else?
- simonmar 5y ago
  Sorry about that, the package was still set to private. Try again now?
  - tclancy 5y ago
    All set, thanks!
avinassh 5y ago
How does this actually work? Where can I learn more about the indexing and searching?
metalliqaz 5y ago
We have used SciTools Understand to do this on local source code. What is the use of putting this in the cloud? The website doesn't really explain that.
_jezell_ 5y ago
Is this basically Facebook's version of SourceGraph?

213 points by dons 5y ago

z3t4 5y ago

This seems very interesting, would love to see more alternatives to TreeSitter and microsoft LSP - what makes those hard to use is lack of examples and tutorials. So I hope tbere will be examples and tutorials. For example: How do you find all variables in scope when the text cursor is on line x and col y in /file/path/file.js

log101 5y ago

I didn't understand what it does

sealeck 5y ago
It allows you to use a query language (think SQL) to analyze source code.
- tyingq 5y ago
  The docs seem to get to queries here: https://glean.software/docs/angle/guide https://glean.software/docs/angle/guide

soonnow 5y ago

I had a look at the site and it seems to be parsing source code in multiple languages and storing the parsed "syntax trees" into a database for querying. I would love to know what the usecase for this tool is aside from maybe being a source for presentations? (We have 5 million if statements). How can this be used to improve code quality or any other aspect of the code lifecycle? Or is it solving problems in a completely different problem area?

lazamar 5y ago
Glean is focused on storing and querying data about the code. The idea is that you have your own program to collect that data, then you use Glean to store that compactly and to have snappy queries. You would create entries like "this is a declaration of X", "this is a use of X". Then you can query things like "give me all uses of X" in sub-millisecond time. You hook that up to an LSP server then you get almost zero-cost find-references, jump-to-definition, etc. The snappy queries also mean it becomes possible to perform whole codebase (and cross-language) analysis. That is, answering questions like "what code is not referenced from this root?", "does this Haskell function use anything that calls malloc?" (analysis through the ffi barrier). One can also attach all kinds of information from different sources to code entities, not only things derived from the source itself. You add things like run-time costs, frequency of use, common errors, etc, and an LSP server could make all of it available right in your editor. For very large or complex codebases, where it is just too expensive or too complicated to calculate this information locally a system like this becomes very useful.
- soonnow 5y ago
  > For very large or complex codebases, where it is just too expensive or too complicated to calculate this information locally a system like this becomes very useful. Thanks I guess I get it now. But to enable this functionality you'd need to have some form of frontend or integration into the existing build lifecycle? Or IDE integration I guess.
- scns 5y ago
  Oh wow, mindblowing stuff. Glad to see tech like this being open sourced, fuels the imagination about possible future scenarios. Do you use it on the Linux Kernel?
  - minxomat 5y ago
    A comparable, powerful system (CodeQL) was used recently on the kernel[1] and Chrome. You can learn more about it here: https://codeql.github.com/docs/codeql-overview/about-codeql/ https://codeql.github.com/docs/codeql-overview/about-codeql/ (disclosure: I work at GH on CQL) [1] https://pwning.systems/posts/sequoia-variant-analysis/ https://pwning.systems/posts/sequoia-variant-analysis/
    - X6S1x6Okd1st 5y ago
      Oof on the terms & conditions: https://securitylab.github.com/tools/codeql/license/ https://securitylab.github.com/tools/codeql/license/
- gigatexal 5y ago
  Thank you for this summary I was unsure of how this is really useful. That before step is missing I think.

dons 5y ago

We use this to power things like find-references or jump-to-def, "symbol search" and autocomplete, or more complicated code queries and analysis (even across languages). Imagine rich LSPs without a local checkout, web-based code queries, or seeding fuzzers and static analyzers with entry points in code. Our focus has been on very large scale, multi-language code indexing, and then low latency (e.g. hundreds of micros) query times, to drive highly interactive developer workflows.

the_duke 5y ago
This is really cool. Seems like there are only indexers for Flow and Hack though. Will there be more indexers built by Facebook, or will it rely on community contributions?
- dons 5y ago
  A bit of both I think.
- simonmar 5y ago
  There will be more indexers: we have Python, C++/Objective C, Rust, Java and Haskell. It's just a case of getting them ready to open source. You can see the schemas for most of these already in the repo: https://github.com/facebookincubator/Glean/tree/main/glean/schema/source https://github.com/facebookincubator/Glean/tree/main/glean/s...
soonnow 5y ago
Does that mean you are using the shell or how is it used to enable these functionalities?
- dons 5y ago
  Most clients hit the Glean server via the network (thrift/JSON) and then mostly via language bindings to the Glean query language, Angle. The shell is more for debugging/exploration. Imagine an IDE plugin that queries Glean over the network for symbol information about the current file, then shows that on hover. That sort of thing.

booleandilemma 5y ago

The very first page of the site should have examples of what you can do with it.

_tom_ 5y ago
It’s such a shame to get this much traffic, and have it almost all bounce, because there’s no description of WHY someone would care.

enjikaka 5y ago

Ew, Facebook.

doddsiedodds 5y ago

An excellent talk by Simon Marlow on Glean here: https://youtu.be/-OPN7QPsYKE https://youtu.be/-OPN7QPsYKE

simonmar 5y ago
I should point out that Glean has evolved quite a bit since that talk!

coderdd 5y ago

Great to see this space moving! Any pointers on diff vs Kythe? Kythe has a mostly fixed schema, for one. One of the pain points using Kythe is wiring up the indexer to the build system. Would Glean indexers be easier to wire up for the common cases? Other is the index post-processing, which is not very scalable in the open source version (due to go-beam having rough Flunk support, for example). Third, how does it link up references across compilation units? Is it heuristic, or relies on unique keys from indexers matching? Or across languages?

simonmar 5y ago
Kythe has one schema, whereas with Glean each language has its own schema with arbitrary amounts of language-specific detail. You can get a language-agnostic view by defining an abstraction layer as a schema. Our current (work in progress) language-agnostic layer is called "codemarkup" https://github.com/facebookincubator/Glean/blob/main/glean/schema/source/codemarkup.angle https://github.com/facebookincubator/Glean/blob/main/glean/s... For wiring up the indexer, there are various methods, it tends to depend very much on the language and the build system. For Flow for example, Glean output is just built into the typechecker, you just run it with some flags to spit out the Glean data. For C++, you need to get the compiler flags from the build system to pass to the Clang frontend. For Java the indexer is a compiler plugin; for Python it's built on libCST. Some indexers send their data directly to a Glean server, others generate files of JSON that get sent using a separate command-line tool. References use different methods depending on the language. For Flow for example there is a fact for an import that matches up with a fact for the export in the other file. For C++ there are facts that connect declarations with definitions, and references with declarations.
- Game_Ender 5y ago
  Is there an example of using the C++ indexer? I saw hack and JS on your site but missed C++ (Python would also be amazing!).
  - simonmar 5y ago
    We want to open-source the C++ and Python indexers but they're not ready yet - we have to separate them from internal build-system-specific bits.

erlich 5y ago

I can't believe Facebook hasn't canned Flowtype yet and moved to TypeScript. They will have to do it eventually.

muglug 5y ago
I'm not sure you understand the scale at which Facebook operates. They don't have to do anything. As long as billions of people keep using Facebook they can maintain their own static analysis tooling for Javascript for as long as they want.
- scns 5y ago
  You do have a point, a rewrite on that scale would be a colossal waste of manyears/$$. Your delivery could be nicer though.
ctvo 5y ago
Doesn't look like they're stopping their use of Hack either. Eventually is a long time so you're right.
wingspan 5y ago
The problem is that TypeScript does not scale to the size of the giant monorepo at Facebook, with hundreds of thousands, if not millions of files. Since they aren't organized into packages, it is just one giant flat namespace (any JS file can import any other JS file by the filename). It is pretty amazing to change a core file and see type errors across the entire codebase in a few seconds. The main way to scale in TypeScript is Project References, which don't work when you haven't separated your code into packages. (Worked at Facebook until June 2021).

simonw 5y ago

Feature request: a live demo! I would love to try out the web interface described at https://glean.software/docs/trying https://glean.software/docs/trying without pulling down a 7GB Docker image first.

deleted 5y ago
[deleted]
jamessb 5y ago
Even just a short video of someone using the web interface would be helpful.

da39a3ee 5y ago

I was recently looking for a library that takes a few lines of source code as input, and predicts the programming language as output. That seems like a very tractable machine learning problem, yet all I could find was a single python library which looks nice, but doesn't have much adoption, and requires installing the entirety of tensorflow despite the fact that users just want a trained model and a predict() function. Why doesn't a popular library like this exist?

jamessb 5y ago
GitHub's linguist library can be used to identify the programming language of a single file (edit: or of a whole project): https://github.com/github/linguist#single-file https://github.com/github/linguist#single-file
- da39a3ee 5y ago
  Thanks! My searches completely failed to find that. I can’t use it as a ruby library, but perhaps I can pull out the heuristics.yml and the naive bayes classifier weights to use in another language.

aabaker99 5y ago

Cool! I would love to play around with this. How do I write a schema and indexer for my favorite programming language that isn't currently (and won't be) supported with official releases? For Schemas, [1] says to modify (or base new ones off) these: https://github.com/facebookincubator/Glean/tree/main/glean/schema/source https://github.com/facebookincubator/Glean/tree/main/glean/s... For Indexers, it's a little less clear but it looks like I need to write my own type checker? [1] https://glean.software/docs/schema/workflow https://glean.software/docs/schema/workflow

marcodiego 5y ago

Is it a modern cscope?

rognjen 5y ago

Meta: should the ?open tracking part of the URL be removed?

ctvo 5y ago

Great job with this. What's your roadmap for releasing some of the tooling for editor integration? Really, the question is should I build something or wait a few weeks?

Grimm1 5y ago

Very cool! How does this differ algorithmically from the trigram based search that everything uses from google code search from like 20 years ago? And continuing off of that theme in practical terms how does it stand up against zoekt? I’m curious because zoekt is kind of slow when it comes to ingesting large amounts of code like all of the publicly available code on GitHub The few people using that commercially have basically had to spend a lot of time rewriting parts of it to make their goal of public codesearch for all attainable. I and a few people I know are pretty convinced that there are better and easier ways / technologies to make that happen.

ExtraE 5y ago

What, uh, is this? This is a space that I’m not familiar with and the linked site doesn’t make it super clear.

balddenimhero 5y ago

Datalog-ish query languages sure is a fun area to be working in. Such DSLs exist for various domains and, like Semmle's codeQL or the more academic Soufflé, Glean focuses on the domain of programming languages. Glean seems to still be work in progress, e.g. no support for recursive queries yet, but I wonder where they're heading. I'll certainly keep an eye on the project but I wonder how exactly Glean aims to -- or maybe it already does -- improve upon the alternatives? From the talk linked in another comment I guess the distinctive feature may be the planned integration with IDEs. Correct me if I'm wrong. Other contenders provide great querying technology but there is indeed no strong focus on making such tech really convenient and integrated yet.

dons 5y ago
I think the point in the space Glean hits well is efficiency/latency (enough to power real time editing, like in IDE autocomplete or navigation), while having a schema and query language generic enough to do multiple languages and code-like things. You can accurately query JavaScript or Rust or PHP or Python or C++ with a common interface, which is a bit nuts :D

ing33k 5y ago

7GB docker image !

justinmchase 5y ago

But whats an example of a fact? Looks cool but I have no idea what its for.

conductor 5y ago

To prevent any confusion, this is a different product than Mozilla's Glean [0][1]. [0] https://docs.telemetry.mozilla.org/concepts/glean/glean.html https://docs.telemetry.mozilla.org/concepts/glean/glean.html [1] https://github.com/mozilla/glean/ https://github.com/mozilla/glean/

deleted 5y ago
[deleted]
senden9 5y ago
I was also confused first if (Mozilla) glean gained some out-of-scope features.

tclancy 5y ago

Getting a 401 when trying `docker pull ghcr.io/facebookincubator/glean/demo:latest` -- is that true for anyone else?

simonmar 5y ago
Sorry about that, the package was still set to private. Try again now?
- tclancy 5y ago
  All set, thanks!

avinassh 5y ago

How does this actually work? Where can I learn more about the indexing and searching?

metalliqaz 5y ago

We have used SciTools Understand to do this on local source code. What is the use of putting this in the cloud? The website doesn't really explain that.

_jezell_ 5y ago

Is this basically Facebook's version of SourceGraph?