4 ms·

This is a great intro / overview of full-text search for those wondering how to build your own search engine. It's a great 101-level exercise to write an inver

This is a great intro / overview of full-text search for those wondering how to build your own search engine. It's a great 101-level exercise to write an inverted index implementation you can do it in an afternoon , and then expand to a leaf /aggregator in follow-up exercises.

jeffbee 4y ago
It is indeed Information Retrieval 101 level stuff which leads to the question of why this is the best GitHub can do with all the resources of Microsoft behind them. It's almost useless, at least for C++. It can't tell the difference between foo(int) and foo(double) or this::foo vs. that::foo. If I wanted the kind of search engine I can get a teenager to write in 16 weeks why would I expect my org to be paying $$$ for the service?
- 100k 4y ago
  Have you tried the new search? Thanks to the variable length ngram indexing mentioned in the post, it can handle all of those cases. Sign up here to try it: https://github.com/features/code-search https://github.com/features/code-search Symbol extraction for C and C++ is currently disabled because we were having problems with the performance of the tree-sitter queries we were using, but we are planning to bring that back.
  - jeffbee 4y ago
    Sorry, it cannot handle any of those cases. You're talking about the ability to find the literal `this::foo` but that's not how it would normally appear. It normally will appear anywhere inside a `namespace this` scope, which cs.github does not grok. And cs.github cannot address finding the definition related to a given call site. It doesn't even try.
    - 100k 4y ago
      You are correct, as I mentioned, we do not analyze symbols for C and C++ at this time.
      - exikyut 4y ago
        Grimoire (I see that in the HTML and HTTP requests for cs.chromium.org and cs.android.com, so I presume that's what it's called) is really cool, although sadly obviously not OSS. It completely falls apart when faced with JS, which is increasingly being checked in as part of frontend and Mojo glue code, so it's (from the perspective of an outsider trying to get their feet wet) a bit creaky, but being able to click around in C++, which I don't really understand at this point, and learn something new almost every time, is really cool, and IMO representative of at least one concrete beneficial outcome whenever you do get to this. I wonder if it would be possible to leverage LSP as a kind of tokenization generalization framework, or even piggyback off of the existing effort by incorporating search-friendly/-helpful metadata into future versions of the protocol spec.
- burntsushi 4y ago
  What a shit take. The article itself is perhaps a nice light overview of 101-ish level concepts, although knowing how and when to apply them in a real engineering context is not something I would consider 101 level. And certainly, building something that is actually at the scale of GitHub Search is nowhere near 101 level. This is what a 101-level inverted index implementation looks like: https://github.com/BurntSushi/imdb-rename https://github.com/BurntSushi/imdb-rename In other words, absolutely nothing like what GitHub built. Nowhere close.
- __float 4y ago
  "all the resources of Microsoft" doesn't really say anything about the size of the team involved here. Frankly, it sounds like a pretty small one: clearly GitHub is a successful business with many customers and a significant user base even with very basic code search. It seems to me that you know what you want from such a service, but focusing on making C++ exceptionally great in this service would come at the cost of, say, general quality across all languages, or frontend usability. A very reasonable tradeoff for a beta-quality product. Why must you be so harsh on them?
anitil 4y ago
I did this for our organization using sqlite's FTS module and datasette and boy was it fast. Unfortunately I did get (temporarily) banned from the organisations github account, but it was definitely worth it. Even now I find myself using it despite the index being a few months out of date.
- interactivecode 4y ago
  I would love to see how to do something like this. Are you open to sharing your implementation?

jeffbee 4y ago

It is indeed Information Retrieval 101 level stuff which leads to the question of why this is the best GitHub can do with all the resources of Microsoft behind them. It's almost useless, at least for C++. It can't tell the difference between foo(int) and foo(double) or this::foo vs. that::foo. If I wanted the kind of search engine I can get a teenager to write in 16 weeks why would I expect my org to be paying $$$ for the service?

100k 4y ago
Have you tried the new search? Thanks to the variable length ngram indexing mentioned in the post, it can handle all of those cases. Sign up here to try it: https://github.com/features/code-search https://github.com/features/code-search Symbol extraction for C and C++ is currently disabled because we were having problems with the performance of the tree-sitter queries we were using, but we are planning to bring that back.
- jeffbee 4y ago
  Sorry, it cannot handle any of those cases. You're talking about the ability to find the literal `this::foo` but that's not how it would normally appear. It normally will appear anywhere inside a `namespace this` scope, which cs.github does not grok. And cs.github cannot address finding the definition related to a given call site. It doesn't even try.
  - 100k 4y ago
    You are correct, as I mentioned, we do not analyze symbols for C and C++ at this time.
    - exikyut 4y ago
      Grimoire (I see that in the HTML and HTTP requests for cs.chromium.org and cs.android.com, so I presume that's what it's called) is really cool, although sadly obviously not OSS. It completely falls apart when faced with JS, which is increasingly being checked in as part of frontend and Mojo glue code, so it's (from the perspective of an outsider trying to get their feet wet) a bit creaky, but being able to click around in C++, which I don't really understand at this point, and learn something new almost every time, is really cool, and IMO representative of at least one concrete beneficial outcome whenever you do get to this. I wonder if it would be possible to leverage LSP as a kind of tokenization generalization framework, or even piggyback off of the existing effort by incorporating search-friendly/-helpful metadata into future versions of the protocol spec.
burntsushi 4y ago
What a shit take. The article itself is perhaps a nice light overview of 101-ish level concepts, although knowing how and when to apply them in a real engineering context is not something I would consider 101 level. And certainly, building something that is actually at the scale of GitHub Search is nowhere near 101 level. This is what a 101-level inverted index implementation looks like: https://github.com/BurntSushi/imdb-rename https://github.com/BurntSushi/imdb-rename In other words, absolutely nothing like what GitHub built. Nowhere close.
__float 4y ago
"all the resources of Microsoft" doesn't really say anything about the size of the team involved here. Frankly, it sounds like a pretty small one: clearly GitHub is a successful business with many customers and a significant user base even with very basic code search. It seems to me that you know what you want from such a service, but focusing on making C++ exceptionally great in this service would come at the cost of, say, general quality across all languages, or frontend usability. A very reasonable tradeoff for a beta-quality product. Why must you be so harsh on them?

anitil 4y ago

I did this for our organization using sqlite's FTS module and datasette and boy was it fast. Unfortunately I did get (temporarily) banned from the organisations github account, but it was definitely worth it. Even now I find myself using it despite the index being a few months out of date.

interactivecode 4y ago
I would love to see how to do something like this. Are you open to sharing your implementation?