4 ms·

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product,

by sqs 3y ago

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing to search everything exhaustively). Happy to chat with anyone who's building this kind of thing, including with folks at Val Town, which is awesome.

isker 3y ago
And when you're ready to do indexed search, Zoekt (over which Sourcegraph graciously took maintainership a while ago) is the best way to do it that I've found. After discounting both Livegrep and Hound (they both struggled to perform in various dimensions with the amount of stuff we wanted indexed, Hound moreso than Livegrep), we migrated to Zoekt from a (necessarily) very old and creaky deployment of OpenGrok and it's night and day, both in terms of indexing performance and search performance/ergonomics. Sourcegraph of course adds many more sophisticated features on top of just the code search that Zoekt provides.
morgante 3y ago
I've been surprised at how far you can get without indexing. Ex. I always assume we'll need to add an index to speed up GritQL (https://github.com/getgrit/gritql https://github.com/getgrit/gritql), but we've gotten pretty far with doing search entirely on the fly.
- worldsayshi 3y ago
  What does 'on the fly' entail here?
  - simonw 3y ago
    I'm going to guess brute force - scan everything for the search term, rather than trying to use an index. I'm always amazed at how fast ripgrep (rg) can brute force it's way through hundreds of MBs of source code.
    - morgante 3y ago
      Yes, exactly. When doing a search, we parse and search every file without any indexing. Of course, it could still be sped up considerably with an index but brute force is surprisingly effective (we use some of the same techniques/crates as ripgrep).
klysm 3y ago
I apply this thinking to lots of problems. Do the dumb thing that involves the least state and prove we need to lean more towards memory for speed. It’s much simpler to keep things correct when nothing is cached
hinkley 3y ago
There was someone doing temporal databases that was compressing blocks on disk and doing streaming decompress and search on them. Things in L2 cache go very very fast.
baobun 3y ago
You'll also be in a much better spot to pick appropriate indexing when you actually have sizable and representative workloads.
mechanicker 3y ago
I hope this and SCIP becomes a standard and we have more programming languages emitting symbols in SCIP format.
- mdaniel 3y ago
  I thought SCIP got promoted into https://lsif.dev/ https://lsif.dev/ but chasing the https://github.com/sourcegraph/lsif-java https://github.com/sourcegraph/lsif-java link resolves to https://github.com/sourcegraph/scip-java https://github.com/sourcegraph/scip-java so maybe I had the evolution relationship backward. Anyway, I'm thankful at least that code is still Apache 2 https://github.com/topics/lsif https://github.com/topics/lsif may interest this audience, too, since the scip topic tag seems to clash with something else Also, I learned last night that GitLab embraces LSIF, too https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-code-intelligence https://docs.gitlab.com/ee/topics/autodevops/stages.html#aut...
  - __float 3y ago
    SCIP is an evolution of LSIF, basically: https://sourcegraph.com/blog/announcing-scip https://sourcegraph.com/blog/announcing-scip
- ivanovm 3y ago
  My, and my friends experiences with SCIP indexers built by Sourcegraph have been less than stellar. They are buggy and sparsely maintained
  - erikars 3y ago
    Hello! I am Head of Engineering at Sourcegraph. I'd love to get feedback on which SCIP indexers you've had issues with, and, if you have the time, feedback on what sort of problems you've had with them. Thank you so much!
    - erikars 2y ago
      [dead]
beembeem 3y ago
Any opinions on mozilla's DXR?
FalconSensei 3y ago
Do you plan on ever allowing users to change the font size?

by sqs 3y ago

isker 3y ago

And when you're ready to do indexed search, Zoekt (over which Sourcegraph graciously took maintainership a while ago) is the best way to do it that I've found. After discounting both Livegrep and Hound (they both struggled to perform in various dimensions with the amount of stuff we wanted indexed, Hound moreso than Livegrep), we migrated to Zoekt from a (necessarily) very old and creaky deployment of OpenGrok and it's night and day, both in terms of indexing performance and search performance/ergonomics. Sourcegraph of course adds many more sophisticated features on top of just the code search that Zoekt provides.

morgante 3y ago

I've been surprised at how far you can get without indexing. Ex. I always assume we'll need to add an index to speed up GritQL (https://github.com/getgrit/gritql https://github.com/getgrit/gritql), but we've gotten pretty far with doing search entirely on the fly.

worldsayshi 3y ago
What does 'on the fly' entail here?
- simonw 3y ago
  I'm going to guess brute force - scan everything for the search term, rather than trying to use an index. I'm always amazed at how fast ripgrep (rg) can brute force it's way through hundreds of MBs of source code.
  - morgante 3y ago
    Yes, exactly. When doing a search, we parse and search every file without any indexing. Of course, it could still be sped up considerably with an index but brute force is surprisingly effective (we use some of the same techniques/crates as ripgrep).

klysm 3y ago

I apply this thinking to lots of problems. Do the dumb thing that involves the least state and prove we need to lean more towards memory for speed. It’s much simpler to keep things correct when nothing is cached

hinkley 3y ago

There was someone doing temporal databases that was compressing blocks on disk and doing streaming decompress and search on them. Things in L2 cache go very very fast.

baobun 3y ago

You'll also be in a much better spot to pick appropriate indexing when you actually have sizable and representative workloads.

mechanicker 3y ago

I hope this and SCIP becomes a standard and we have more programming languages emitting symbols in SCIP format.

mdaniel 3y ago
I thought SCIP got promoted into https://lsif.dev/ https://lsif.dev/ but chasing the https://github.com/sourcegraph/lsif-java https://github.com/sourcegraph/lsif-java link resolves to https://github.com/sourcegraph/scip-java https://github.com/sourcegraph/scip-java so maybe I had the evolution relationship backward. Anyway, I'm thankful at least that code is still Apache 2 https://github.com/topics/lsif https://github.com/topics/lsif may interest this audience, too, since the scip topic tag seems to clash with something else Also, I learned last night that GitLab embraces LSIF, too https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-code-intelligence https://docs.gitlab.com/ee/topics/autodevops/stages.html#aut...
- __float 3y ago
  SCIP is an evolution of LSIF, basically: https://sourcegraph.com/blog/announcing-scip https://sourcegraph.com/blog/announcing-scip
ivanovm 3y ago
My, and my friends experiences with SCIP indexers built by Sourcegraph have been less than stellar. They are buggy and sparsely maintained
- erikars 3y ago
  Hello! I am Head of Engineering at Sourcegraph. I'd love to get feedback on which SCIP indexers you've had issues with, and, if you have the time, feedback on what sort of problems you've had with them. Thank you so much!
  - erikars 2y ago
    [dead]

beembeem 3y ago

Any opinions on mozilla's DXR?

FalconSensei 3y ago

Do you plan on ever allowing users to change the font size?