5 ms·

Building Sourcegraph - A large-scale code search engine in Go (2014)

82 points by philip1209 10y ago

jschrf 10y ago
Looks great. I've been building almost the exact same idea for fun as a side project, but initially focused on deep analysis of TypeScript and ES2015+ specifically. Runs in Docker as well, spinning up containers to do analysis work, but written in TypeScript and run via node. Definitely going to be following the blog. Thanks for sharing!
- sqs 10y ago
  Very cool. Would love to chat and see if we can push this stuff forward together. I can't seem to find your email address. Could you please email me at sqs@sourcegraph.com?
kozikow 10y ago
Is there a way of "git cloning" the https://sourcegraph.com/sourcegraph.com/sourcegraph/sourcegraph https://sourcegraph.com/sourcegraph.com/sourcegraph/sourcegr...? It is linked in the article. It does not seem to be at https://github.com/sourcegraph https://github.com/sourcegraph .
- shadowmint 10y ago
  apt-get install docker protobuf-compiler mkdir foo cd foo export GOPATH=`pwd` export PATH=${PATH}:`pwd`/bin mkdir bin mkdir -p src/sourcegraph.com/sourcegraph cd src/sourcegraph.com/sourcegraph git clone https://sourcegraph.com/sourcegraph.com/sourcegraph/sourcegraph sourcegraph cd sourcegraph.com/sourcegraph/sourcegraph make make serve-dev good luck. It doesn't actually run for me, just crashes, but you'll get kind of... most of the way there with that. (oh, you probably dont need protobuf unless you run `make generate`)
  - sqs 10y ago
    Sourcegraph CEO here. We're going to do a big source release soon. Then this should all be easier. We're growing quickly and pushing out a lot of product releases soon and have been busy on those. Email is in my profile if you really want to figure it out, and I can direct you to the right person to help.
sqs 10y ago
This was my talk. Thanks for posting it! Happy to answer questions. To try out Sourcegraph, sign up at https://sourcegraph.com https://sourcegraph.com and it'll take you on the tour. Also, I'll shamelessly plug our Chrome extension, which "Sourcegraphifies" GitHub (adding jump-to-def and cross-references): https://chrome.google.com/webstore/detail/sourcegraph-for-github/dgjhfomjieaadpoljlnidmbgkdffpack https://chrome.google.com/webstore/detail/sourcegraph-for-gi.... It's the #1 Chrome extension for GitHub. Finally, check out https://sourcegraph.com/beta https://sourcegraph.com/beta to get early access to some new languages and super neat features we're adding soon.
- kozikow 10y ago
  Extension is cool. I know that hackernews comment is not the most appropriate place for bug report, but pressing DAG in https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_bash_operator.py#L27 https://github.com/apache/incubator-airflow/blob/master/airf... or set_downstream in https://github.com/apache/incubator-airflow#L37 https://github.com/apache/incubator-airflow#L37 does not work. Goto definition works with both locally with emacs-ycmd.
  - sqs 10y ago
    Sourcegraph CEO here. Filed! Thanks. I've taken bug reports and feature requests from airplane seatmates and when walking down the sidewalk in SF, so this is actually not a bad place to post them. :)
- neurocroc 10y ago
  Any plans to make a Safari extension? Thank you.
  - sqs 10y ago
    Yep, we plan to release it for Safari as well. I don't have a date, but it's planned.
deleted 10y ago
[deleted]
the_arun 10y ago
Sorry for my naive question. What are the features this is adding over GitHub's source search?
- sqs 10y ago
  Sourcegraph CEO here. Unlike GitHub, Sourcegraph actually understands the code like your IDE. Unlike your IDE, it has all the code on GitHub and some other repo hosts already loaded up and built. (We're working on adding more languages beyond Go, etc.; check https://sourcegraph.com/beta https://sourcegraph.com/beta for early access.) Try out jumping to definition and seeing hovers at https://sourcegraph.com/github.com/gorilla/mux/-/blob/mux.go#L93 https://sourcegraph.com/github.com/gorilla/mux/-/blob/mux.go.... This helps you understand code that you're browsing. Check out global usage examples and refs of any function, such as those at https://sourcegraph.com/github.com/golang/go/-/info/GoPackage/net/http/-/ResponseWriter/WriteHeader https://sourcegraph.com/github.com/golang/go/-/info/GoPackag.... You can get to this page by right-clicking on tokens in the code view above. This helps you quickly figure out how to use a piece of code and see best practices. Finally, the code search is also semantic and is ranked by popularity. Try it out at https://sourcegraph.com/search?q=new+http+request https://sourcegraph.com/search?q=new+http+request. Code analysis and search at this scale is tough, but we're going to do it right, because it'll help developers learn more quickly, write better code, and reuse more existing stuff. (We're not totally there yet, but if you want to help by adding language support or joining our team, email me at sqs@sourcegraph.com.)
  - the_arun 10y ago
    Thanks for the details. Is this available as a self hosted application or enterprise edition? Useful for enterprises.
    - cuddy 10y ago
      Hey there, James from Sourcegraph here. Right now we don't offer a self-hosted solution and won't have one for the foreseeable future. Please email me your contact details so I can let you know when we do though. james@sourcegraph.com Thanks, James
electic 10y ago
This seems quite limited. I went to the site and it seems to work only on a few languages. Honestly, the best code search out there is GitHub. I learn so much on there and it works across every language. Right now I am coding in Obj-C, Elixir, Ruby, and doing web development in React. That is my day to day so this seems quite limited.
- sqs 10y ago
  Sourcegraph CEO here. Support for more languages is our #1 request by far. We're going to have more language support very soon. If you want to get early access, head over to https://sourcegraph.com/beta https://sourcegraph.com/beta and we'll be in touch.
barpet 10y ago
I do not understand these free tiers + steep pricing. This is like you are looking to bank on those that really really need your stuff.. But I do not want to invest myself in your ecosystem just to see it die eventually. I want to pay you because I do not want the service to die. Do you plan to serve ads in the free tier ? Why not have the pricing more dynamic ?
- sqs 10y ago
  Sourcegraph CEO here. We have a generous free tier that lets you see the value of the product (and get a lot of use out of it). Happy to help you get your company to see the value, too, if that's what you're interested in. If you want to chat further, my email is sqs@sourcegraph.com.
boyter 10y ago
I find Sourcegraph really interesting as a different approach to code indexing. Opengrok, Krugle, BitBucket Server and my own poor attempt searchcode and the downloadable server version simply index the source code as is which is to just index everything thats text (with some smarts on top usually). By comparison Sourchgraph builds parse trees and attempts to understand it. I am not sure as to which approach is better for the average user, but I do think that if you are using a language currently supported you should give it a try and compare to one of the afore mentioned solutions.
- sqs 10y ago
  Sourcegraph CEO here. Thanks for the kind words! Ben here is just being modest...searchcode is fantastic. It has come in handy in many crucial moments when I couldn't find the code I needed anywhere else. Here's a link for those who aren't familiar with it: https://searchcode.com/ https://searchcode.com/.
CptMauli 10y ago
How can one add other public repositories not hosted on github (for instance from the Eclipse foundation)?
- sqs 10y ago
  Sourcegraph CEO here. Right now, just email us (hi@sourcegraph.com) with the Git clone URLs and we'll add them. Soon, you'll be able to add them yourself. Thanks!
  - CptMauli 10y ago
    Ah great, I'll do that! Thx.

82 points by philip1209 10y ago

Looks great. I've been building almost the exact same idea for fun as a side project, but initially focused on deep analysis of TypeScript and ES2015+ specifically. Runs in Docker as well, spinning up containers to do analysis work, but written in TypeScript and run via node. Definitely going to be following the blog. Thanks for sharing!

sqs 10y ago
Very cool. Would love to chat and see if we can push this stuff forward together. I can't seem to find your email address. Could you please email me at sqs@sourcegraph.com?

kozikow 10y ago

Is there a way of "git cloning" the https://sourcegraph.com/sourcegraph.com/sourcegraph/sourcegraph https://sourcegraph.com/sourcegraph.com/sourcegraph/sourcegr...? It is linked in the article. It does not seem to be at https://github.com/sourcegraph https://github.com/sourcegraph .

shadowmint 10y ago
apt-get install docker protobuf-compiler mkdir foo cd foo export GOPATH=`pwd` export PATH=${PATH}:`pwd`/bin mkdir bin mkdir -p src/sourcegraph.com/sourcegraph cd src/sourcegraph.com/sourcegraph git clone https://sourcegraph.com/sourcegraph.com/sourcegraph/sourcegraph sourcegraph cd sourcegraph.com/sourcegraph/sourcegraph make make serve-dev good luck. It doesn't actually run for me, just crashes, but you'll get kind of... most of the way there with that. (oh, you probably dont need protobuf unless you run `make generate`)
- sqs 10y ago
  Sourcegraph CEO here. We're going to do a big source release soon. Then this should all be easier. We're growing quickly and pushing out a lot of product releases soon and have been busy on those. Email is in my profile if you really want to figure it out, and I can direct you to the right person to help.

sqs 10y ago

This was my talk. Thanks for posting it! Happy to answer questions. To try out Sourcegraph, sign up at https://sourcegraph.com https://sourcegraph.com and it'll take you on the tour. Also, I'll shamelessly plug our Chrome extension, which "Sourcegraphifies" GitHub (adding jump-to-def and cross-references): https://chrome.google.com/webstore/detail/sourcegraph-for-github/dgjhfomjieaadpoljlnidmbgkdffpack https://chrome.google.com/webstore/detail/sourcegraph-for-gi.... It's the #1 Chrome extension for GitHub. Finally, check out https://sourcegraph.com/beta https://sourcegraph.com/beta to get early access to some new languages and super neat features we're adding soon.

kozikow 10y ago
Extension is cool. I know that hackernews comment is not the most appropriate place for bug report, but pressing DAG in https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_bash_operator.py#L27 https://github.com/apache/incubator-airflow/blob/master/airf... or set_downstream in https://github.com/apache/incubator-airflow#L37 https://github.com/apache/incubator-airflow#L37 does not work. Goto definition works with both locally with emacs-ycmd.
- sqs 10y ago
  Sourcegraph CEO here. Filed! Thanks. I've taken bug reports and feature requests from airplane seatmates and when walking down the sidewalk in SF, so this is actually not a bad place to post them. :)
neurocroc 10y ago
Any plans to make a Safari extension? Thank you.
- sqs 10y ago
  Yep, we plan to release it for Safari as well. I don't have a date, but it's planned.

deleted 10y ago

[deleted]

the_arun 10y ago

Sorry for my naive question. What are the features this is adding over GitHub's source search?

sqs 10y ago
Sourcegraph CEO here. Unlike GitHub, Sourcegraph actually understands the code like your IDE. Unlike your IDE, it has all the code on GitHub and some other repo hosts already loaded up and built. (We're working on adding more languages beyond Go, etc.; check https://sourcegraph.com/beta https://sourcegraph.com/beta for early access.) Try out jumping to definition and seeing hovers at https://sourcegraph.com/github.com/gorilla/mux/-/blob/mux.go#L93 https://sourcegraph.com/github.com/gorilla/mux/-/blob/mux.go.... This helps you understand code that you're browsing. Check out global usage examples and refs of any function, such as those at https://sourcegraph.com/github.com/golang/go/-/info/GoPackage/net/http/-/ResponseWriter/WriteHeader https://sourcegraph.com/github.com/golang/go/-/info/GoPackag.... You can get to this page by right-clicking on tokens in the code view above. This helps you quickly figure out how to use a piece of code and see best practices. Finally, the code search is also semantic and is ranked by popularity. Try it out at https://sourcegraph.com/search?q=new+http+request https://sourcegraph.com/search?q=new+http+request. Code analysis and search at this scale is tough, but we're going to do it right, because it'll help developers learn more quickly, write better code, and reuse more existing stuff. (We're not totally there yet, but if you want to help by adding language support or joining our team, email me at sqs@sourcegraph.com.)
- the_arun 10y ago
  Thanks for the details. Is this available as a self hosted application or enterprise edition? Useful for enterprises.
  - cuddy 10y ago
    Hey there, James from Sourcegraph here. Right now we don't offer a self-hosted solution and won't have one for the foreseeable future. Please email me your contact details so I can let you know when we do though. james@sourcegraph.com Thanks, James

electic 10y ago

This seems quite limited. I went to the site and it seems to work only on a few languages. Honestly, the best code search out there is GitHub. I learn so much on there and it works across every language. Right now I am coding in Obj-C, Elixir, Ruby, and doing web development in React. That is my day to day so this seems quite limited.

sqs 10y ago
Sourcegraph CEO here. Support for more languages is our #1 request by far. We're going to have more language support very soon. If you want to get early access, head over to https://sourcegraph.com/beta https://sourcegraph.com/beta and we'll be in touch.

barpet 10y ago

I do not understand these free tiers + steep pricing. This is like you are looking to bank on those that really really need your stuff.. But I do not want to invest myself in your ecosystem just to see it die eventually. I want to pay you because I do not want the service to die. Do you plan to serve ads in the free tier ? Why not have the pricing more dynamic ?

sqs 10y ago
Sourcegraph CEO here. We have a generous free tier that lets you see the value of the product (and get a lot of use out of it). Happy to help you get your company to see the value, too, if that's what you're interested in. If you want to chat further, my email is sqs@sourcegraph.com.

boyter 10y ago

I find Sourcegraph really interesting as a different approach to code indexing. Opengrok, Krugle, BitBucket Server and my own poor attempt searchcode and the downloadable server version simply index the source code as is which is to just index everything thats text (with some smarts on top usually). By comparison Sourchgraph builds parse trees and attempts to understand it. I am not sure as to which approach is better for the average user, but I do think that if you are using a language currently supported you should give it a try and compare to one of the afore mentioned solutions.

sqs 10y ago
Sourcegraph CEO here. Thanks for the kind words! Ben here is just being modest...searchcode is fantastic. It has come in handy in many crucial moments when I couldn't find the code I needed anywhere else. Here's a link for those who aren't familiar with it: https://searchcode.com/ https://searchcode.com/.

CptMauli 10y ago

How can one add other public repositories not hosted on github (for instance from the Eclipse foundation)?

sqs 10y ago
Sourcegraph CEO here. Right now, just email us (hi@sourcegraph.com) with the Git clone URLs and we'll add them. Soon, you'll be able to add them yourself. Thanks!
- CptMauli 10y ago
  Ah great, I'll do that! Thx.