7 ms·

Sourcegraph: Why we're indexing the OSS universe

58 points by jka 5y ago

AltF4me 5y ago
Sourcegraph have been sending unsolicited emails to my personal and work email addresses, and there's no way I signed up to any marketing on either of them. Very dodgy practices going on.
- ithkuil 5y ago
  Same here, whole at my previous job, they contacted me and since I knew the product already I really wanted to use it. I engaged with them because I wanted to convince my management to pay for the damn thing instead of running some crappy hound instance. But you know how those things go, management didn't want to invest time in running the trial, nobody could just install this on their spare time because the very fact that we started this engagement with the trial program effectively turned fun into work and so nothing happened. That's why opensource wins. Nobody is pushing it down your throat. You have a problem, you find a solution, you have the incentive to actually follow through your own drive and just use the best tool you found. Some companies have understood that and monetize on support and other things that happen long after you've been hooked to the free and open product. Not sure sourcegraph can't do that as well, but I'm sure they have their good reason. Business is hard, I'm just an engineer. I'm sharing this perspective just because, when you seel stuff to engineers, you should know how engineers think. Yes, some products for engineers are sold to managers/execs instead; you can tell because engineers hate to use those, but they usually have no choice. Sourcegraph is a pleasure to use. I'm regularly impressed by the snappiness of the UI. No wonder they're trying to sell it to engineers, it seems to have been indeed built for engineers! But yeah, their reach out strategy can feel a bit invasive and sloppy.
  - ushakov 5y ago
    Sourcegraph is literally pushing their trial and upgrade down your throat, once you have the open source edition Then they send you emails from a mailing list you didn’t sign up for: Thanks for installing Sourcegraph Hello- Mark here with the Sourcegraph Team! I've shared a couple of links below to help you get started with Sourcegraph. https://docs.sourcegraph.com/getting-started https://docs.sourcegraph.com/getting-started https://docs.sourcegraph.com/integration https://docs.sourcegraph.com/integration Out of curiosity, how did you hear about Sourcegraph, and is there a specific use case you're evaluating Sourcegraph for? Best, Mark Muldez — DevTools Advocate
  - unicodeveloper 5y ago
    Hi, apologies for the invasiveness. I'm glad you are impressed with Sourcegraph and we're working hard to make the experience better.
- ushakov 5y ago
  Can confirm, same thing happened to me and i’ve never signed up for any newsletter (and i always make sure not to)
- unicodeveloper 5y ago
  Hello, Sorry about this. How can I reach you privately? I need your email addresses so that we can remove it from our communication line moving forward. PS: I work at Sourcegraph.
- beliu 5y ago
  Sourcegraph CTO here, I'm very sorry this happened with you. If you shoot me your email (I'm reachable at beyang@sourcegraph.com), I'll ensure you don't receive any future sales/marketing emails from us. This is something we should be clearer and more upfront about. We collect emails for sales purposes from self-hosted Sourcegraph instances, but I agree that entering your email as part of the installation process doesn't mean you want us to email you directly. Going to fix this!
markl42 5y ago
Big fan of sourcegraph, we use it at work. super excited to see this happen. (I've been using grep.app until now which is awesome too!)
- unicodeveloper 5y ago
  Thanks for the great feedback.
bArray 5y ago
> By the end of the year, we will have grown our index to include every open source repository with more than 1 star on GitHub and GitLab (that’s 5M+ repos). And what if your source code has a license that says "Do as you like, as long as you're not Sourcegraph"? I'm not entirely sold on the legalities of what they are doing... With search engines you have your robots file and many other ways to opt out of indexing, I wonder what Sourcegraph have. My guess is nothing.
- jka 5y ago
  Are you thinking along the lines of GitHub co-pilot with this concern, out of interest? Or are there other scenarios where you think people would want their code to be public, but not indexed/crawled?
- 542458 5y ago
  License files (and copyright) aren’t omnipotent. I understand why you might feel the ethics lie in a different direction, but the established legal precedent is that parsing and building search indexes or neural nets is generally fair game regardless of what the copyright holder wants. You mentioned robots.txt - there’s no legal enforcement behind that either. Notably, the Internet Archive bot ignores it completely.
- electroly 5y ago
  > "Do as you like, as long as you're not Sourcegraph" It wouldn't matter that this provision is in the license that you offer to your users, because when you post your code publicly on GitHub you agree to their TOS which includes a section called "License Grant to Other Users". This grants all GitHub users, including Sourcegraph, the right to "use, display, and perform" your code. Even if this were not true, it has been claimed (I'm not qualified to judge whether it's correct) that fair use covers this type of usage even if there is no license.
mistrial9 5y ago
I admin a dozen FOSS repos for my dot-org away from Microsoft-Github; I spun up Sourcegraph but it "phones home" so much that it made me nervous, and I uninstalled it. It seems to assume a high-bandwidth, always-on sort of world, and who-knows-what-else it does because it was certainly a black box. Overall: not safe for privacy or off-beat FOSS "In My Opinion"
- jka 5y ago
  Thanks - I wasn't aware of that before submitting this story, and it's a good concern to have. It led me to https://docs.sourcegraph.com/dev/background-information/telemetry https://docs.sourcegraph.com/dev/background-information/tele... and https://docs.sourcegraph.com/admin/pings https://docs.sourcegraph.com/admin/pings. Neither of which mention opt-in or opt-out settings. Upvoting you to raise awareness.
  - beliu 5y ago
    Hey, I'm the CTO/co-founder of Sourcegraph. Thanks for the feedback here. I've opened up a pull request to make it clearer how to disable non-critical telemetry: https://github.com/sourcegraph/sourcegraph/pull/24171 https://github.com/sourcegraph/sourcegraph/pull/24171. Please let us know if you have any feedback on that! Will respond to the OP's concerns, which I agree are important, in a direct reply.
    - jka 5y ago
      Thanks for responding here, in the pull request, and committing to respond to the parent poster. That's more than required, and welcome. In general I reckon a lot of us in tech believe that our e-mail addresses "are out there anyway", and as such we start to think that it's reasonable to opt-in collection of personal information on the behalf of others (per the Critical Telemetry[1] section) without their full consent. I don't personally think that's OK behaviour. Good products grow and are shared by worth-of-mouth and network effects over time when they're genuinely useful (as I think sourcegraph is), and I'd debate whether there's greater overall value in silently transmitting e-mail addresses (something that many people will only learn about at a later date) versus the potential privacy and reputation costs (such as could arise from conversations like this). There may be some kind of argument that the information collection's required in order to send security and policy update notices; I'm uncertain about that: it's honest and useful to announce relevant information to the public when ready, but some consumers may wish to stay current on those themselves rather than be (unwittingly, at least) added to push-based messaging. These would probably be considerations you'd have to reconcile not only with your own codebase and perspective, but also with your colleagues and peers, and I understand that friction - this is just honest feedback from my perspective. [1] - https://github.com/sourcegraph/sourcegraph/blob/66ce1f81494615f5ac8ebacf774190af83fc3d60/doc/admin/pings.md#critical-telemetry https://github.com/sourcegraph/sourcegraph/blob/66ce1f814946...
asy22 5y ago
If you're looking for a desktop application you can run locally, https://www.sourcetrail.com/ https://www.sourcetrail.com/ is quite nice.
c17r 5y ago
There's https://github.com/livegrep/livegrep https://github.com/livegrep/livegrep for your own personal stuff.

58 points by jka 5y ago

Sourcegraph have been sending unsolicited emails to my personal and work email addresses, and there's no way I signed up to any marketing on either of them. Very dodgy practices going on.

ithkuil 5y ago
Same here, whole at my previous job, they contacted me and since I knew the product already I really wanted to use it. I engaged with them because I wanted to convince my management to pay for the damn thing instead of running some crappy hound instance. But you know how those things go, management didn't want to invest time in running the trial, nobody could just install this on their spare time because the very fact that we started this engagement with the trial program effectively turned fun into work and so nothing happened. That's why opensource wins. Nobody is pushing it down your throat. You have a problem, you find a solution, you have the incentive to actually follow through your own drive and just use the best tool you found. Some companies have understood that and monetize on support and other things that happen long after you've been hooked to the free and open product. Not sure sourcegraph can't do that as well, but I'm sure they have their good reason. Business is hard, I'm just an engineer. I'm sharing this perspective just because, when you seel stuff to engineers, you should know how engineers think. Yes, some products for engineers are sold to managers/execs instead; you can tell because engineers hate to use those, but they usually have no choice. Sourcegraph is a pleasure to use. I'm regularly impressed by the snappiness of the UI. No wonder they're trying to sell it to engineers, it seems to have been indeed built for engineers! But yeah, their reach out strategy can feel a bit invasive and sloppy.
- ushakov 5y ago
  Sourcegraph is literally pushing their trial and upgrade down your throat, once you have the open source edition Then they send you emails from a mailing list you didn’t sign up for: Thanks for installing Sourcegraph Hello- Mark here with the Sourcegraph Team! I've shared a couple of links below to help you get started with Sourcegraph. https://docs.sourcegraph.com/getting-started https://docs.sourcegraph.com/getting-started https://docs.sourcegraph.com/integration https://docs.sourcegraph.com/integration Out of curiosity, how did you hear about Sourcegraph, and is there a specific use case you're evaluating Sourcegraph for? Best, Mark Muldez — DevTools Advocate
- unicodeveloper 5y ago
  Hi, apologies for the invasiveness. I'm glad you are impressed with Sourcegraph and we're working hard to make the experience better.
ushakov 5y ago
Can confirm, same thing happened to me and i’ve never signed up for any newsletter (and i always make sure not to)
unicodeveloper 5y ago
Hello, Sorry about this. How can I reach you privately? I need your email addresses so that we can remove it from our communication line moving forward. PS: I work at Sourcegraph.
beliu 5y ago
Sourcegraph CTO here, I'm very sorry this happened with you. If you shoot me your email (I'm reachable at beyang@sourcegraph.com), I'll ensure you don't receive any future sales/marketing emails from us. This is something we should be clearer and more upfront about. We collect emails for sales purposes from self-hosted Sourcegraph instances, but I agree that entering your email as part of the installation process doesn't mean you want us to email you directly. Going to fix this!

markl42 5y ago

Big fan of sourcegraph, we use it at work. super excited to see this happen. (I've been using grep.app until now which is awesome too!)

unicodeveloper 5y ago
Thanks for the great feedback.

bArray 5y ago

> By the end of the year, we will have grown our index to include every open source repository with more than 1 star on GitHub and GitLab (that’s 5M+ repos). And what if your source code has a license that says "Do as you like, as long as you're not Sourcegraph"? I'm not entirely sold on the legalities of what they are doing... With search engines you have your robots file and many other ways to opt out of indexing, I wonder what Sourcegraph have. My guess is nothing.

jka 5y ago
Are you thinking along the lines of GitHub co-pilot with this concern, out of interest? Or are there other scenarios where you think people would want their code to be public, but not indexed/crawled?
542458 5y ago
License files (and copyright) aren’t omnipotent. I understand why you might feel the ethics lie in a different direction, but the established legal precedent is that parsing and building search indexes or neural nets is generally fair game regardless of what the copyright holder wants. You mentioned robots.txt - there’s no legal enforcement behind that either. Notably, the Internet Archive bot ignores it completely.
electroly 5y ago
> "Do as you like, as long as you're not Sourcegraph" It wouldn't matter that this provision is in the license that you offer to your users, because when you post your code publicly on GitHub you agree to their TOS which includes a section called "License Grant to Other Users". This grants all GitHub users, including Sourcegraph, the right to "use, display, and perform" your code. Even if this were not true, it has been claimed (I'm not qualified to judge whether it's correct) that fair use covers this type of usage even if there is no license.

mistrial9 5y ago

I admin a dozen FOSS repos for my dot-org away from Microsoft-Github; I spun up Sourcegraph but it "phones home" so much that it made me nervous, and I uninstalled it. It seems to assume a high-bandwidth, always-on sort of world, and who-knows-what-else it does because it was certainly a black box. Overall: not safe for privacy or off-beat FOSS "In My Opinion"

jka 5y ago
Thanks - I wasn't aware of that before submitting this story, and it's a good concern to have. It led me to https://docs.sourcegraph.com/dev/background-information/telemetry https://docs.sourcegraph.com/dev/background-information/tele... and https://docs.sourcegraph.com/admin/pings https://docs.sourcegraph.com/admin/pings. Neither of which mention opt-in or opt-out settings. Upvoting you to raise awareness.
- beliu 5y ago
  Hey, I'm the CTO/co-founder of Sourcegraph. Thanks for the feedback here. I've opened up a pull request to make it clearer how to disable non-critical telemetry: https://github.com/sourcegraph/sourcegraph/pull/24171 https://github.com/sourcegraph/sourcegraph/pull/24171. Please let us know if you have any feedback on that! Will respond to the OP's concerns, which I agree are important, in a direct reply.
  - jka 5y ago
    Thanks for responding here, in the pull request, and committing to respond to the parent poster. That's more than required, and welcome. In general I reckon a lot of us in tech believe that our e-mail addresses "are out there anyway", and as such we start to think that it's reasonable to opt-in collection of personal information on the behalf of others (per the Critical Telemetry[1] section) without their full consent. I don't personally think that's OK behaviour. Good products grow and are shared by worth-of-mouth and network effects over time when they're genuinely useful (as I think sourcegraph is), and I'd debate whether there's greater overall value in silently transmitting e-mail addresses (something that many people will only learn about at a later date) versus the potential privacy and reputation costs (such as could arise from conversations like this). There may be some kind of argument that the information collection's required in order to send security and policy update notices; I'm uncertain about that: it's honest and useful to announce relevant information to the public when ready, but some consumers may wish to stay current on those themselves rather than be (unwittingly, at least) added to push-based messaging. These would probably be considerations you'd have to reconcile not only with your own codebase and perspective, but also with your colleagues and peers, and I understand that friction - this is just honest feedback from my perspective. [1] - https://github.com/sourcegraph/sourcegraph/blob/66ce1f81494615f5ac8ebacf774190af83fc3d60/doc/admin/pings.md#critical-telemetry https://github.com/sourcegraph/sourcegraph/blob/66ce1f814946...

asy22 5y ago

If you're looking for a desktop application you can run locally, https://www.sourcetrail.com/ https://www.sourcetrail.com/ is quite nice.

c17r 5y ago

There's https://github.com/livegrep/livegrep https://github.com/livegrep/livegrep for your own personal stuff.