8 ms·

Sourcegraph: Incident involving unauthorized admin access

81 points by 0xedb 3y ago

beanjuiceII 3y ago
yikes, can i even trust this company now
- bastardoperator 3y ago
  Don't ever trust a company... you're going to be disappointed every time.
temp0010222 3y ago
[flagged]
- Awaawawawa 3y ago
  [flagged]
- mistrial9 3y ago
  that is .. odd. Can someone please explain-like-I-am-five that the ** is going on in that forum? huggingface links and expiring proxies get mentioned a lot
  - fein 3y ago
    It's an archive of 4chans /g/ board. Many things are discussed, many insults are leveled. Are you asking what desuarchive is or are you asking what 4chan is?
    - mistrial9 3y ago
      no - the transactions .. what are those lists of numbers being traded ? How are those valuable ? "What is going on in that forum?" that chat environment seems to be both a cause and a result of ADHD-like existance! can I say "hellish" ?
      - procone 3y ago
        It's an anonymous board. Those numbers are unique post ID's referenced by >>. The thread IDs have 3 chevrons. If you wish to reply to someone, you click the "reply" button and a >>$ID is placed in the comment box.
  - neom 3y ago
    It's really hard to understand, I originally thought it was a spam link, heh. It looks like you can somehow use Sourcegraphs "Cody" gateway to utilize Claude by Anthropic. I guess SG was rate limiting, and user HopeMan gained access to SGs admin systems and is asking people to drop their emails to get them un-rate-limited? Best I could make of the thread. https://docs.sourcegraph.com/cody/explanations/cody_gateway https://docs.sourcegraph.com/cody/explanations/cody_gateway https://claude.ai/ https://claude.ai/ (desuarchive seems to be a 4chan archive)
yorick 3y ago
Sourcegraph seems to have collected a bunch of these (now leaked) email addresses from signups on self-hosted instances. I remember being very surprised when I was signed up to their mailing list after I made an account on my self-hosted instance, and I'm not sure about the ethics (and legality) of collecting these in the first place.
- HumanOstrich 3y ago
  They need these email addresses so they can send you an automated welcome email that claims it is not automated and then ignore you when you reply. At least that was my experience.
  - connoro13c 3y ago
    Hey, I run a lot of our customer outreach and experience. There is some nuance to what is / isn't automated. When people sign up, their email is shared with our customer experience team (combo of inbound sales reps, customer engineers, and support depending on what information is shared), and then that assigned individual reaches out manually AND puts you into what is a called a 'nurture' campaign that provides helpful information, updates on Cody, etc. Those are automated unless you specifically request to talk with someone. If / when someone wants to talk to Sourcegraph, a member of our GTM team (sales / customer engineering) reaches out. If you can provide me with your email (DM it) I can look into what happened with you and get you the help / support you need. Otherwise, I think someone on our end just dropped the ball. Sorry about that! It happens when there is a human in the process and it isn't 100% automated. To your larger point, we agree that having a real human to interact with is important to a great experience. Please let us know how we can help you try and use Cody!
mirekrusin 3y ago
Overall very well handled and communicated. To get 11 points out of 10 remove adjectives from communication. When people read “quickly” they don’t think “oh great they were quick, I’m going to trust them more now”, they think “communication is biased”. Just write facts (ie time stamps or if you don’t have them use “hours” or “same day”).
- tana750cc 3y ago
  12 out of 10 for this very astute tip about any type of technical communication: "remove adjectives from communication". Agree; either use facts (days, hours, etc) or nothing, but no vague adjectives.
mgiannopoulos 3y ago
This my new personal record (3 days) between signing up for a service and getting a notice that my email address has been leaked :D
- deathbypenguin 3y ago
  Hey! you and me both, 4 days in my case, but it's a record for me.
- leke 3y ago
  #metoo
ttyyzz 3y ago
I don't know but it always seems to be worded like in this instance: "...A small subset of customers’ Sourcegraph license keys may have been accessed..." I don't buy that, it always seems to me like an attempt to downplay something.
- dcomas 3y ago
  I lead security at Sourcegraph and have been overseeing the incident investigation and resolution. To give you more specific details, I can say that we saw that the attacker viewed a page where they would have only seen the first 20 items, and we were able to determine what those were at the time of viewing because of stable sorting.
  - stuzenz 3y ago
    Kudos. Thanks for the good communications and security response.
  - 9dev 3y ago
    That… didn’t age well
    - jamietanna 3y ago
      In what way? Has something else happened since?
dspillett 3y ago
> accidentally committed a code change that contained an active site-admin access token Our regular reminder to try keep credentials and other security tokens well away from any source code where-ever possible, even if that might mean making things a touch less convenient. I'd guess that most of us have checked in or otherwise posted a credential at some point in our careers. I've certainly done it in the past with an application DB connection string and had to do the quick reconfigure to revoke that access¹ – in that instance resolution was quick & easy but for other environments it might be a lot more admin. Being careful isn't the solution because mistakes will always happen, making it damn near impossible to accidentally post credentials is the way to go. -- [1] even though the repo checked into could only be accessed from within the company, and the DB instance in question was locked down so only the application servers and the limited few with access to a VPN connecting to its subnet, good practise dictated immediate full revocation just in case
deleted 3y ago
[deleted]
lopkeny12ko 3y ago
[flagged]
- abtinf 3y ago
  That would be a solid approach to ensure that the next incident is more severe and longer lasting.
- pc86 3y ago
  People make mistakes, get over it.
  - lopkeny12ko 3y ago
    So we are expected to just shrug off a leak of PII collected from self-hosted instances? I would normally agree with you, but this was an egregious violation of trust.
    - Operyl 3y ago
      Realistically, you don't want to fire the person that makes this mistake. It pretty much is _never_ going to happen again for that employee, it's burned in their mind now.
    - sqs 3y ago
      The information that leaked was not collected from self-hosted instances. It was from Sourcegraph.com, which contains generally less sensitive information than self-hosted instances. See https://about.sourcegraph.com/blog/security-update-august-2023 https://about.sourcegraph.com/blog/security-update-august-20... for full information.
      - lopkeny12ko 3y ago
        Do you have a response to the current top comment of this submission? > Sourcegraph seems to have collected a bunch of these (now leaked) email addresses from signups on self-hosted instances. > I remember being very surprised when I was signed up to their mailing list after I made an account on my self-hosted instance, and I'm not sure about the ethics (and legality) of collecting these in the first place.

81 points by 0xedb 3y ago

beanjuiceII 3y ago

yikes, can i even trust this company now

bastardoperator 3y ago
Don't ever trust a company... you're going to be disappointed every time.

temp0010222 3y ago

[flagged]

Awaawawawa 3y ago
[flagged]
mistrial9 3y ago
that is .. odd. Can someone please explain-like-I-am-five that the ** is going on in that forum? huggingface links and expiring proxies get mentioned a lot
- fein 3y ago
  It's an archive of 4chans /g/ board. Many things are discussed, many insults are leveled. Are you asking what desuarchive is or are you asking what 4chan is?
  - mistrial9 3y ago
    no - the transactions .. what are those lists of numbers being traded ? How are those valuable ? "What is going on in that forum?" that chat environment seems to be both a cause and a result of ADHD-like existance! can I say "hellish" ?
    - procone 3y ago
      It's an anonymous board. Those numbers are unique post ID's referenced by >>. The thread IDs have 3 chevrons. If you wish to reply to someone, you click the "reply" button and a >>$ID is placed in the comment box.
- neom 3y ago
  It's really hard to understand, I originally thought it was a spam link, heh. It looks like you can somehow use Sourcegraphs "Cody" gateway to utilize Claude by Anthropic. I guess SG was rate limiting, and user HopeMan gained access to SGs admin systems and is asking people to drop their emails to get them un-rate-limited? Best I could make of the thread. https://docs.sourcegraph.com/cody/explanations/cody_gateway https://docs.sourcegraph.com/cody/explanations/cody_gateway https://claude.ai/ https://claude.ai/ (desuarchive seems to be a 4chan archive)

yorick 3y ago

Sourcegraph seems to have collected a bunch of these (now leaked) email addresses from signups on self-hosted instances. I remember being very surprised when I was signed up to their mailing list after I made an account on my self-hosted instance, and I'm not sure about the ethics (and legality) of collecting these in the first place.

HumanOstrich 3y ago
They need these email addresses so they can send you an automated welcome email that claims it is not automated and then ignore you when you reply. At least that was my experience.
- connoro13c 3y ago
  Hey, I run a lot of our customer outreach and experience. There is some nuance to what is / isn't automated. When people sign up, their email is shared with our customer experience team (combo of inbound sales reps, customer engineers, and support depending on what information is shared), and then that assigned individual reaches out manually AND puts you into what is a called a 'nurture' campaign that provides helpful information, updates on Cody, etc. Those are automated unless you specifically request to talk with someone. If / when someone wants to talk to Sourcegraph, a member of our GTM team (sales / customer engineering) reaches out. If you can provide me with your email (DM it) I can look into what happened with you and get you the help / support you need. Otherwise, I think someone on our end just dropped the ball. Sorry about that! It happens when there is a human in the process and it isn't 100% automated. To your larger point, we agree that having a real human to interact with is important to a great experience. Please let us know how we can help you try and use Cody!

mirekrusin 3y ago

Overall very well handled and communicated. To get 11 points out of 10 remove adjectives from communication. When people read “quickly” they don’t think “oh great they were quick, I’m going to trust them more now”, they think “communication is biased”. Just write facts (ie time stamps or if you don’t have them use “hours” or “same day”).

tana750cc 3y ago
12 out of 10 for this very astute tip about any type of technical communication: "remove adjectives from communication". Agree; either use facts (days, hours, etc) or nothing, but no vague adjectives.

mgiannopoulos 3y ago

This my new personal record (3 days) between signing up for a service and getting a notice that my email address has been leaked :D

deathbypenguin 3y ago
Hey! you and me both, 4 days in my case, but it's a record for me.
leke 3y ago
#metoo

ttyyzz 3y ago

I don't know but it always seems to be worded like in this instance: "...A small subset of customers’ Sourcegraph license keys may have been accessed..." I don't buy that, it always seems to me like an attempt to downplay something.

dcomas 3y ago
I lead security at Sourcegraph and have been overseeing the incident investigation and resolution. To give you more specific details, I can say that we saw that the attacker viewed a page where they would have only seen the first 20 items, and we were able to determine what those were at the time of viewing because of stable sorting.
- stuzenz 3y ago
  Kudos. Thanks for the good communications and security response.
- 9dev 3y ago
  That… didn’t age well
  - jamietanna 3y ago
    In what way? Has something else happened since?

dspillett 3y ago

> accidentally committed a code change that contained an active site-admin access token Our regular reminder to try keep credentials and other security tokens well away from any source code where-ever possible, even if that might mean making things a touch less convenient. I'd guess that most of us have checked in or otherwise posted a credential at some point in our careers. I've certainly done it in the past with an application DB connection string and had to do the quick reconfigure to revoke that access¹ – in that instance resolution was quick & easy but for other environments it might be a lot more admin. Being careful isn't the solution because mistakes will always happen, making it damn near impossible to accidentally post credentials is the way to go. -- [1] even though the repo checked into could only be accessed from within the company, and the DB instance in question was locked down so only the application servers and the limited few with access to a VPN connecting to its subnet, good practise dictated immediate full revocation just in case

deleted 3y ago

[deleted]

lopkeny12ko 3y ago

[flagged]

abtinf 3y ago
That would be a solid approach to ensure that the next incident is more severe and longer lasting.
pc86 3y ago
People make mistakes, get over it.
- lopkeny12ko 3y ago
  So we are expected to just shrug off a leak of PII collected from self-hosted instances? I would normally agree with you, but this was an egregious violation of trust.
  - Operyl 3y ago
    Realistically, you don't want to fire the person that makes this mistake. It pretty much is _never_ going to happen again for that employee, it's burned in their mind now.
  - sqs 3y ago
    The information that leaked was not collected from self-hosted instances. It was from Sourcegraph.com, which contains generally less sensitive information than self-hosted instances. See https://about.sourcegraph.com/blog/security-update-august-2023 https://about.sourcegraph.com/blog/security-update-august-20... for full information.
    - lopkeny12ko 3y ago
      Do you have a response to the current top comment of this submission? > Sourcegraph seems to have collected a bunch of these (now leaked) email addresses from signups on self-hosted instances. > I remember being very surprised when I was signed up to their mailing list after I made an account on my self-hosted instance, and I'm not sure about the ethics (and legality) of collecting these in the first place.