152 ms·

Slack's private GitHub code repositories stolen over holidays

347 points by mindracer 4y ago

photoGrant 4y ago
This is how LastPass started the efficient downward spiral, no? This isn't shocking, but also is. Is it wise to simply keep private repositories away from GitHub at this point? It seems the best way to avoid being drawn in with the rest as a target. Internal bad actors? So location wouldn't matter?
- verdverm 4y ago
  If you read the details... > The incident involves threat actors gaining access to Slack's externally hosted GitHub repositories via a "limited" number of Slack employee tokens that were stolen. > While some of Slack's private code repositories were breached, Slack’s primary codebase and customer data remains unaffected, according to the company.
  - throwaway010523 4y ago
    Wouldn't private git hosting add another barrier for the attackers? In addition to the tokens, they would also need to somehow gain access to the network through VPN or through an opening in the firewall?
    - hardware2win 4y ago
      Github repos can require using company's vpn
    - trollied 4y ago
      Yes. By way of example, my employer does this: * gitlab hosted internally * VPN needed to access gitlab * VPN requires a gsuite login, with 2FA * You can only login to a gsuite account on a employer-provided machine (so no access to anything on a non-employer machine, even email) You'd have to steal a company laptop, or social engineer yourself into a building with a desktop machine, to even start to get near the code repository. The "must login from a employer-provided machine" thing can be disabled on a per-user basis by remote policy (via an administrator), if needed.
      - tcmb 4y ago
        > You can only login to a gsuite account on a employer-provided machine How is this achieved?
        trollied 4y ago
        There's an Endpoint Verification plugin that integrates with gsuite.
  - KingOfCoders 4y ago
    Would a hardware token have prevented this (noob question)?
    - dijit 4y ago
      It's a fair question, but likely not. There are "application credentials" or "API tokens" that get generated for various purposes and live long. There are many programs that do not support "credential helpers", abd those would need you to input an API token that is generated usually just once and has almost always has enough permission to clone a repository (at a minimum). The `hub` and `lab` CLI tools work exactly in this manner. If you are familiar with intellij: certain plugins that integrate with github or gitlab also require the generation of an access token so that you can do API calls to see merge requests and such: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token https://docs.github.com/en/authentication/keeping-your-accou... & https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html https://docs.gitlab.com/ee/user/profile/personal_access_toke... Applications that support credential helpers are docker (for optionally authenticating with docker registries) and git itself.
      - KingOfCoders 4y ago
        I thought "Maybe" because it mentioned "Slack employee tokens".
    - jchw 4y ago
      Is it possible for a hardware token to prevent this in theory? You could use a hardware token to compute challenge-response answers without having external access to the private key matter, as opposed to using simple secret API keys like we do today. This is basically how an HSM works too. But does it solve the problem? It prevents the keymatter from leaking, but the problem is that the API key was put somewhere sensitive and then stolen. Presumably you'd need access to be able to use the keymatter from any location where you'd normally need API keys: CI machines, developer workstations, production, etc. Worse, if you have some machines (CI, HSM) that are accessed over network traffic and that traffic is secured with an API key, you are back to square one again, where a stolen API key allows for continuous access. Would this help for auditing? No, not really. You can already have plenty of auditing and alerting to try to prevent exfiltration anywhere an API key is used. In other words, probably not. It might help a little bit just by making exploitation harder, in optimal circumstances, but it doesn't fundamentally change the problem; if dev workstations or CI are compromised, you're going to have a bad time.
- cj 4y ago
  > Is it wise to simply keep private repositories away from GitHub at this point? This sounds like the equivalent of "Use MacOS because most viruses target Windows". Which some people do, and yes it probably lowers your risk, but there are ways to use Windows (and Github) securely. If you're really paranoid about your source code, you should probably self-host your own instance of Github/Gitlab/etc (and lock it down through VPNs, IP white lists, etc) rather than using their cloud service. This approach also gives you first hand access to logs that would expose internal bad actors, if that's included in your threat model.
  - lrvick 4y ago
    > there are ways to use Windows (and Github) securely. No proprietary software you have no control over or ability to audit can ever be considered secure.
    - ElevenLathe 4y ago
      It depends, as always, on the threat model you are trying to defend against.
      - lrvick 4y ago
        If you give a third party essentially limitless access to execute arbitrary code on your hardware, then you cannot really assert anything with certainty about that machine. That makes meeting most/all threat models difficult.
        ElevenLathe 4y ago
        I get what you're saying but if my threat model doesn't include Microsoft or state-level actors that can co-opt Microsoft or buy/develop hoarded Windows zero-days, then a patched Windows box is probably good enough.
    - intelVISA 4y ago
      Fuzzing any 'enterprise' SaaS is a frightful experience.
- jerf 4y ago
  The most basic way to analyze security is to look at it in terms of cost to penetrate security versus the benefit to the attacker. Things are not a binary "secure" or "not secure", they are secure when the expected costs are greater than the benefits, preferably by a comfortable margin. The problem is that there are some resources where the benefit to the attacker goes up faster than the company can afford to cover, and the company has to cover all the attack avenues. You can imagine that the value of Slack's source code can be high to certain people, and it can be difficult to completely seal off something like that when source code by its nature pretty much has to be distributed to hundreds or thousands of people. (At least overall; any given source may not be that distributed, but there will be thousands with some sort of access.) Major corporations basically have to act as if source compromise is inevitable. Best practices obviously include partitioning who can have access to what (perhaps the best practical argument against a one-company mono-repo, if de facto it has to be broken up by access permissions anyhow) and not including anything in the repo that is directly security catastrophic if it gets out (security secrets mostly), but this is still limiting blast radius rather than "solving" the problem. It's a perfect storm of being extremely expensive to cover, in terms of money, internal process friction, and having a lot of heterogeneous vectors you need to cover, and for certain companies, being very high value to a number of different kinds of entities. Being in GitHub specifically would only matter if GitHub was itself compromised. GitHub can not do very much about legit credentials being stolen through completely non-GitHub related means. (Most things, if not all things, that might leap to your mind will also block legitimate usage quite a bit. No matter what crazy thing you can imagine, someone's probably doing it somewhere and it's probably mission critical.) I expect there have been a great deal more source code compromises than we've heard about.
openplatypus 4y ago
What's the most interesting in this whole write-up is this: > Security update hidden from search engines Slack is selectively and deliberately limiting public access (discoverability) to the security breach announcements.
- have_faith 4y ago
  > marked with 'noindex' How do they even provide a plausible excuse for this? very bad look.
  - r4vik 4y ago
    it's totally plausible. and best practice. If you have a localised blog for multiple regions you will probably have your main blog set up to be indexed and your regional blogs set up to be no-indexed so that Google isn't indexing the same stuff twice.
    - sleepychu 4y ago
      > For starters, this "news" item doesn't appear on the company's international news blog aside other articles, at the time of writing. From the linked article.
      - kindofpenguin 4y ago
        They likely published it as soon as they could and translations will be a couple of days behind. Other blog posts will be pre-planned and pre-translated. The noindex on non-us versions is probably also related to it not being fully translated. Check in a few days to see if I'm right
        nanis 4y ago
        > BleepingComputer further observed that the "meta" tag containing the "noindex" attribute was itself placed towards the bottom within the page's HTML code, in an elongated line that overflows without breaking. Placing meta tags at the bottom, out of sight ... an _interesting_ choice given the nature of the news.
Suvitruf 4y ago
> No downloaded repositories contained customer data, means to access customer data, or Slack’s primary codebase.
tedk-42 4y ago
Doesn't sound too bad to be honest
BoardsOfCanada 4y ago
I hope microsoft can have a look and improve teams.
- blackoil 4y ago
  There was a project to completely rewrite Teams UI. no idea where is that.
  - brynjolf 4y ago
    Teams is written in Angularjs aka Angular 1 [0] They rewrote parts of it to React when they made the weird half client for Windows 11. Not sure why those two clients haven't merged yet. https://news.ycombinator.com/item?id=27633756 https://news.ycombinator.com/item?id=27633756
    - saiya-jin 4y ago
      That might explain why its so damn slow for such basic text things... not even going into bugs
    - tgv 4y ago
      Angularjs? I'm impressed. I've got to maintain (infrequently) a Django monstrosity with a (small) Angularjs front-end, and I've always had a much harder time fixing Angularjs than e.g. Vue, even though there is relatively little Angular code. They must have designed it properly from the start.
- gchamonlive 4y ago
  I wish, but teams is bad not because Microsoft isn't capable of fixing it. It is bad because regardless of how infuriating and frustrating the user experience is, people are still gonna be forced to use it under the massive pressure of Microsofts weight.
- modo_mario 4y ago
  I neither expect it nor hope so. I still remember when they bought Skype. Said they'd keep the Linux support to stem community worries then dropped it like a brick about a year after.
nickjj 4y ago
They say no customer data was compromised and are investigating "potential impact" to customers. The potential impact here is an attacker now has access to some of their code which could let them find and take advantage of vulnerabilities resulting in customer data being accessed. Technically "potential impact" is correct but I think companies often underplay how severe a source code leak is. It's the exact blueprint of how their app is built.
- moonchrome 4y ago
  OK so does this mean that OSS is severely vulnerable by default ?
  - thot_experiment 4y ago
    I don't know, I guess if one known malicious actor having access to a piece of information is the same as everyone having access to it, then yes.
  - nickjj 4y ago
    > OK so does this mean that OSS is vulnerable by default ? No, but I don't think that's a fair comparison. An open source tool or app is open source by choice so they have an advantage of knowing what they're getting into. In Slack's case maybe they have a bunch of undocumented APIs which are publicly accessible and now with access to the source code you know what they are, and when you hit them they result in customer data being returned. It runs in their production environment off the live site so it doesn't involve anything crazy like you needing VPN access to their DB to get production data. That's just 1 basic example of what could happen when a private code base becomes public due to a leak. I'd like to think an open source site wouldn't do that because at a fundamental level the app is built in the open. Also, there's likely many sets of eyes from different folks looking at it from different angles. Having undocumented publicly accessible API endpoints isn't an option in an open source world but a private code base could maybe get by with security through obscurity on a few things thinking "well, the code is private...".
    - 4y ago
hoseja 4y ago
Do they not have them anymore?
captn3m0 4y ago
Timeline matches with the Travis breach. So far likely impacted: 1. Slack 2. Okta (https://news.ycombinator.com/item?id=34081154 https://news.ycombinator.com/item?id=34081154) 3. Coa (https://github.com/veged/coa/issues/99#issuecomment-961696885 https://github.com/veged/coa/issues/99#issuecomment-96169688...)
- dijit 4y ago
  Add yesterdays announcement about CircleCI[0] and you have even more plausible options. [0]: https://circleci.com/blog/january-4-2023-security-alert/ https://circleci.com/blog/january-4-2023-security-alert/
  - rad_gruchalski 4y ago
    Wow wow wow. It would have been great for CircleCI to actually send an email with this info. I have a couple of OSS repos with them…
    - baltoo 4y ago
      They did. Or at least I got it
      - flanbiscuit 4y ago
        I got one at 9:31pm Eastern Time last night (Jan 4, 2022): here's what it said: We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well. Action request: Out of an abundance of caution, we strongly recommend that all customers take the following actions: - Immediately rotate any and all secrets stored in CircleCI. These may be stored in project environment variables or in contexts. - We also recommend customers review internal logs for their systems for any unauthorized access starting from December 21, 2022 through today, January 4, 2023, or upon completion of your secrets rotation. Additionally, if your project uses Project API tokens, we have invalidated those and you will need to replace them. You can find more information on how to do that in our documentation here. We apologize for any disruption to your work. We take the security of our systems and our customers’ systems extremely seriously. While we are actively investigating this incident, we are committed to sharing more details with customers in the coming days. Thank you for your urgent attention to rotating your secrets.
blkhp19 4y ago
So, where can these be found? I’d like to dig through the code.
halukakin 4y ago
I hope github would start allowing all users to use ip whitelisting features. I understand it is a sell point for enterprise accounts, but tokens/passwords are not secure enough in today's environment.
rvz 4y ago
Like I said before [0][1], a reminder that private repositories are not private on GitHub. A great time to self-host. [0] https://news.ycombinator.com/item?id=34232347 https://news.ycombinator.com/item?id=34232347 [1] https://news.ycombinator.com/item?id=23102942 https://news.ycombinator.com/item?id=23102942
mytailorisrich 4y ago
I don't understand why some companies host their code externally on Github. For a software company the code repo is the #1 asset.
- cj 4y ago
  Hate to nitpick, but is code really the #1 asset of most software companies? if AWS/Amazon's entire source code were leaked tomorrow, how would competitors use it to their advantage? Would it help Lyft at all if they gained access to Uber's source code? Probably not. Is some startup going to be able to take Slack's source code and use it to build a competing service? Probably not, it would be easier to write the code from scratch and reverse engineer features. Source code is definitely a company asset, but most companies wouldn't be incredibly damaged or threatened in any meaningful way if their source code were exposed... unless said source code exposes security flaws that are used for some attack on the service. But security through obscurity (e.g. keeping insecure code secret) is not a valid way to keep things secure.
  - londons_explore 4y ago
    There are instances of an overseas company stealing a large amount of code and just running it as-is. As long as you do it outside the jurisdiction of the USA, you'll probably get away with it. I have seen two instances of this in my professional career (can't talk about either I'm afraid!), but in both cases it had a pretty big impact on the original owner of the code, and in neither case did the company manage to get any compensation.
  - mytailorisrich 4y ago
    Leak of code can be very damaging, not just because it might reveal vulnerabilities but also because it will reveal IP and will allow all sorts of 'IP theft'. Then, losing the code altogether is the death of the company.
    - cj 4y ago
      I guess my point is that for 90%+ of software companies, you can very easily reverse engineer the software or just rewrite it from scratch. If I were given $50 million to build a Slack competitor, even if I had Slack's source code I can't imagine it would be that helpful. More than likely it would be easier to rewrite and re-architect in order to not inherit all of the technical debt of very old software. Not to mention in most cases, access to the source code doesn't mean you can easily recreate the AWS / Google Cloud / Azure environments necessary to run the source code reliably with scalability. Edit: There are obvious exceptions to this, but most apps these days don't require a whole lot of proprietary algorithms that can't be easily cloned without source code access.
shanebellone 4y ago
Does it actually matter? I might be overlooking something but... Slack is more than its code. The product is really the aggregate of their engineer's knowledge and internal processes. It's not practical to steal code and build a business or spinoff product around it. The only legitimate threat seems to be the potential for exploitation. I suppose this might also threaten any backend improvements (economic leverage) they made with proprietary algorithms.
- nightpool 4y ago
  It's not even their main codebase—if you read closely, it says it was only private code repos stored on Github.com, and that their main repo wasn't.
  - shanebellone 4y ago
    I understand. Just posing a question for discussion.
muglug 4y ago
I work for Slack, opinions my own. IMO the headline should be updated — much less than 1% of Slack’s private code can be accessed via github.com.
- jbotdev 4y ago
  The article did mention that this breach did not include the “primary codebase”. Where is the rest of the code hosted? I’m guessing some private Git server (e.g. “on-prem” GitHub/GitLab Enterprise)?
  - nightpool 4y ago
    I would be surprised if Slack disclosed publicly exactly which source code provider they used, but something hosted like Github EE sounds like a very reasonable assumption given their response. Maybe someone was given a goal to migrate their main source code repo to a shared Salesforce instance for security or cross team integration reasons, and everything left on Github.com was just the stuff too unimportant to migrate. If so, seems like it payed off for them in this case.
- oxfordmale 4y ago
  If you don't communicate, you lose the narrative. As the article highlights, it seems Slack has gone out of its way to exclude this incident report from appearing in Google searches. As a result this article will likely appear first. It is also worth nothing that other major breaches (LastPass) were preceded by source code breaches. The corresponding incident reports also included re-assurances that there was no impact on customer safety. The Slack incident report doesn't specify what type of Github repos were accessed, so it is hard to judge if any sensitive code has been leaked. From the recent LastPass security incident report: Based on our investigation to date, we have learned that an unknown threat actor accessed a cloud-based storage environment leveraging information obtained from the incident we previously disclosed in August of 2022 (source code breach)
  - muglug 4y ago
    Slack communicated about the breach here: https://slack.com/blog/news/slack-security-update https://slack.com/blog/news/slack-security-update There is no `noindex` tag on that page (check the source). There is a `noindex` tag on an alternate "en-gb" version of that page: https://slack.com/intl/en-gb/blog/news/slack-security-update https://slack.com/intl/en-gb/blog/news/slack-security-update This is to be expected — it's presumably SEO equivalent to specifying a canonical URL.
realworldperson 4y ago
[dead]
janosdebugs 4y ago
It would be so nice if GitHub deprecated their one key takes all and would let us create per-org personal access tokens. Also, organizations having power over these access tokens.
- CubsFan1060 4y ago
  They do this now. I'll find a link. Link below. Organizations can choose to allow these, require approval before they are valid, and deny the use of traditional tokens. They are actually per repo. https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/ https://github.blog/2022-10-18-introducing-fine-grained-pers...
  - jrochkind1 4y ago
    Note "Public Beta". Seems like a good step forward; I'm still somewhat shocked it took this long, people have been pointing out the security issues for at least 5 years.
ergonaught 4y ago
Offtopic, but, I made the mistake of trying to read this on my mobile phone this morning, where I don't use adblockers per se. Served at least 6 ads, even ignoring the nonsense at the bottom of the page. So annoying and disruptive that I didn't manage to read the story, and will find the details somewhere else. We really needed a reaction against this nonsense which wasn't just adblockers. Bummer.
- idonotknowwhy 4y ago
  This is why Firefox is my default browser on Android and I use Ublock Origin. Lack of real Firefox on iOS is the absolute deal breaker for me switching to iPhone
  - acdha 4y ago
    Firefox Focus works as a Safari content blocker: https://www.mozilla.org/en-US/firefox/browsers/mobile/focus/ https://www.mozilla.org/en-US/firefox/browsers/mobile/focus/ It's quite effective at defanging the annoying ads without breaking things I care about.
IgorPartola 4y ago
Stolen means Slack no longer has something that the thief has, right?
- gkedzierski 4y ago
  By your definition piracy would not be a theft.
  - IgorPartola 4y ago
    Correct. It isn’t. It is copyright infringement, and there is a lot of case law to support this.
danwee 4y ago
We've migrated from Slack to Mattermost some time ago. Best decision ever. Not sure why companies keep using Slack nowadays. It's like those companies many years ago using HipChat instead of Slack. Slack's is the HipChat of today.
- tourist2d 4y ago
  What does Mattermost offer which Slack doesn't?
  - paulryanrogers 4y ago
    Open source, can be self hosted, you can secure it yourself and how you want (for better or worse)
- michaelmior 4y ago
  I think part of the problem is that there are a lot of integrations from Slack that aren't available on Mattermost. You can certainly make the argument that those integrations won't be developed until more users abandon Slack for Mattermost, but for some it's not worth the loss of functionality.
- dx034 4y ago
  Self-hosted Mattermost and Gitlab can be done reliably at very low cost, had better uptime than their cloud services for us and would've likely prevented leaks like these. At least you can add some layers of security for self-hosted services so that stolen tokens aren't enough to steal code or messages.
  - splittingTimes 4y ago
    Do you have write up or guide somewhere on how to specifically set this up, configure and add security?
    - dx034 4y ago
      We run it in our VPN with SSO for authentication. You'd need a company device, user credentials and a location that's whitelisted to access. For smaller companies something like Cloudflare Teams and/or Tailscale would probably also add a good layer of extra security. I guess it's important to have some device whitelisting in addition to credentails, so that stolen credentials alone are not enough to access code or chat.
ianpurton 4y ago
Slack could open source their code anyway. Slacks competitive advantage is their network of users which is difficult to copy.
- vaseem 4y ago
  Source code is stolen to explore vulnerabilities and develop zero day exploits. More in this podcast https://darknetdiaries.com/episode/19/ https://darknetdiaries.com/episode/19/
  - michaelmior 4y ago
    Arguably vulnerabilities would be more likely to be found and disclosed if the code was open source.
  - phpisthebest 4y ago
    That old classic chestnut of lies by Closed Source companies intent on making CIOs believe Open Source is not secure...
  - saagarjha 4y ago
    That’s not typically how these exploits are developed.
    - pionar 4y ago
      Not typically, but that is how the LastPass breach occurred.
- ltbarcly3 4y ago
  Nope, its a bunch of little communities that are centrally managed and can force their users to a new platform easily. Network effects are minimal, sunk cost of integrations is a greater barrier to moving off platform but that is pretty weak as well.
  - schnable 4y ago
School-Cotton 4y ago
Oh good, maybe someone will finally fix their many annoying bugs
bjd2385 4y ago
This is exactly why secrets should not be stashed in any Git repository. Granted, I'm not sure they're much safer in other managed services dedicated to protecting our secrets, with the news of several breaches as of late :/
j45 4y ago
It remains surprising to see such private and valuable codebases hosted in the cloud in an external parties service. The cloud remains someone else’s shared computer.
marban 4y ago
Streisand effect anyone?

347 points by mindracer 4y ago

This is how LastPass started the efficient downward spiral, no? This isn't shocking, but also is. Is it wise to simply keep private repositories away from GitHub at this point? It seems the best way to avoid being drawn in with the rest as a target. Internal bad actors? So location wouldn't matter?

verdverm 4y ago
If you read the details... > The incident involves threat actors gaining access to Slack's externally hosted GitHub repositories via a "limited" number of Slack employee tokens that were stolen. > While some of Slack's private code repositories were breached, Slack’s primary codebase and customer data remains unaffected, according to the company.
- throwaway010523 4y ago
  Wouldn't private git hosting add another barrier for the attackers? In addition to the tokens, they would also need to somehow gain access to the network through VPN or through an opening in the firewall?
  - hardware2win 4y ago
    Github repos can require using company's vpn
  - trollied 4y ago
    Yes. By way of example, my employer does this: * gitlab hosted internally * VPN needed to access gitlab * VPN requires a gsuite login, with 2FA * You can only login to a gsuite account on a employer-provided machine (so no access to anything on a non-employer machine, even email) You'd have to steal a company laptop, or social engineer yourself into a building with a desktop machine, to even start to get near the code repository. The "must login from a employer-provided machine" thing can be disabled on a per-user basis by remote policy (via an administrator), if needed.
    - tcmb 4y ago
      > You can only login to a gsuite account on a employer-provided machine How is this achieved?
      - trollied 4y ago
        There's an Endpoint Verification plugin that integrates with gsuite.
- KingOfCoders 4y ago
  Would a hardware token have prevented this (noob question)?
  - dijit 4y ago
    It's a fair question, but likely not. There are "application credentials" or "API tokens" that get generated for various purposes and live long. There are many programs that do not support "credential helpers", abd those would need you to input an API token that is generated usually just once and has almost always has enough permission to clone a repository (at a minimum). The `hub` and `lab` CLI tools work exactly in this manner. If you are familiar with intellij: certain plugins that integrate with github or gitlab also require the generation of an access token so that you can do API calls to see merge requests and such: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token https://docs.github.com/en/authentication/keeping-your-accou... & https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html https://docs.gitlab.com/ee/user/profile/personal_access_toke... Applications that support credential helpers are docker (for optionally authenticating with docker registries) and git itself.
    - KingOfCoders 4y ago
      I thought "Maybe" because it mentioned "Slack employee tokens".
  - jchw 4y ago
    Is it possible for a hardware token to prevent this in theory? You could use a hardware token to compute challenge-response answers without having external access to the private key matter, as opposed to using simple secret API keys like we do today. This is basically how an HSM works too. But does it solve the problem? It prevents the keymatter from leaking, but the problem is that the API key was put somewhere sensitive and then stolen. Presumably you'd need access to be able to use the keymatter from any location where you'd normally need API keys: CI machines, developer workstations, production, etc. Worse, if you have some machines (CI, HSM) that are accessed over network traffic and that traffic is secured with an API key, you are back to square one again, where a stolen API key allows for continuous access. Would this help for auditing? No, not really. You can already have plenty of auditing and alerting to try to prevent exfiltration anywhere an API key is used. In other words, probably not. It might help a little bit just by making exploitation harder, in optimal circumstances, but it doesn't fundamentally change the problem; if dev workstations or CI are compromised, you're going to have a bad time.
cj 4y ago
> Is it wise to simply keep private repositories away from GitHub at this point? This sounds like the equivalent of "Use MacOS because most viruses target Windows". Which some people do, and yes it probably lowers your risk, but there are ways to use Windows (and Github) securely. If you're really paranoid about your source code, you should probably self-host your own instance of Github/Gitlab/etc (and lock it down through VPNs, IP white lists, etc) rather than using their cloud service. This approach also gives you first hand access to logs that would expose internal bad actors, if that's included in your threat model.
- lrvick 4y ago
  > there are ways to use Windows (and Github) securely. No proprietary software you have no control over or ability to audit can ever be considered secure.
  - ElevenLathe 4y ago
    It depends, as always, on the threat model you are trying to defend against.
    - lrvick 4y ago
      If you give a third party essentially limitless access to execute arbitrary code on your hardware, then you cannot really assert anything with certainty about that machine. That makes meeting most/all threat models difficult.
      - ElevenLathe 4y ago
        I get what you're saying but if my threat model doesn't include Microsoft or state-level actors that can co-opt Microsoft or buy/develop hoarded Windows zero-days, then a patched Windows box is probably good enough.
  - intelVISA 4y ago
    Fuzzing any 'enterprise' SaaS is a frightful experience.
jerf 4y ago
The most basic way to analyze security is to look at it in terms of cost to penetrate security versus the benefit to the attacker. Things are not a binary "secure" or "not secure", they are secure when the expected costs are greater than the benefits, preferably by a comfortable margin. The problem is that there are some resources where the benefit to the attacker goes up faster than the company can afford to cover, and the company has to cover all the attack avenues. You can imagine that the value of Slack's source code can be high to certain people, and it can be difficult to completely seal off something like that when source code by its nature pretty much has to be distributed to hundreds or thousands of people. (At least overall; any given source may not be that distributed, but there will be thousands with some sort of access.) Major corporations basically have to act as if source compromise is inevitable. Best practices obviously include partitioning who can have access to what (perhaps the best practical argument against a one-company mono-repo, if de facto it has to be broken up by access permissions anyhow) and not including anything in the repo that is directly security catastrophic if it gets out (security secrets mostly), but this is still limiting blast radius rather than "solving" the problem. It's a perfect storm of being extremely expensive to cover, in terms of money, internal process friction, and having a lot of heterogeneous vectors you need to cover, and for certain companies, being very high value to a number of different kinds of entities. Being in GitHub specifically would only matter if GitHub was itself compromised. GitHub can not do very much about legit credentials being stolen through completely non-GitHub related means. (Most things, if not all things, that might leap to your mind will also block legitimate usage quite a bit. No matter what crazy thing you can imagine, someone's probably doing it somewhere and it's probably mission critical.) I expect there have been a great deal more source code compromises than we've heard about.

openplatypus 4y ago

What's the most interesting in this whole write-up is this: > Security update hidden from search engines Slack is selectively and deliberately limiting public access (discoverability) to the security breach announcements.

have_faith 4y ago
> marked with 'noindex' How do they even provide a plausible excuse for this? very bad look.
- r4vik 4y ago
  it's totally plausible. and best practice. If you have a localised blog for multiple regions you will probably have your main blog set up to be indexed and your regional blogs set up to be no-indexed so that Google isn't indexing the same stuff twice.
  - sleepychu 4y ago
    > For starters, this "news" item doesn't appear on the company's international news blog aside other articles, at the time of writing. From the linked article.
    - kindofpenguin 4y ago
      They likely published it as soon as they could and translations will be a couple of days behind. Other blog posts will be pre-planned and pre-translated. The noindex on non-us versions is probably also related to it not being fully translated. Check in a few days to see if I'm right
      - nanis 4y ago
        > BleepingComputer further observed that the "meta" tag containing the "noindex" attribute was itself placed towards the bottom within the page's HTML code, in an elongated line that overflows without breaking. Placing meta tags at the bottom, out of sight ... an _interesting_ choice given the nature of the news.

Suvitruf 4y ago

> No downloaded repositories contained customer data, means to access customer data, or Slack’s primary codebase.

tedk-42 4y ago

Doesn't sound too bad to be honest

BoardsOfCanada 4y ago

I hope microsoft can have a look and improve teams.

blackoil 4y ago
There was a project to completely rewrite Teams UI. no idea where is that.
- brynjolf 4y ago
  Teams is written in Angularjs aka Angular 1 [0] They rewrote parts of it to React when they made the weird half client for Windows 11. Not sure why those two clients haven't merged yet. https://news.ycombinator.com/item?id=27633756 https://news.ycombinator.com/item?id=27633756
  - saiya-jin 4y ago
    That might explain why its so damn slow for such basic text things... not even going into bugs
  - tgv 4y ago
    Angularjs? I'm impressed. I've got to maintain (infrequently) a Django monstrosity with a (small) Angularjs front-end, and I've always had a much harder time fixing Angularjs than e.g. Vue, even though there is relatively little Angular code. They must have designed it properly from the start.
gchamonlive 4y ago
I wish, but teams is bad not because Microsoft isn't capable of fixing it. It is bad because regardless of how infuriating and frustrating the user experience is, people are still gonna be forced to use it under the massive pressure of Microsofts weight.
modo_mario 4y ago
I neither expect it nor hope so. I still remember when they bought Skype. Said they'd keep the Linux support to stem community worries then dropped it like a brick about a year after.

nickjj 4y ago

They say no customer data was compromised and are investigating "potential impact" to customers. The potential impact here is an attacker now has access to some of their code which could let them find and take advantage of vulnerabilities resulting in customer data being accessed. Technically "potential impact" is correct but I think companies often underplay how severe a source code leak is. It's the exact blueprint of how their app is built.

moonchrome 4y ago
OK so does this mean that OSS is severely vulnerable by default ?
- thot_experiment 4y ago
  I don't know, I guess if one known malicious actor having access to a piece of information is the same as everyone having access to it, then yes.
- nickjj 4y ago
  > OK so does this mean that OSS is vulnerable by default ? No, but I don't think that's a fair comparison. An open source tool or app is open source by choice so they have an advantage of knowing what they're getting into. In Slack's case maybe they have a bunch of undocumented APIs which are publicly accessible and now with access to the source code you know what they are, and when you hit them they result in customer data being returned. It runs in their production environment off the live site so it doesn't involve anything crazy like you needing VPN access to their DB to get production data. That's just 1 basic example of what could happen when a private code base becomes public due to a leak. I'd like to think an open source site wouldn't do that because at a fundamental level the app is built in the open. Also, there's likely many sets of eyes from different folks looking at it from different angles. Having undocumented publicly accessible API endpoints isn't an option in an open source world but a private code base could maybe get by with security through obscurity on a few things thinking "well, the code is private...".
  - 4y ago

hoseja 4y ago

Do they not have them anymore?

captn3m0 4y ago

Timeline matches with the Travis breach. So far likely impacted: 1. Slack 2. Okta (https://news.ycombinator.com/item?id=34081154 https://news.ycombinator.com/item?id=34081154) 3. Coa (https://github.com/veged/coa/issues/99#issuecomment-961696885 https://github.com/veged/coa/issues/99#issuecomment-96169688...)

dijit 4y ago
Add yesterdays announcement about CircleCI[0] and you have even more plausible options. [0]: https://circleci.com/blog/january-4-2023-security-alert/ https://circleci.com/blog/january-4-2023-security-alert/
- rad_gruchalski 4y ago
  Wow wow wow. It would have been great for CircleCI to actually send an email with this info. I have a couple of OSS repos with them…
  - baltoo 4y ago
    They did. Or at least I got it
    - flanbiscuit 4y ago
      I got one at 9:31pm Eastern Time last night (Jan 4, 2022): here's what it said: We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well. Action request: Out of an abundance of caution, we strongly recommend that all customers take the following actions: - Immediately rotate any and all secrets stored in CircleCI. These may be stored in project environment variables or in contexts. - We also recommend customers review internal logs for their systems for any unauthorized access starting from December 21, 2022 through today, January 4, 2023, or upon completion of your secrets rotation. Additionally, if your project uses Project API tokens, we have invalidated those and you will need to replace them. You can find more information on how to do that in our documentation here. We apologize for any disruption to your work. We take the security of our systems and our customers’ systems extremely seriously. While we are actively investigating this incident, we are committed to sharing more details with customers in the coming days. Thank you for your urgent attention to rotating your secrets.

blkhp19 4y ago

So, where can these be found? I’d like to dig through the code.

halukakin 4y ago

I hope github would start allowing all users to use ip whitelisting features. I understand it is a sell point for enterprise accounts, but tokens/passwords are not secure enough in today's environment.

rvz 4y ago

Like I said before [0][1], a reminder that private repositories are not private on GitHub. A great time to self-host. [0] https://news.ycombinator.com/item?id=34232347 https://news.ycombinator.com/item?id=34232347 [1] https://news.ycombinator.com/item?id=23102942 https://news.ycombinator.com/item?id=23102942

mytailorisrich 4y ago

I don't understand why some companies host their code externally on Github. For a software company the code repo is the #1 asset.

cj 4y ago
Hate to nitpick, but is code really the #1 asset of most software companies? if AWS/Amazon's entire source code were leaked tomorrow, how would competitors use it to their advantage? Would it help Lyft at all if they gained access to Uber's source code? Probably not. Is some startup going to be able to take Slack's source code and use it to build a competing service? Probably not, it would be easier to write the code from scratch and reverse engineer features. Source code is definitely a company asset, but most companies wouldn't be incredibly damaged or threatened in any meaningful way if their source code were exposed... unless said source code exposes security flaws that are used for some attack on the service. But security through obscurity (e.g. keeping insecure code secret) is not a valid way to keep things secure.
- londons_explore 4y ago
  There are instances of an overseas company stealing a large amount of code and just running it as-is. As long as you do it outside the jurisdiction of the USA, you'll probably get away with it. I have seen two instances of this in my professional career (can't talk about either I'm afraid!), but in both cases it had a pretty big impact on the original owner of the code, and in neither case did the company manage to get any compensation.
- mytailorisrich 4y ago
  Leak of code can be very damaging, not just because it might reveal vulnerabilities but also because it will reveal IP and will allow all sorts of 'IP theft'. Then, losing the code altogether is the death of the company.
  - cj 4y ago
    I guess my point is that for 90%+ of software companies, you can very easily reverse engineer the software or just rewrite it from scratch. If I were given $50 million to build a Slack competitor, even if I had Slack's source code I can't imagine it would be that helpful. More than likely it would be easier to rewrite and re-architect in order to not inherit all of the technical debt of very old software. Not to mention in most cases, access to the source code doesn't mean you can easily recreate the AWS / Google Cloud / Azure environments necessary to run the source code reliably with scalability. Edit: There are obvious exceptions to this, but most apps these days don't require a whole lot of proprietary algorithms that can't be easily cloned without source code access.

shanebellone 4y ago

Does it actually matter? I might be overlooking something but... Slack is more than its code. The product is really the aggregate of their engineer's knowledge and internal processes. It's not practical to steal code and build a business or spinoff product around it. The only legitimate threat seems to be the potential for exploitation. I suppose this might also threaten any backend improvements (economic leverage) they made with proprietary algorithms.

nightpool 4y ago
It's not even their main codebase—if you read closely, it says it was only private code repos stored on Github.com, and that their main repo wasn't.
- shanebellone 4y ago
  I understand. Just posing a question for discussion.

muglug 4y ago

I work for Slack, opinions my own. IMO the headline should be updated — much less than 1% of Slack’s private code can be accessed via github.com.

jbotdev 4y ago
The article did mention that this breach did not include the “primary codebase”. Where is the rest of the code hosted? I’m guessing some private Git server (e.g. “on-prem” GitHub/GitLab Enterprise)?
- nightpool 4y ago
  I would be surprised if Slack disclosed publicly exactly which source code provider they used, but something hosted like Github EE sounds like a very reasonable assumption given their response. Maybe someone was given a goal to migrate their main source code repo to a shared Salesforce instance for security or cross team integration reasons, and everything left on Github.com was just the stuff too unimportant to migrate. If so, seems like it payed off for them in this case.
oxfordmale 4y ago
If you don't communicate, you lose the narrative. As the article highlights, it seems Slack has gone out of its way to exclude this incident report from appearing in Google searches. As a result this article will likely appear first. It is also worth nothing that other major breaches (LastPass) were preceded by source code breaches. The corresponding incident reports also included re-assurances that there was no impact on customer safety. The Slack incident report doesn't specify what type of Github repos were accessed, so it is hard to judge if any sensitive code has been leaked. From the recent LastPass security incident report: Based on our investigation to date, we have learned that an unknown threat actor accessed a cloud-based storage environment leveraging information obtained from the incident we previously disclosed in August of 2022 (source code breach)
- muglug 4y ago
  Slack communicated about the breach here: https://slack.com/blog/news/slack-security-update https://slack.com/blog/news/slack-security-update There is no `noindex` tag on that page (check the source). There is a `noindex` tag on an alternate "en-gb" version of that page: https://slack.com/intl/en-gb/blog/news/slack-security-update https://slack.com/intl/en-gb/blog/news/slack-security-update This is to be expected — it's presumably SEO equivalent to specifying a canonical URL.

realworldperson 4y ago

[dead]

janosdebugs 4y ago

It would be so nice if GitHub deprecated their one key takes all and would let us create per-org personal access tokens. Also, organizations having power over these access tokens.

CubsFan1060 4y ago
They do this now. I'll find a link. Link below. Organizations can choose to allow these, require approval before they are valid, and deny the use of traditional tokens. They are actually per repo. https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/ https://github.blog/2022-10-18-introducing-fine-grained-pers...
- jrochkind1 4y ago
  Note "Public Beta". Seems like a good step forward; I'm still somewhat shocked it took this long, people have been pointing out the security issues for at least 5 years.

ergonaught 4y ago

Offtopic, but, I made the mistake of trying to read this on my mobile phone this morning, where I don't use adblockers per se. Served at least 6 ads, even ignoring the nonsense at the bottom of the page. So annoying and disruptive that I didn't manage to read the story, and will find the details somewhere else. We really needed a reaction against this nonsense which wasn't just adblockers. Bummer.

idonotknowwhy 4y ago
This is why Firefox is my default browser on Android and I use Ublock Origin. Lack of real Firefox on iOS is the absolute deal breaker for me switching to iPhone
- acdha 4y ago
  Firefox Focus works as a Safari content blocker: https://www.mozilla.org/en-US/firefox/browsers/mobile/focus/ https://www.mozilla.org/en-US/firefox/browsers/mobile/focus/ It's quite effective at defanging the annoying ads without breaking things I care about.

IgorPartola 4y ago

Stolen means Slack no longer has something that the thief has, right?

gkedzierski 4y ago
By your definition piracy would not be a theft.
- IgorPartola 4y ago
  Correct. It isn’t. It is copyright infringement, and there is a lot of case law to support this.

danwee 4y ago

We've migrated from Slack to Mattermost some time ago. Best decision ever. Not sure why companies keep using Slack nowadays. It's like those companies many years ago using HipChat instead of Slack. Slack's is the HipChat of today.

tourist2d 4y ago
What does Mattermost offer which Slack doesn't?
- paulryanrogers 4y ago
  Open source, can be self hosted, you can secure it yourself and how you want (for better or worse)
michaelmior 4y ago
I think part of the problem is that there are a lot of integrations from Slack that aren't available on Mattermost. You can certainly make the argument that those integrations won't be developed until more users abandon Slack for Mattermost, but for some it's not worth the loss of functionality.
dx034 4y ago
Self-hosted Mattermost and Gitlab can be done reliably at very low cost, had better uptime than their cloud services for us and would've likely prevented leaks like these. At least you can add some layers of security for self-hosted services so that stolen tokens aren't enough to steal code or messages.
- splittingTimes 4y ago
  Do you have write up or guide somewhere on how to specifically set this up, configure and add security?
  - dx034 4y ago
    We run it in our VPN with SSO for authentication. You'd need a company device, user credentials and a location that's whitelisted to access. For smaller companies something like Cloudflare Teams and/or Tailscale would probably also add a good layer of extra security. I guess it's important to have some device whitelisting in addition to credentails, so that stolen credentials alone are not enough to access code or chat.

ianpurton 4y ago

Slack could open source their code anyway. Slacks competitive advantage is their network of users which is difficult to copy.

vaseem 4y ago
Source code is stolen to explore vulnerabilities and develop zero day exploits. More in this podcast https://darknetdiaries.com/episode/19/ https://darknetdiaries.com/episode/19/
- michaelmior 4y ago
  Arguably vulnerabilities would be more likely to be found and disclosed if the code was open source.
- phpisthebest 4y ago
  That old classic chestnut of lies by Closed Source companies intent on making CIOs believe Open Source is not secure...
- saagarjha 4y ago
  That’s not typically how these exploits are developed.
  - pionar 4y ago
    Not typically, but that is how the LastPass breach occurred.
ltbarcly3 4y ago
Nope, its a bunch of little communities that are centrally managed and can force their users to a new platform easily. Network effects are minimal, sunk cost of integrations is a greater barrier to moving off platform but that is pretty weak as well.
- schnable 4y ago

School-Cotton 4y ago

Oh good, maybe someone will finally fix their many annoying bugs

bjd2385 4y ago

This is exactly why secrets should not be stashed in any Git repository. Granted, I'm not sure they're much safer in other managed services dedicated to protecting our secrets, with the news of several breaches as of late :/

j45 4y ago

It remains surprising to see such private and valuable codebases hosted in the cloud in an external parties service. The cloud remains someone else’s shared computer.

marban 4y ago

Streisand effect anyone?