4 ms·

For a bit of context: Matrix.org infrastructure has been hacked a second time in 24h, after restoring everything they went down again, story developing here: ht

by m_b 7y ago

For a bit of context: Matrix.org infrastructure has been hacked a second time in 24h, after restoring everything they went down again, story developing here: https://twitter.com/matrixdotorg/status/1116304867683905537 https://twitter.com/matrixdotorg/status/1116304867683905537

m_b 7y ago
The hacker is now doing a post-mortem in the GitHub issues of the project: https://github.com/matrix-org/matrix.org/issues https://github.com/matrix-org/matrix.org/issues
- nothrabannosir 7y ago
  This is gold... > I noticed in your blog post that you were talking about doing a postmortem and steps you need to take. As someone who is intimately familiar with your entire infrastructure, I thought I could help you out. > There I was, just going about my business, looking for ways I could get higher levels of access and explore your network more, when I stumbled across GPG keys that were used for signing your debian packages. It gave me many nefarious ideas. I would recommend that you don't keep any signing keys on production hosts, and instead do all of your signing in a secure environment.
  - paavoova 7y ago
    Why would they do this? It's pure negligence. I don't even sign anything important and still worry about my keys.
    - henvic 7y ago
      I have been asked twice or more why I insisted on not using a Continuous Integration environment for publishing some software releases that are installed by third-parties. My team was automating the infrastructure to build internal software and naturally they wanted to be able to simplify things. The idea that was proposed to me was the following: once I push a new version tag to GitHub, the deployment CI server is going to build and release it as an unstable version. Some important detail here: I use the same key to sign packages regardless if they are released as unstable or stable. That would mean that if someone, somehow, managed to push a tag that was pushed upstream to GitHub, hypothetically they would be able to eventually gain access to consumers machines (basically, developers) when the consumers update it after getting a notification telling them a new version is available. No way I'd allow this to happen, but I would not be surprised if most people just took this as an acceptable risk.
      - simias 7y ago
        Depending on your threat model I think that signing packages directly from your CI is acceptable, assuming that your CI runs is a reasonably isolated environment (e.g. on your company's LAN) and people who are able to trigger a release are correctly vetted. If I understand the parent comment correctly they were somehow shipping the release signing key on their production environment which is a whole other level of bad.
      - wcoenen 7y ago
        > That would mean that if someone, somehow, managed to push a tag that was pushed upstream to GitHub You have to define what the signature means. IMHO it is fine for it to mean "this software was built on our build server from a well-defined state of the source code, which is only changable by our employees and contractors, and for which we have the full change log". So I deploy the code signing key to build servers, which is the only place where it is used. I'm interested in what alternative meaning you would give to a signature. I have considered the possibility of tying it to the QA processes, but then a build can only be signed after checking it manually, which is problematic when many signatures are needed at multiple packaging layers (exe/dll, msi, setup.exe).
        rakoo 7y ago
        The problem is when a malicious package is produced, either because a flaw was introduced in the code, or because a dev machine was compromised, or _when the CI machine sad compromised_; the malicious package will be signed as if it were legit. One middle point between automated and manual signing is, as usual, key rotation: have the signing keys expire in a short duration of time (say 2 weeks) and manually push them every week, so that the window of attack is as small as possible.
        yolo1 7y ago
        What does a key rotation solve? Either your build server is compromised or it's not.
        notyourday 7y ago
  - lelf 7y ago
    Another gem: RRREEEEEEEE> I noticed you missed a doctype in your html page. In order for web browsers to know what type of html to render you should include a doctype. Thanks! matrixnotorg> @RRREEEEEEEE Thank you, I will consider that for the next release Edit: it got deleted But see also: https://github.com/matrixnotorg/matrixnotorg.github.io/pull/2 https://github.com/matrixnotorg/matrixnotorg.github.io/pull/...
    - justaj 7y ago
      Wait, did Github delete matrixnotorg's profile or did matrixnotorg? If Github deleted that profile, I don't really see that as being very hacker-friendly.
      - aeternus 7y ago
        Although 'hacker' is often used as a positive term on HN, breaking into a company's production server is clearly illegal activity and should not be condoned. If Github deleted the account, they are simply acting in accordance to published TOS & policy.
        justaj 7y ago
        If the attacker placed sensitive information on Github, that would indeed warrant a deletion of the account. However from what I saw from the archives, the attacker merely published details about Matrix.org's infrastructure and its vulnerabilities. Is that something that's against Github's ToS?
- m00dy 7y ago
  hilarious...
  - meruru 7y ago
    I can only think of the despair I would be in if I was in Aragorn's shoes
    - rabidrat 7y ago
      Arathorn (Aragorn is son of Arathorn).
      - meruru 7y ago
        Does being a parent prepare you to handle emergency situations like this better?
- deleted 7y ago
  [deleted]
- atsjie 7y ago
  > Escalation could have been avoided if developers only had the access they absolutely required and did not have root access to all of the servers. I would like to take a moment to thank whichever developer forwarded their agent to Flywheel. I'd feel so small if I were this developer right now :-| A couple of his issues appear to have to do with the use of SSH. An Ops-guy whom I worked with had setup a bastion host with ip whitelisting that automatically shut down after 1 hour. He didn't like it as his credo was "if you're using SSH when using a cloud provider you're probably doing something wrong"; meaning to say you should automate and be able to recreate any infra at all times with logs accessible without the need for SSH. I never forgot that.
  - robrtsql 7y ago
    Did you work with Rich Adams? https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/ https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-...
    - atsjie 7y ago
      no, but I wouldn't be surprised if my colleague got the idea from him. It was around the same time (end of 2014 I think).
- BartBoch 7y ago
  I have just checked quickly the comments and post-mortem, and I start wondering - it seems that the attack itself would be not really possible if Matrix would not be open source (as this would restrict access to the sensitive data)? Is that right?
  - t3chguy 7y ago
    This is not right at all, the hack was due to an outdated Jenkins instance and could have happened regardless of what other software was running on the infrastructure.
    - notyourday 7y ago
      Absolutely not. There's a zero reason for development infrastructure ( which includes Jenkins ) to have any connection to production outside a well defined "transfer this tarball" to a deploy staging server path during a known deploy window which cannot be controlled via development credentials.
F30 7y ago
I am highly skeptical when people taking about "rebuilding [the whole] infrastructure" in a few hours. Even more so when restoring all data from breached systems and before a thorough incident analysis. Show me the org which can just pull that off.
- pferde 7y ago
  It's quite common in enterprise to have a RTO of a few hours, and RPO of a few minutes, even for infrastructure with terrabytes of data. Of course, many moneys are paid for being able to do that.
- erikbye 7y ago
  This is doable with proper IaC implementation, and if your org does not have RPO/RTO on lock they're doing it wrong. Events like Matrix experienced now do not lead to panicked frenzy when this is in place.
  - F30 7y ago
    It is certainly doable, but I doubt that most people have IaC which is complete, reproducible and tested enough. And the data migration from the breached host still means some risk.
proboscis 7y ago
They closed the threads and deleted all the comments, but luckily the issues page was archived beforehand: https://web.archive.org/web/20190412090123/https://github.com/matrix-org/matrix.org/issues https://web.archive.org/web/20190412090123/https://github.co...
- gtirloni 7y ago
  Luckily yeah. So much wisdom in those comments. /s

by m_b 7y ago

m_b 7y ago

The hacker is now doing a post-mortem in the GitHub issues of the project: https://github.com/matrix-org/matrix.org/issues https://github.com/matrix-org/matrix.org/issues

nothrabannosir 7y ago
This is gold... > I noticed in your blog post that you were talking about doing a postmortem and steps you need to take. As someone who is intimately familiar with your entire infrastructure, I thought I could help you out. > There I was, just going about my business, looking for ways I could get higher levels of access and explore your network more, when I stumbled across GPG keys that were used for signing your debian packages. It gave me many nefarious ideas. I would recommend that you don't keep any signing keys on production hosts, and instead do all of your signing in a secure environment.
- paavoova 7y ago
  Why would they do this? It's pure negligence. I don't even sign anything important and still worry about my keys.
  - henvic 7y ago
    I have been asked twice or more why I insisted on not using a Continuous Integration environment for publishing some software releases that are installed by third-parties. My team was automating the infrastructure to build internal software and naturally they wanted to be able to simplify things. The idea that was proposed to me was the following: once I push a new version tag to GitHub, the deployment CI server is going to build and release it as an unstable version. Some important detail here: I use the same key to sign packages regardless if they are released as unstable or stable. That would mean that if someone, somehow, managed to push a tag that was pushed upstream to GitHub, hypothetically they would be able to eventually gain access to consumers machines (basically, developers) when the consumers update it after getting a notification telling them a new version is available. No way I'd allow this to happen, but I would not be surprised if most people just took this as an acceptable risk.
    - simias 7y ago
      Depending on your threat model I think that signing packages directly from your CI is acceptable, assuming that your CI runs is a reasonably isolated environment (e.g. on your company's LAN) and people who are able to trigger a release are correctly vetted. If I understand the parent comment correctly they were somehow shipping the release signing key on their production environment which is a whole other level of bad.
    - wcoenen 7y ago
      > That would mean that if someone, somehow, managed to push a tag that was pushed upstream to GitHub You have to define what the signature means. IMHO it is fine for it to mean "this software was built on our build server from a well-defined state of the source code, which is only changable by our employees and contractors, and for which we have the full change log". So I deploy the code signing key to build servers, which is the only place where it is used. I'm interested in what alternative meaning you would give to a signature. I have considered the possibility of tying it to the QA processes, but then a build can only be signed after checking it manually, which is problematic when many signatures are needed at multiple packaging layers (exe/dll, msi, setup.exe).
      - rakoo 7y ago
        The problem is when a malicious package is produced, either because a flaw was introduced in the code, or because a dev machine was compromised, or _when the CI machine sad compromised_; the malicious package will be signed as if it were legit. One middle point between automated and manual signing is, as usual, key rotation: have the signing keys expire in a short duration of time (say 2 weeks) and manually push them every week, so that the window of attack is as small as possible.
        yolo1 7y ago
        What does a key rotation solve? Either your build server is compromised or it's not.
      - notyourday 7y ago
- lelf 7y ago
  Another gem: RRREEEEEEEE> I noticed you missed a doctype in your html page. In order for web browsers to know what type of html to render you should include a doctype. Thanks! matrixnotorg> @RRREEEEEEEE Thank you, I will consider that for the next release Edit: it got deleted But see also: https://github.com/matrixnotorg/matrixnotorg.github.io/pull/2 https://github.com/matrixnotorg/matrixnotorg.github.io/pull/...
  - justaj 7y ago
    Wait, did Github delete matrixnotorg's profile or did matrixnotorg? If Github deleted that profile, I don't really see that as being very hacker-friendly.
    - aeternus 7y ago
      Although 'hacker' is often used as a positive term on HN, breaking into a company's production server is clearly illegal activity and should not be condoned. If Github deleted the account, they are simply acting in accordance to published TOS & policy.
      - justaj 7y ago
        If the attacker placed sensitive information on Github, that would indeed warrant a deletion of the account. However from what I saw from the archives, the attacker merely published details about Matrix.org's infrastructure and its vulnerabilities. Is that something that's against Github's ToS?
m00dy 7y ago
hilarious...
- meruru 7y ago
  I can only think of the despair I would be in if I was in Aragorn's shoes
  - rabidrat 7y ago
    Arathorn (Aragorn is son of Arathorn).
    - meruru 7y ago
      Does being a parent prepare you to handle emergency situations like this better?
deleted 7y ago
[deleted]
atsjie 7y ago
> Escalation could have been avoided if developers only had the access they absolutely required and did not have root access to all of the servers. I would like to take a moment to thank whichever developer forwarded their agent to Flywheel. I'd feel so small if I were this developer right now :-| A couple of his issues appear to have to do with the use of SSH. An Ops-guy whom I worked with had setup a bastion host with ip whitelisting that automatically shut down after 1 hour. He didn't like it as his credo was "if you're using SSH when using a cloud provider you're probably doing something wrong"; meaning to say you should automate and be able to recreate any infra at all times with logs accessible without the need for SSH. I never forgot that.
- robrtsql 7y ago
  Did you work with Rich Adams? https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/ https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-...
  - atsjie 7y ago
    no, but I wouldn't be surprised if my colleague got the idea from him. It was around the same time (end of 2014 I think).
BartBoch 7y ago
I have just checked quickly the comments and post-mortem, and I start wondering - it seems that the attack itself would be not really possible if Matrix would not be open source (as this would restrict access to the sensitive data)? Is that right?
- t3chguy 7y ago
  This is not right at all, the hack was due to an outdated Jenkins instance and could have happened regardless of what other software was running on the infrastructure.
  - notyourday 7y ago
    Absolutely not. There's a zero reason for development infrastructure ( which includes Jenkins ) to have any connection to production outside a well defined "transfer this tarball" to a deploy staging server path during a known deploy window which cannot be controlled via development credentials.

F30 7y ago

I am highly skeptical when people taking about "rebuilding [the whole] infrastructure" in a few hours. Even more so when restoring all data from breached systems and before a thorough incident analysis. Show me the org which can just pull that off.

pferde 7y ago
It's quite common in enterprise to have a RTO of a few hours, and RPO of a few minutes, even for infrastructure with terrabytes of data. Of course, many moneys are paid for being able to do that.
erikbye 7y ago
This is doable with proper IaC implementation, and if your org does not have RPO/RTO on lock they're doing it wrong. Events like Matrix experienced now do not lead to panicked frenzy when this is in place.
- F30 7y ago
  It is certainly doable, but I doubt that most people have IaC which is complete, reproducible and tested enough. And the data migration from the breached host still means some risk.

proboscis 7y ago

They closed the threads and deleted all the comments, but luckily the issues page was archived beforehand: https://web.archive.org/web/20190412090123/https://github.com/matrix-org/matrix.org/issues https://web.archive.org/web/20190412090123/https://github.co...

gtirloni 7y ago
Luckily yeah. So much wisdom in those comments. /s