32 ms·

There shouldn't be any coming back from this. There are failures on multiple levels here & CircleCI demonstrated no one should keep any sensitive data with them

• bdcravens 4y ago
  From the outside, it looks like many key infrastructure operations are running things like they are still startups, moving fast and breaking things. No access controls, engineers with machines they can take anywhere and install anything on, with full access to everything. Can't get in the way of moving fast, after all.
  • kodah 4y ago
    Developer laptops are the next logical vector from zero trust networks backed by SSO. Reading between the lines it seems like the session was an SSO token. I think you're right about giving too broad of access. A lot of companies I've been at assign you to LDAP groups over time and if you've been there a while you'll belong to a lot of LDAP groups. There is no ritual to minifying access that I've ever been made aware of. We definitely need some developer oriented antivirus software, because moving to a life where every bit of software needs to be approved would be a literal nightmare.
• lemon_bottle 4y ago
  Unpopular opinion but I think whenever these 'hack' incidents happen, there should be a full disclosure and the company should be made to tell us what exactly was hacked right down to the nuts and blots, sql queries and server processes level. There should be full transparency on this and all the open source eyeballs should be able to study and scrutinize this. If not, anyone will be able to get away with data theft tomorrow saying, "my server got hacked". What will cause them to not do it except some sense of personal ethics which is rapidly degrading these days?
  • stingraycharles 4y ago
    That is not an unpopular opinion, people have been arguing on this forum for proper legislation around these types of security incidents and breaches for years.
  • foepys 4y ago
    GDPR requires a full disclosure of which data and users are affected and when it happened, which I think is fair for the average user.
    • AYBABTME 4y ago
      But that's like a bridge collapsing and only having to produce a list of who got injured and what their injuries were.
      • true_religion 4y ago
        Bridges tend to be owned by the government which in democratic counties gives an accounting to the public. Private bridges, have owners who give an accounting to no one when they collapse. At best, if you have standing you can sue them and they will defend themselves by giving an accounting for why it’s not their fault or they did their best.
    • pedro2 4y ago
      Those things need to be supervisioned. FlyTAP has gotten their entire rewards DB (if not all their DB) hacked and if it were not for IHBP I would never know because the company never told me. As far as I know FlyTAP never got a fine for ignoring its obligations. GDPR is a good idea but there must be supervision for it being applied as defined.
  • deleted 4y ago
    [deleted]
  • BaculumMeumEst 4y ago
    That opinion seems like common sense. Why do you think it would be unpopular?
• bryan0 4y ago
  > There shouldn't be any coming back from this. There are failures on multiple levels here & CircleCI demonstrated no one should keep any sensitive data with them. Are other companies any better? It seems like now it’s assumed that all of these companies will eventually be hacked, so if you use them you need to have systems in place to mitigate damage. And if you don’t use them them then you need to have mitigation strats anyway. Basically either way you’re screwed.
  • jay-barronville 4y ago
    Not necessarily. Yes, it can be hard to fight off dedicated hackers, but it shouldn’t be that easy. I know it’s easy to judge them when we’re not in their shoes, but one compromised employee shouldn’t be able to produce these results. There wasn’t even proper key management protocols in place here. I can set up a production-ready Vault (HashiCorp) cluster (with decentralized key shares) in a weekend and exercise better OPSEC than them. That’s actually disappointing.
    • AYBABTME 4y ago
      Using Vault isn't going to make this impossible. The secrets still end up somewhere on a host in an env var. People still get permission at some point to read from Vault. Combine enough parts of the puzzle and you can find cracks in the system.
    • komuW 4y ago
      Note that dumping the Vault's process memory is beyond hashicorp/Vault's threat model. See: https://github.com/hashicorp/vault/issues/1446#issuecomment-221625921 https://github.com/hashicorp/vault/issues/1446#issuecomment-... I'm bringing this up because the circleCI blogpost says that the attacker did memory-dump encryption keys from a running process. See https://circleci.com/blog/jan-4-2023-incident-report/ https://circleci.com/blog/jan-4-2023-incident-report/ So even if they were using hashicorp/vault, the attacker could probably still have been able to mem-dump vault's process.
      • robszumski 4y ago
• dehrmann 4y ago
  Who has a better CI offering these days? I moved to CircleCI after Travis CI abandoned its open-source support.
  • m00x 4y ago
    Github actions are pretty amazing. It can run most CI workflows.
    • riddlemethat 4y ago
      Azure Pipelines + GitHub Actions
  • alex_suzuki 4y ago
    Gitlab pipelines work well for me. I’m sure the Github and Azure offerings are on-par.
    • jeltz 4y ago
      From my experience: no, Gitlab has a much better offering than GitHub.
  • lclc 4y ago
    It's very easy to run your own GitLab-Runner (their open-source CI: https://docs.gitlab.com/runner/ https://docs.gitlab.com/runner/) on your server (or any cloud). Set up within seconds using a few lines of cloud-init: https://gitlab.com/21analytics/gitlab-runner-cloud-init https://gitlab.com/21analytics/gitlab-runner-cloud-init Most of the time, it's also cheaper and maintenance is close to zero.
    • stingraycharles 4y ago
      I second GitLab runner, used it a lot on bare metal CI servers, because we have some really exotic hardware requirements. Happy customer for 3 years now, and the CI part costs us nothing.
    • 4y ago
• AYBABTME 4y ago
  Stuff is so complex nowadays that there's a ton of vectors an employee can use to eventually access almost anything on production systems. Unless you kneecap their ability to reach anything at all. Even if say you use Vault for secrets, extensive ACLs, on-demand access, there's an almost infinite amount of ways to squeeze out data in ways it hasn't been anticipated. Kuberneters, containers, etc doesn't always help, often sometimes it makes things worse. Defending against internal threat seems like a losing battle that can only be mitigated, slowing down more than preventing an attack.
  • debarshri 4y ago
    Stopping employee from accessing anything and everything in production impacts productivity and engineer often do not want to work in that kind of environment.
    • AYBABTME 4y ago
      It's also not possible, by definition someone needs some access. Compromise enough of those people and there's no way around it.
      • debarshri 4y ago
        Exactly. Only thing you can do is make a drill where you assume everything is hacked and audit access daily, weekly and monthly.
        • wepzen 4y ago
          Implementing a zero-trust architecture with a trust-score system for users and a dynamic policy for accessing resources can help to limit potential damage in the event of a security incident. But I agree that the balance between protecting against attacks and maintaining productivity can be delicate.
• maccard 4y ago
  At some point, someone somewhere has to have access to infrastructure, and be able to deploy it (if not, then generate a token of set of credentials that _can_). Circleci's business model is hosting CI runners for you, so of course they need to be able to decrypt the data, and if you have access to dept new infrastructure, you likely have permission to read encryption keys (or deploy new infrastructure that can read said keys and then use those machines to get the keys). What CircleCI _have_ done is set up enough logging and auditing that they were able to figure out who was compromised, how they were compromised, the time frame and the resources they accessed, which IMO is about as much as you can ask for.
  • oxfordmale 4y ago
    CircleCI have sufficient logging, however, failed to do fraud analysis to detect irregular access. Fraud analysis can be basic and just have a threshold on the number of encryption keys accessed each day. For this particular employee, there will have been a spike. They also point their finger at anti-virus for not detecting the malware. That is a lame excuses. Professional malware developers will check if their product goes undetected by the major anti-virus providers. Anti-virus should not be relied upon to protect against sophisticated attacks
    • maccard 4y ago
      They didn't blame the anti malware. In the blog post[0] all they say is > This machine was compromised on December 16, 2022. The malware was not detected by our antivirus software That's not blaming that's telling people that their antivirus didn't detect it. If that wasn't there, people would be talking about why they didn't use X antivirus which would probably detect it. > CircleCI have sufficient logging, however, failed to do fraud analysis to detect irregular access. This is a significant move of the goalposts. Of course CircleCI messed up, but to go back to the OP's point of "nobody should have that much access to production", well that's just not true. [0] https://circleci.com/blog/jan-4-2023-incident-report/ https://circleci.com/blog/jan-4-2023-incident-report/
      • oxfordmale 4y ago