4 ms·

> Those sources said the breach appears to have started when the attackers somehow gained access to the company’s Gitlab code repository, and in that repository

by koolba 3y ago

> Those sources said the breach appears to have started when the attackers somehow gained access to the company’s Gitlab code repository, and in that repository was a token or credential that gave the bad guys access to Sisense’s Amazon S3 buckets in the cloud. So plaintext AWS credentials checked into source control. > Both sources said the attackers used the S3 access to copy and exfiltrate several terabytes worth of Sisense customer data, which apparently included millions of access tokens, email account passwords, and even SSL certificates. And those credentials can access tons of other credentials, also stored in plaintext. Even if the security policies didn’t pick this up, didn’t they at least having billing alerts for the terabytes of AWS bandwidth?

EdwardDiego 3y ago
This feels... ...ridiculous in this day and age.
- doctorpangloss 3y ago
  On the other hand, once they had source code access, does it matter? People play make believe about this stuff. “The practice I feel strongly about, it’s secure. And the other guy is a moron.” The XZ crisis shows how little credential isolation matters. Maybe even at Google, you could work there and feel very strongly about their security practices, and then governments have access to everything anyway, at the front end proxies or whatever. It’s important to pay attention to security, but I think programmer resources should focus on “cheating,” not “security.”
  - deleted 3y ago
    [deleted]
  - mdekkers 3y ago
    You must be trolling, right? Right?! Programmer resources should not focus on security? Credential isolation doesn’t matter?
  - EdwardDiego 3y ago
    Yeah it does, because you left the key for the place where you stored everyone else's keys out in the relative open. It reduces your defensive depth to 1, as in, 1 compromise was all it took. For a tortured analogy, let's say you keep your guns in a gun-safe in your garage, and then hang the key for the safe in the garage. With a sign saying "gun safe key". Not leaving the safe key out in the open doesn't guarantee your guns are impervious, of course, a sufficiently motivated and skilled burglar willing to put time and effort into breaking open the safe would be able to access your guns. But, that takes time. And it makes noise. Both of which increase the chance of being caught. But if the key is right there? Easy as. Yep, they had obtained access to the repo, like the burglar had obtained access to the garage. But if the AWS secrets weren't just hanging on the wall, then progressing from Gitlab compromise to S3 compromise would a) be harder and b) take longer, both of which increase the chance of discovery. Just saying, if you care about the security of your guns and/or customer data, don't leave the key hanging about in plain sight as a good first step. Make them compromise multiple things, not just one thing. I'm desperately trying to work encryption at rest into this analogy. Um... trigger locks?
    - doctorpangloss 3y ago
      > Yep, they had obtained access to the repo, like the burglar had obtained access to the garage. I'm not going to talk in analogies. There's no more depth at the stage that the source code is compromised, regardless of how many credentials are put where. It's sort of a matter of opinion. I mean this is what I am saying by make-believe: everything you say is, quite literally, a bunch of analogies, because there isn't any actual hard evidence to any of it, it is wood carving traditions. Of course I agree that they're good practices, but man, there are unlimited good security practices, and only limited money and time. Another point of view is, it's easy to provide generalized, vendor-colored advice about credential security. You know, just buy AWS KMS whatever the fuck, right? It is hard to write software that accommodates arbitrary changes to authorization requirements & stories. There are a lot of commenters here giving these Sisense people a hard time, and I doubt they're stupid, more so that they are really unlucky. I feel like the comments miss the forest for the trees. Most startups have admin roles, and if you're using something like Next.js, Elixir, Ruby on Rails or Go, your application's API methods have ad-hoc, in-source policy enforcement and definitions. This is what happened here: the developer's GitHub credentials are an Admin role. It could happen in any scheme. Can you develop an application without an admin role? Not a SaaS. Now you should see why it's okay for Postgres to have all its source code open, but not your SaaS company's, for the purposes of security.
mikeyouse 3y ago
Isn't S3 egress billed at something like $0.01/GB? Even if you assume they exfiltrated 100TB, you're only talking about maybe $1k. I can't imagine they'd have alerts fine-tuned enough to catch that without driving their on-call staff crazy.
leetrout 3y ago
My previous team argued with me when i said to avoid using terraform with vault with our state in S3. I eventually got my way after 18 months and found out it was really bad - hard coded s3 access keys were floating around that could read any bucket and they had used terraform with vault and that had pulled all sorts of root level creds into the plain text state files. Lots of other ways to control this risk but I removed vault from TF.
- captn3m0 3y ago
  opentofu is solving this with proper state encryption support: https://github.com/opentofu/opentofu/issues/874 https://github.com/opentofu/opentofu/issues/874
  - lmeyerov 3y ago
    Do you think a feature like this is tied to distancing OpenTofu from Hashicorp, e.g., AWS & friends have less incentive to do enterprise feature gating around security?
    - captn3m0 3y ago
      State Encryption was one of those long requested features[0] (I had it on my ideas list for years[1]) that Hashicorp didn't have much incentive to build. I don't think it has to with distancing opentofu as such, but the opentofu team prioritizing the right things that customers actually need. [0]: https://github.com/hashicorp/terraform/issues/9556 https://github.com/hashicorp/terraform/issues/9556 [1]: https://github.com/captn3m0/ideas#-mars-terraform-remote-http-backend-with-end-to-end-encryption https://github.com/captn3m0/ideas#-mars-terraform-remote-htt...
      - lmeyerov 3y ago
        Interesting! The 'much incentive' part is the interesting thing to me. Hashicorp did build encryption-at-rest for state into their paid remote cloud service. They explicitly recommend using the paid service instead when security is required, and instead of having both versions secure, the paid form simply disables the local one [1], drawing a strong line here between paid/secure and unpaid/insecure. So I'm wondering if that created a disincentive, from a perspective of enterprises upselling for security & compliance. OpenTofu community members aren't relying on selling a remote management service, and when a cloud provider, more usage in general is better, and that includes security for everyone. I can be off, so not quite clear to me the more I look. [1] https://developer.hashicorp.com/terraform/language/state/sensitive-data https://developer.hashicorp.com/terraform/language/state/sen...
bsder 3y ago
> didn’t they at least having billing alerts for the terabytes of AWS bandwidth? AWS is notoriously bad about this. You can't set hard limits. Even if you set a limit, your notification won't come in until long after that limit has been hit and blown through.
- mschuster91 3y ago
  At least according to general "HN hivemind experience", AWS is reasonable in handing out refunds if you got hacked.
  - TheNewsIsHere 3y ago
    My business has not (yet?) been compromised, but I did accidentally send my S3 bill into the stratosphere in the past 12-18 months by miscalculating API charges for some intra-AWS changes I was making with lifecycle management and bucket-to-bucket moves. I wrote to support asking for a refund and explained my calculations. I pointed out where I had made the mistake, and asked for some relief on the bill. Honesty went a long way in that case, and they gave me a credit on the bill equal to about 70% of the total S3 spend. So they ate 70% of my mistake. That was more than fair enough; it was my error. The amounts weren’t huge. They could have eaten the entire amount and it wouldn’t have even been a rounding error on their books, but I didn’t ask and I didn’t want to expect that. I can’t speak for others.
ali_piccioni 3y ago
Long lived credentials are a security red flag. We setup our AWS organization’s policies (SCPs) to prohibit long-lived tokens. Instead access goes through SSO or OIDC. It’s difficult to track usage behind access tokens, prevent leaks, and effectively revoke them.
- specialist 3y ago
  Yup. TTL (leases) must become the norm for All The Things.
about3fitty 3y ago
I know this says credentials were in a repo but I wonder if this has to do with their self-hosted (EE?) GitLab instance. I've found mine (CE) sort of difficult to upgrade sometimes. Not sure how they store keys/tokens but it is sort of easy to leak to logs, and perhaps there was some exploit. Should maybe be using cloud hosted GitLab at their level of scale, even though that option has had problems with uptime among other issues. A past company used Sisense because the new manager came in penny wise and pound foolish. However, the other ETL/visualization products (e.g. Tableau) truly were expensive. I found Sisense's ETL / viz product to be meh.
nucleardog 3y ago
> So plaintext AWS credentials checked into source control. Could also have been keys set as variables for their CI pipelines to support IaC/etc deployment.

by koolba 3y ago

EdwardDiego 3y ago

This feels... ...ridiculous in this day and age.

doctorpangloss 3y ago
On the other hand, once they had source code access, does it matter? People play make believe about this stuff. “The practice I feel strongly about, it’s secure. And the other guy is a moron.” The XZ crisis shows how little credential isolation matters. Maybe even at Google, you could work there and feel very strongly about their security practices, and then governments have access to everything anyway, at the front end proxies or whatever. It’s important to pay attention to security, but I think programmer resources should focus on “cheating,” not “security.”
- deleted 3y ago
  [deleted]
- mdekkers 3y ago
  You must be trolling, right? Right?! Programmer resources should not focus on security? Credential isolation doesn’t matter?
- EdwardDiego 3y ago
  Yeah it does, because you left the key for the place where you stored everyone else's keys out in the relative open. It reduces your defensive depth to 1, as in, 1 compromise was all it took. For a tortured analogy, let's say you keep your guns in a gun-safe in your garage, and then hang the key for the safe in the garage. With a sign saying "gun safe key". Not leaving the safe key out in the open doesn't guarantee your guns are impervious, of course, a sufficiently motivated and skilled burglar willing to put time and effort into breaking open the safe would be able to access your guns. But, that takes time. And it makes noise. Both of which increase the chance of being caught. But if the key is right there? Easy as. Yep, they had obtained access to the repo, like the burglar had obtained access to the garage. But if the AWS secrets weren't just hanging on the wall, then progressing from Gitlab compromise to S3 compromise would a) be harder and b) take longer, both of which increase the chance of discovery. Just saying, if you care about the security of your guns and/or customer data, don't leave the key hanging about in plain sight as a good first step. Make them compromise multiple things, not just one thing. I'm desperately trying to work encryption at rest into this analogy. Um... trigger locks?
  - doctorpangloss 3y ago
    > Yep, they had obtained access to the repo, like the burglar had obtained access to the garage. I'm not going to talk in analogies. There's no more depth at the stage that the source code is compromised, regardless of how many credentials are put where. It's sort of a matter of opinion. I mean this is what I am saying by make-believe: everything you say is, quite literally, a bunch of analogies, because there isn't any actual hard evidence to any of it, it is wood carving traditions. Of course I agree that they're good practices, but man, there are unlimited good security practices, and only limited money and time. Another point of view is, it's easy to provide generalized, vendor-colored advice about credential security. You know, just buy AWS KMS whatever the fuck, right? It is hard to write software that accommodates arbitrary changes to authorization requirements & stories. There are a lot of commenters here giving these Sisense people a hard time, and I doubt they're stupid, more so that they are really unlucky. I feel like the comments miss the forest for the trees. Most startups have admin roles, and if you're using something like Next.js, Elixir, Ruby on Rails or Go, your application's API methods have ad-hoc, in-source policy enforcement and definitions. This is what happened here: the developer's GitHub credentials are an Admin role. It could happen in any scheme. Can you develop an application without an admin role? Not a SaaS. Now you should see why it's okay for Postgres to have all its source code open, but not your SaaS company's, for the purposes of security.

mikeyouse 3y ago

Isn't S3 egress billed at something like $0.01/GB? Even if you assume they exfiltrated 100TB, you're only talking about maybe $1k. I can't imagine they'd have alerts fine-tuned enough to catch that without driving their on-call staff crazy.

leetrout 3y ago

My previous team argued with me when i said to avoid using terraform with vault with our state in S3. I eventually got my way after 18 months and found out it was really bad - hard coded s3 access keys were floating around that could read any bucket and they had used terraform with vault and that had pulled all sorts of root level creds into the plain text state files. Lots of other ways to control this risk but I removed vault from TF.

captn3m0 3y ago
opentofu is solving this with proper state encryption support: https://github.com/opentofu/opentofu/issues/874 https://github.com/opentofu/opentofu/issues/874
- lmeyerov 3y ago
  Do you think a feature like this is tied to distancing OpenTofu from Hashicorp, e.g., AWS & friends have less incentive to do enterprise feature gating around security?
  - captn3m0 3y ago
    State Encryption was one of those long requested features[0] (I had it on my ideas list for years[1]) that Hashicorp didn't have much incentive to build. I don't think it has to with distancing opentofu as such, but the opentofu team prioritizing the right things that customers actually need. [0]: https://github.com/hashicorp/terraform/issues/9556 https://github.com/hashicorp/terraform/issues/9556 [1]: https://github.com/captn3m0/ideas#-mars-terraform-remote-http-backend-with-end-to-end-encryption https://github.com/captn3m0/ideas#-mars-terraform-remote-htt...
    - lmeyerov 3y ago
      Interesting! The 'much incentive' part is the interesting thing to me. Hashicorp did build encryption-at-rest for state into their paid remote cloud service. They explicitly recommend using the paid service instead when security is required, and instead of having both versions secure, the paid form simply disables the local one [1], drawing a strong line here between paid/secure and unpaid/insecure. So I'm wondering if that created a disincentive, from a perspective of enterprises upselling for security & compliance. OpenTofu community members aren't relying on selling a remote management service, and when a cloud provider, more usage in general is better, and that includes security for everyone. I can be off, so not quite clear to me the more I look. [1] https://developer.hashicorp.com/terraform/language/state/sensitive-data https://developer.hashicorp.com/terraform/language/state/sen...

bsder 3y ago

> didn’t they at least having billing alerts for the terabytes of AWS bandwidth? AWS is notoriously bad about this. You can't set hard limits. Even if you set a limit, your notification won't come in until long after that limit has been hit and blown through.

mschuster91 3y ago
At least according to general "HN hivemind experience", AWS is reasonable in handing out refunds if you got hacked.
- TheNewsIsHere 3y ago
  My business has not (yet?) been compromised, but I did accidentally send my S3 bill into the stratosphere in the past 12-18 months by miscalculating API charges for some intra-AWS changes I was making with lifecycle management and bucket-to-bucket moves. I wrote to support asking for a refund and explained my calculations. I pointed out where I had made the mistake, and asked for some relief on the bill. Honesty went a long way in that case, and they gave me a credit on the bill equal to about 70% of the total S3 spend. So they ate 70% of my mistake. That was more than fair enough; it was my error. The amounts weren’t huge. They could have eaten the entire amount and it wouldn’t have even been a rounding error on their books, but I didn’t ask and I didn’t want to expect that. I can’t speak for others.

ali_piccioni 3y ago

Long lived credentials are a security red flag. We setup our AWS organization’s policies (SCPs) to prohibit long-lived tokens. Instead access goes through SSO or OIDC. It’s difficult to track usage behind access tokens, prevent leaks, and effectively revoke them.

specialist 3y ago
Yup. TTL (leases) must become the norm for All The Things.

about3fitty 3y ago

I know this says credentials were in a repo but I wonder if this has to do with their self-hosted (EE?) GitLab instance. I've found mine (CE) sort of difficult to upgrade sometimes. Not sure how they store keys/tokens but it is sort of easy to leak to logs, and perhaps there was some exploit. Should maybe be using cloud hosted GitLab at their level of scale, even though that option has had problems with uptime among other issues. A past company used Sisense because the new manager came in penny wise and pound foolish. However, the other ETL/visualization products (e.g. Tableau) truly were expensive. I found Sisense's ETL / viz product to be meh.

nucleardog 3y ago

> So plaintext AWS credentials checked into source control. Could also have been keys set as variables for their CI pipelines to support IaC/etc deployment.