5 ms·

Snowflake is blaming its own customers for using single-factor auth and that access to the affected accounts was gained through infostealers or acquired credent

Snowflake is blaming its own customers for using single-factor auth and that access to the affected accounts was gained through infostealers or acquired credentials. With their recent joint statement[0] together with Mandiant and CrowdStrike, they've completely removed themselves from the discussion of having had their systems breached. I'm not a Snowflake user myself and don't really know much about their internals, but perhaps there is someone here who can clarify if this scenario can happen: - Credentials acquired for a Ticketmaster account at Snowflake. - Log-in effortlessly because of single-factor auth. - Gain immediate access to a production environment and dump the database. Because that is how it sounds for the time being. It is Ticketmaster themselves who have confirmed that the database was stolen through Snowflake. If the process outlined above is how the DB was dumped, then what the hell? That's the most ridiculous and crazy and easily the dumbest blunder of all time, is it not? It's either that or there are still missing details somewhere along the way. [0]: https://community.snowflake.com/s/question/0D5VI00000Emyl00AB/detecting-and-preventing-unauthorized-user-access https://community.snowflake.com/s/question/0D5VI00000Emyl00A... [0]: https://archive.is/acJ6p https://archive.is/acJ6p (Archive link because the original URL can show an old cached version sometimes)

hipadev23 2y ago
Didn’t we already read this was entirely on Snowflake? They had employee credentials leaked — Solutions Engineers specifically. These were support credentials hanging around from when large clients integrated with Snowflakes So all it took was one master-key credential to be leaked to affect thousands of customer accounts.
- skilled 2y ago
  The post that alleged this[0][1] was removed/withdrawn. I do think that they spoke with the real perpetrator, because there is a lot that connects with that story. But there are a lot of smaller details that don't add up. And I don't believe Snowflake hired Mandiant/CrowdStrike just to come out with a lie that their systems weren't breached. Maybe someone else will speak to the guy who did all this to get a much clearer overview of the situation. [0]: https://news.ycombinator.com/item?id=40534868 https://news.ycombinator.com/item?id=40534868 [1]: https://web.archive.org/web/20240531140540/https://www.hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection https://web.archive.org/web/20240531140540/https://www.hudso...
  - hipadev23 2y ago
    So we’re at the “two is coincidence” stage and the next leak will change the narrative back to Snowflake.
  - xyst 2y ago
    The last exchange with the alleged hacker is a bit sus: researcher: > [snowflake] should have bought protection from Hudson Rock > could have saved them from this one hacker: > yes I agree > it would have helped Smells like a guerilla advertisement; or a hit piece to disparage competitors. Or maybe it was because some in-house legal team pulled it back. For now, I am getting bearish signals on SNOW. The initial reports of them blaming it on customers not enabling multi factor authentication is a way to deflect blame until they can figure out a way to deal with pending dump on their stock. Honestly, if this exchange with the hacker is legit. The $20M the hacker extorted the company for is going to be peanuts compared to the reputation loss and stock dump. Yet another L for SaaS based products. A centralized source of failure.
- aaronharnly 2y ago
  We read that[0], and then the company that published it un-published it without any accompanying notice[1]. [0] https://web.archive.org/web/20240531140540/https://www.hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection https://web.archive.org/web/20240531140540/https://www.hudso... [1] https://news.ycombinator.com/item?id=40547921 https://news.ycombinator.com/item?id=40547921
mahogany 2y ago
> Gain immediate access to a production environment and dump the database I don’t know how easy it is to dump an entire environment, but the “gain access” part depends on the user and what roles they have. If these were highly privileged users, then yes, they could get a lot. It could be mixed responsibility here. Possibly Snowflake should require MFA (do other cloud providers do this?) but their posture seems to be “we make it possible for anyone to sign up easily”. On the other hand if Ticketmaster had an admin account with access to entire production database which was not configured for MFA, that’s definitely on them too. On the third hand, if these were regular users that Ticketmaster did not restrict access for, that is also on them.
itscrush 2y ago
Good questions and I've been talking / wondering similar. I generally dislike Snowflake in this case and think they're attempting to deflect the blame off themselves for the architecture you get forced into as a tenant of theirs. I have some experience going through their docs and asking some exactly-how questions for implementations on the tenant end. I think Snowflake can both deflect blame correctly but inherently be the root cause making it hard to get perfect and right in a security context as a tenant like Ticket Master (unaffiliated to me). If they had real dedicated tenant services available, I'd probably feel they could defend themselves here. But that'll start an interpretation of meaning argument of dedicated vs. private with folks at SF due again to their architecture and thus the only context they speak. Private or dedicated to org only is neither easy to achieve via a lengthy list of security controls nor available to buy outright ... and their ask of SF tenant's to do better about the variety of hard-to-get-layered-right controls is too much sand paper for my liking. Another starter point on interpretation and SF's context, I've had at length discussions with them on; IP filtering such as actual packet drops / time outs at network firewalls vs. HTTP 403 errors responses at ip filtering layer, which occur at SF's shared layer7 ALB's services, actually matters. Tenant's should not have externally resolvable endpoints to worry about IP filtering or their interpretation of default deny (like their IP filtering choice of interpretation). [0] > - Credentials acquired for a Ticketmaster account at Snowflake. Tenant organizations like Ticketmaster here must enforce 2FA on accounts themselves proactively. Most times, regardless of 100% of interactive user accounts with 2FA, you'll still need system to system 1FA API type credentials for automation or large batch processing. These 1FA Service type accounts are usually static API key type credentials or a role where the credentials rotate but the role becomes static / assumable by another federation layer before that like in your IDP, LDAP, etc. These types of accounts still need to be usable and stored somewhere, it's a bad assumption that everyone does SOPS, key store, hashi vault, credentials manager, secrets store, and the like perfectly at all times. I'll hold off on the variety of ways you can choose to expose and share your data to 3rd parties. I don't think it's entirely a factor but folks should worry about this in other contexts too. > - Log-in effortlessly because of single-factor auth. Perhaps not effortlessly find, but it's not exactly easy to actually only expose your Snowflake instance to only your organization. Variety of reasons like shared authz/n happening in Layer7 at Snowflake ensure you can't go private exposure via private-link only, without also having externally resolvable addresses existing for your instance. Thus forcing you to layer IP whitelists, again poorly chosen by Snowflake at Layer7, wherever you can find how to apply it to all your instances (prod, dev, other subdomain tenant named instances). Thus creating an open loop in Ops for maintaining these SAAS software firewall lists. And eventually always having some ALB level HTTP web servers exposed to internet doing the IP filtering HTTP:40X error somewhere at Snowflake ALB's ensuring externally resolvable at least. Snowflake's fault here for never building true dedicated org capabilities with private hosting so far as I can tell. > - Gain immediate access to a production environment and dump the database. Not everyone will get a 403 when finding these Snowflake doors... and when you get a 200 with an SSO prompt you can usually start with username enumeration as a feature for getting true/false exists. These tenant's are almost always subdomain resolvable, picking one at random https://epa06486.snowflakecomputing.com/ https://epa06486.snowflakecomputing.com/ exists and happily receiving the IP filter at HTTP 403. You can find more using tools like https://dnsdumpster.com/ https://dnsdumpster.com/ (unaffiliated) or other relevant DNS toolsets you can walk subdomains for a lot of this information. As mentioned, these always end up with externally resolvable points of Snowflake tenant's interest in my experience because of Snowflake architecture choices not the tenant's mistakes like how to get IP filtering 100% right and have confidence in what is/not exposed to login attempts. [0]https://docs.snowflake.com/en/user-guide/network-policies https://docs.snowflake.com/en/user-guide/network-policies [0]https://docs.snowflake.com/en/user-guide/security-encryption-manage https://docs.snowflake.com/en/user-guide/security-encryption... [0]https://docs.snowflake.com/en/sql-reference/functions/system_allowlist https://docs.snowflake.com/en/sql-reference/functions/system... [0]https://docs.snowflake.com/en/developer-guide/sql-api/authenticating https://docs.snowflake.com/en/developer-guide/sql-api/authen...

hipadev23 2y ago

Didn’t we already read this was entirely on Snowflake? They had employee credentials leaked — Solutions Engineers specifically. These were support credentials hanging around from when large clients integrated with Snowflakes So all it took was one master-key credential to be leaked to affect thousands of customer accounts.

skilled 2y ago
The post that alleged this[0][1] was removed/withdrawn. I do think that they spoke with the real perpetrator, because there is a lot that connects with that story. But there are a lot of smaller details that don't add up. And I don't believe Snowflake hired Mandiant/CrowdStrike just to come out with a lie that their systems weren't breached. Maybe someone else will speak to the guy who did all this to get a much clearer overview of the situation. [0]: https://news.ycombinator.com/item?id=40534868 https://news.ycombinator.com/item?id=40534868 [1]: https://web.archive.org/web/20240531140540/https://www.hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection https://web.archive.org/web/20240531140540/https://www.hudso...
- hipadev23 2y ago
  So we’re at the “two is coincidence” stage and the next leak will change the narrative back to Snowflake.
- xyst 2y ago
  The last exchange with the alleged hacker is a bit sus: researcher: > [snowflake] should have bought protection from Hudson Rock > could have saved them from this one hacker: > yes I agree > it would have helped Smells like a guerilla advertisement; or a hit piece to disparage competitors. Or maybe it was because some in-house legal team pulled it back. For now, I am getting bearish signals on SNOW. The initial reports of them blaming it on customers not enabling multi factor authentication is a way to deflect blame until they can figure out a way to deal with pending dump on their stock. Honestly, if this exchange with the hacker is legit. The $20M the hacker extorted the company for is going to be peanuts compared to the reputation loss and stock dump. Yet another L for SaaS based products. A centralized source of failure.
aaronharnly 2y ago
We read that[0], and then the company that published it un-published it without any accompanying notice[1]. [0] https://web.archive.org/web/20240531140540/https://www.hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection https://web.archive.org/web/20240531140540/https://www.hudso... [1] https://news.ycombinator.com/item?id=40547921 https://news.ycombinator.com/item?id=40547921

mahogany 2y ago

> Gain immediate access to a production environment and dump the database I don’t know how easy it is to dump an entire environment, but the “gain access” part depends on the user and what roles they have. If these were highly privileged users, then yes, they could get a lot. It could be mixed responsibility here. Possibly Snowflake should require MFA (do other cloud providers do this?) but their posture seems to be “we make it possible for anyone to sign up easily”. On the other hand if Ticketmaster had an admin account with access to entire production database which was not configured for MFA, that’s definitely on them too. On the third hand, if these were regular users that Ticketmaster did not restrict access for, that is also on them.

itscrush 2y ago

Good questions and I've been talking / wondering similar. I generally dislike Snowflake in this case and think they're attempting to deflect the blame off themselves for the architecture you get forced into as a tenant of theirs. I have some experience going through their docs and asking some exactly-how questions for implementations on the tenant end. I think Snowflake can both deflect blame correctly but inherently be the root cause making it hard to get perfect and right in a security context as a tenant like Ticket Master (unaffiliated to me). If they had real dedicated tenant services available, I'd probably feel they could defend themselves here. But that'll start an interpretation of meaning argument of dedicated vs. private with folks at SF due again to their architecture and thus the only context they speak. Private or dedicated to org only is neither easy to achieve via a lengthy list of security controls nor available to buy outright ... and their ask of SF tenant's to do better about the variety of hard-to-get-layered-right controls is too much sand paper for my liking. Another starter point on interpretation and SF's context, I've had at length discussions with them on; IP filtering such as actual packet drops / time outs at network firewalls vs. HTTP 403 errors responses at ip filtering layer, which occur at SF's shared layer7 ALB's services, actually matters. Tenant's should not have externally resolvable endpoints to worry about IP filtering or their interpretation of default deny (like their IP filtering choice of interpretation). [0] > - Credentials acquired for a Ticketmaster account at Snowflake. Tenant organizations like Ticketmaster here must enforce 2FA on accounts themselves proactively. Most times, regardless of 100% of interactive user accounts with 2FA, you'll still need system to system 1FA API type credentials for automation or large batch processing. These 1FA Service type accounts are usually static API key type credentials or a role where the credentials rotate but the role becomes static / assumable by another federation layer before that like in your IDP, LDAP, etc. These types of accounts still need to be usable and stored somewhere, it's a bad assumption that everyone does SOPS, key store, hashi vault, credentials manager, secrets store, and the like perfectly at all times. I'll hold off on the variety of ways you can choose to expose and share your data to 3rd parties. I don't think it's entirely a factor but folks should worry about this in other contexts too. > - Log-in effortlessly because of single-factor auth. Perhaps not effortlessly find, but it's not exactly easy to actually only expose your Snowflake instance to only your organization. Variety of reasons like shared authz/n happening in Layer7 at Snowflake ensure you can't go private exposure via private-link only, without also having externally resolvable addresses existing for your instance. Thus forcing you to layer IP whitelists, again poorly chosen by Snowflake at Layer7, wherever you can find how to apply it to all your instances (prod, dev, other subdomain tenant named instances). Thus creating an open loop in Ops for maintaining these SAAS software firewall lists. And eventually always having some ALB level HTTP web servers exposed to internet doing the IP filtering HTTP:40X error somewhere at Snowflake ALB's ensuring externally resolvable at least. Snowflake's fault here for never building true dedicated org capabilities with private hosting so far as I can tell. > - Gain immediate access to a production environment and dump the database. Not everyone will get a 403 when finding these Snowflake doors... and when you get a 200 with an SSO prompt you can usually start with username enumeration as a feature for getting true/false exists. These tenant's are almost always subdomain resolvable, picking one at random https://epa06486.snowflakecomputing.com/ https://epa06486.snowflakecomputing.com/ exists and happily receiving the IP filter at HTTP 403. You can find more using tools like https://dnsdumpster.com/ https://dnsdumpster.com/ (unaffiliated) or other relevant DNS toolsets you can walk subdomains for a lot of this information. As mentioned, these always end up with externally resolvable points of Snowflake tenant's interest in my experience because of Snowflake architecture choices not the tenant's mistakes like how to get IP filtering 100% right and have confidence in what is/not exposed to login attempts. [0]https://docs.snowflake.com/en/user-guide/network-policies https://docs.snowflake.com/en/user-guide/network-policies [0]https://docs.snowflake.com/en/user-guide/security-encryption-manage https://docs.snowflake.com/en/user-guide/security-encryption... [0]https://docs.snowflake.com/en/sql-reference/functions/system_allowlist https://docs.snowflake.com/en/sql-reference/functions/system... [0]https://docs.snowflake.com/en/developer-guide/sql-api/authenticating https://docs.snowflake.com/en/developer-guide/sql-api/authen...