5 ms·
The Ticketmaster Data Breach May Be Just the Beginning
- skilled 2y agohttps://archive.is/gOdBL https://archive.is/gOdBL
- deleted 2y ago[deleted]
- skilled 2y agoSnowflake is blaming its own customers for using single-factor auth and that access to the affected accounts was gained through infostealers or acquired credentials. With their recent joint statement[0] together with Mandiant and CrowdStrike, they've completely removed themselves from the discussion of having had their systems breached. I'm not a Snowflake user myself and don't really know much about their internals, but perhaps there is someone here who can clarify if this scenario can happen: - Credentials acquired for a Ticketmaster account at Snowflake. - Log-in effortlessly because of single-factor auth. - Gain immediate access to a production environment and dump the database. Because that is how it sounds for the time being. It is Ticketmaster themselves who have confirmed that the database was stolen through Snowflake. If the process outlined above is how the DB was dumped, then what the hell? That's the most ridiculous and crazy and easily the dumbest blunder of all time, is it not? It's either that or there are still missing details somewhere along the way. [0]: https://community.snowflake.com/s/question/0D5VI00000Emyl00AB/detecting-and-preventing-unauthorized-user-access https://community.snowflake.com/s/question/0D5VI00000Emyl00A... [0]: https://archive.is/acJ6p https://archive.is/acJ6p (Archive link because the original URL can show an old cached version sometimes)
- hipadev23 2y agoDidn’t we already read this was entirely on Snowflake? They had employee credentials leaked — Solutions Engineers specifically. These were support credentials hanging around from when large clients integrated with Snowflakes So all it took was one master-key credential to be leaked to affect thousands of customer accounts.
- skilled 2y agoThe post that alleged this[0][1] was removed/withdrawn. I do think that they spoke with the real perpetrator, because there is a lot that connects with that story. But there are a lot of smaller details that don't add up. And I don't believe Snowflake hired Mandiant/CrowdStrike just to come out with a lie that their systems weren't breached. Maybe someone else will speak to the guy who did all this to get a much clearer overview of the situation. [0]: https://news.ycombinator.com/item?id=40534868 https://news.ycombinator.com/item?id=40534868 [1]: https://web.archive.org/web/20240531140540/https://www.hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection https://web.archive.org/web/20240531140540/https://www.hudso...
- hipadev23 2y agoSo we’re at the “two is coincidence” stage and the next leak will change the narrative back to Snowflake.
- xyst 2y agoThe last exchange with the alleged hacker is a bit sus: researcher: > [snowflake] should have bought protection from Hudson Rock > could have saved them from this one hacker: > yes I agree > it would have helped Smells like a guerilla advertisement; or a hit piece to disparage competitors. Or maybe it was because some in-house legal team pulled it back. For now, I am getting bearish signals on SNOW. The initial reports of them blaming it on customers not enabling multi factor authentication is a way to deflect blame until they can figure out a way to deal with pending dump on their stock. Honestly, if this exchange with the hacker is legit. The $20M the hacker extorted the company for is going to be peanuts compared to the reputation loss and stock dump. Yet another L for SaaS based products. A centralized source of failure.
- aaronharnly 2y agoWe read that[0], and then the company that published it un-published it without any accompanying notice[1]. [0] https://web.archive.org/web/20240531140540/https://www.hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection https://web.archive.org/web/20240531140540/https://www.hudso... [1] https://news.ycombinator.com/item?id=40547921 https://news.ycombinator.com/item?id=40547921
- mahogany 2y ago> Gain immediate access to a production environment and dump the database I don’t know how easy it is to dump an entire environment, but the “gain access” part depends on the user and what roles they have. If these were highly privileged users, then yes, they could get a lot. It could be mixed responsibility here. Possibly Snowflake should require MFA (do other cloud providers do this?) but their posture seems to be “we make it possible for anyone to sign up easily”. On the other hand if Ticketmaster had an admin account with access to entire production database which was not configured for MFA, that’s definitely on them too. On the third hand, if these were regular users that Ticketmaster did not restrict access for, that is also on them.
- itscrush 2y agoGood questions and I've been talking / wondering similar. I generally dislike Snowflake in this case and think they're attempting to deflect the blame off themselves for the architecture you get forced into as a tenant of theirs. I have some experience going through their docs and asking some exactly-how questions for implementations on the tenant end. I think Snowflake can both deflect blame correctly but inherently be the root cause making it hard to get perfect and right in a security context as a tenant like Ticket Master (unaffiliated to me). If they had real dedicated tenant services available, I'd probably feel they could defend themselves here. But that'll start an interpretation of meaning argument of dedicated vs. private with folks at SF due again to their architecture and thus the only context they speak. Private or dedicated to org only is neither easy to achieve via a lengthy list of security controls nor available to buy outright ... and their ask of SF tenant's to do better about the variety of hard-to-get-layered-right controls is too much sand paper for my liking. Another starter point on interpretation and SF's context, I've had at length discussions with them on; IP filtering such as actual packet drops / time outs at network firewalls vs. HTTP 403 errors responses at ip filtering layer, which occur at SF's shared layer7 ALB's services, actually matters. Tenant's should not have externally resolvable endpoints to worry about IP filtering or their interpretation of default deny (like their IP filtering choice of interpretation). [0] > - Credentials acquired for a Ticketmaster account at Snowflake. Tenant organizations like Ticketmaster here must enforce 2FA on accounts themselves proactively. Most times, regardless of 100% of interactive user accounts with 2FA, you'll still need system to system 1FA API type credentials for automation or large batch processing. These 1FA Service type accounts are usually static API key type credentials or a role where the credentials rotate but the role becomes static / assumable by another federation layer before that like in your IDP, LDAP, etc. These types of accounts still need to be usable and stored somewhere, it's a bad assumption that everyone does SOPS, key store, hashi vault, credentials manager, secrets store, and the like perfectly at all times. I'll hold off on the variety of ways you can choose to expose and share your data to 3rd parties. I don't think it's entirely a factor but folks should worry about this in other contexts too. > - Log-in effortlessly because of single-factor auth. Perhaps not effortlessly find, but it's not exactly easy to actually only expose your Snowflake instance to only your organization. Variety of reasons like shared authz/n happening in Layer7 at Snowflake ensure you can't go private exposure via private-link only, without also having externally resolvable addresses existing for your instance. Thus forcing you to layer IP whitelists, again poorly chosen by Snowflake at Layer7, wherever you can find how to apply it to all your instances (prod, dev, other subdomain tenant named instances). Thus creating an open loop in Ops for maintaining these SAAS software firewall lists. And eventually always having some ALB level HTTP web servers exposed to internet doing the IP filtering HTTP:40X error somewhere at Snowflake ALB's ensuring externally resolvable at least. Snowflake's fault here for never building true dedicated org capabilities with private hosting so far as I can tell. > - Gain immediate access to a production environment and dump the database. Not everyone will get a 403 when finding these Snowflake doors... and when you get a 200 with an SSO prompt you can usually start with username enumeration as a feature for getting true/false exists. These tenant's are almost always subdomain resolvable, picking one at random https://epa06486.snowflakecomputing.com/ https://epa06486.snowflakecomputing.com/ exists and happily receiving the IP filter at HTTP 403. You can find more using tools like https://dnsdumpster.com/ https://dnsdumpster.com/ (unaffiliated) or other relevant DNS toolsets you can walk subdomains for a lot of this information. As mentioned, these always end up with externally resolvable points of Snowflake tenant's interest in my experience because of Snowflake architecture choices not the tenant's mistakes like how to get IP filtering 100% right and have confidence in what is/not exposed to login attempts. [0]https://docs.snowflake.com/en/user-guide/network-policies https://docs.snowflake.com/en/user-guide/network-policies [0]https://docs.snowflake.com/en/user-guide/security-encryption-manage https://docs.snowflake.com/en/user-guide/security-encryption... [0]https://docs.snowflake.com/en/sql-reference/functions/system_allowlist https://docs.snowflake.com/en/sql-reference/functions/system... [0]https://docs.snowflake.com/en/developer-guide/sql-api/authenticating https://docs.snowflake.com/en/developer-guide/sql-api/authen...
- sys_64738 2y agoThese companies are allowing customers to be defrauded. They must be held accountable and punitive damages that severely hurts them must be applied.
- hackernewds 2y agoWhich company should be held accountable? Snowflake or Ticketmaster? It's not as straightforward as expected
- jeroenhd 2y agoCustomers have no direct relation with Snowflake, so they should go after Ticketmaster. Ticketmaster can then decide to reclaim their losses from Snowflake or some kind insurance they may have, but that's not their customers' problem.
- Jgrubb 2y agoI’m trying to imagine the size and complexity of the indemnity clause in Ticketmaster’s contract with Snowflake, and the number of hours and redlines and delays, .. all the lawyers. All the delays for the analytics org, all the pain, all the suffering and now … it’s game time.
- lotsofpulp 2y agoThis is how you shut down people from starting a business unless they are super rich. Liability first thinking is gumming up the gears of society.
- eastbound 2y agoYes, those pesky regulations. Now you have to have a several exit doors if you want to open a nightclub. It keep it for the rich people.
- lotsofpulp 2y ago
- isoprophlex 2y agoThat single EC2 box with a couple hundred GBs of RAM running DuckDB is looking better and better each day, compared to getting billed up the butthole for snowflake...
- catchnear4321 2y agofar cheaper, yes, but duckdb leaks just as bad if you compromise credentials. also more than just an ec2 but that is always the risk. someone describes an idea in short form, someone else with insufficient experience tries to implement.
- billy_bitchtits 2y agoHow does that work for 100 data analyst users running reports and analyses?
- llm_trw 2y agoGiven what we've seen here 95 of those 100 analysts aren't employed in the original company.
- WaxProlix 2y agoIs there a built in/plug in feature for dashboarding and viz for duck db? Would be more appealing.
- Jgrubb 2y agoThere's Metabase.
- planetjones 2y agoPrevious HN comments indicated this could just be demo snowflake accounts, which were all compromised from a single individuals account at snowflake. But the announcements don’t seem consistent with this. Do we think propective customers really shared 100s of millions of real customer records for demo accounts? Or more likely the sales person was granted access to production systems by the prospective clients, so their credential without MFA could be used to access many customers real data? I struggle to see how snowflake can blame the customer here; secure by default is something a customer should reasonably expect for their money.
- Jgrubb 2y agoI think if it’s one customer you could maybe blame the customer and get away with it. If it’s multiple at once, all those customers very obviously are just pointers back.
- dhoe 2y agoIt seems unlikely that a French SE would be working on Ticketmaster and Santander.
- alex-korr 2y agoMy guess is that it went down like this. Ticketmaster gave access to their production tenant to sales engineer that was probably attached to their account rep. He got an account with a set password, was not onboarded into their Okta/Azure AD/etc and didn't have MFA enabled for his account or was restricted to a range of IPs for access. He got p0wned and the hackers got in using his creds. Of course he likely had accountadmin or something highly privileged since he was likely routinely asked to look at random things at Ticketmaster... that too didn't help.
- naveen99 2y agoWhen will the shaolin temple start teaching rsa and transformers ? When Kwai chat kane can come save the day ?
- bitcharmer 2y agoNot a single data breach resulted in the guilty company's demise or even noticeable financial or reputational damage. Nothing will ever change and this will keep happening more often.
- throwaway63467 2y agoI don’t get how you can just use production credentials in these demo systems. Even if they don’t get leaked it would mean Snowflake employees have access to your production data? I’m working in the compliance space in Europe and here it would be pretty wild if a large corporate would give such kind of data access to a cloud company without tons and tons of red tape, US companies seem to be way more lenient about this.
- mahogany 2y agoYou might be misunderstanding (or I am). My reading of the situation currently is that the “real” breaches were due to actual production credentials leaking which were used to actually access production. Then some customers were not using MFA on their Snowflake account so credentials were all that were needed. The demo accounts were associated with a Snowflake SE which was just one part of the attack.
- 0x1ceb00da 2y agoAre they using temple OS to host their site?
- danielovichdk 2y agoI don't care who fucked it up. But not having MFA turned on should be reason enough for everyone to be sacked. So lazy
- deleted 2y ago[deleted]
- hatware 2y ago[dead]
- 1vuio0pswjnm7 2y agoCloud computing! Store data on someone else's computers instead of own computers. Benefits outweigh costs?