7 ms·
Marriott CEO shares post-mortem on last year's hack
- jak92 8y ago"The Guardium alert was triggered by a query from an administrator's account to return the count of rows from a table in the database," Sorenson said. Such queries are considered dangerous because the software that runs on top of a database doesn't usually need to make them. Really?
- ekimekim 8y ago"Dangerous" is probably the wrong word - anomalous is a better one. Most software doesn't need to know, say, how many entries there are in the Users table. It generally is querying for a specific user, or a range of users with some particular property. It sounds like they had some software watching for any out-of-the-ordinary queries, which they then manually verified with the apparent person who was executing them. When that person had no idea what they were talking about, that's when the red flags went up.
- sigi45 8y agoYour example with the user table is a little bit 'bad'. Because yes i like to see statistics as an admin on how many user acounts are there :P
- deleted 8y ago[deleted]
- SlowRobotAhead 8y agoWell put. I think that anything I personally do while logged into a database when viewed from any higher level going to look anomalous. That said, I wonder if what’s going on here is that the admin account was never supposed to be used directly, so that’s why the alert triggered.
- bluntfang 8y agowell gosh i hope you're not running queries on the command line on a production database.
- EpicEng 8y agoSo you routinely log in as an admin and run sql queries directly on a production DB? That seems quite a bit more "bad" than the example.
- JetSpiegel 8y agophpMyAdmin or something might run on behalf of the administrator and do these queries.
- kakwa_ 8y agoI do it several times per day. When you are maintaining a piece of garbage application, completely undersized, absurdly opened in term of filtering and extremely badly designed, it's the only way to manage this mess, not to mention the times when I have to fix the DB by hand, replaying, with some manual tweaks, queries normaly executed by the application. And unfortunately, I don't think that's so uncommon.
- munk-a 8y agoIt's not particularly common but I've been in some what of a similar situation. If you're a senior techy person I'd suggest taking a bit of time to either find a new position (as I'm sure all of HN would echo, simply for your sanity) or I'd suggest taking that time to assemble a report about how your spend your hours each day and make sure it bubbles up past your manager to someone who deals with money. That sort of situation is constantly costing the company money for no good reason, it'd be fiscally sound to spell out these costs and contrast them with any estimates you could assemble on making the application start working again. Manually replaying queries is expensive, terrible, terrifying, and can potentially break data-integrity really badly.
- ethbro 8y ago> assemble a report about how your spend your hours each day and make sure it bubbles up past your manager to someone who deals with money Assumes poster works at a company where management cares past "Does it work?" That's less rare than it used to be, but HN sometimes underappreciates how much of the economy is still governed by companies with CFOs who couldn't tell you the first thing about their enterprise infrastructure.
- deleted 8y ago[deleted]
- snazz 8y agoThat is strange. If it came from an administrator account, it didn’t come from the software “that runs on top of the database”.
- jak92 8y agoExactly. But an administrator actually may have a use for doing a count statement.
- wyldfire 8y agoSo then an administrator wouldn't be surprised or concerned if an auditor asked them "Hey did you issue this query the other day?"
- snazz 8y agoThat part seems logical, but the query on its own shouldn’t be cause for concern.
- ars 8y agoUse the word "atypical" instead. If your audit system knows that this application never runs such queries, and today it did, it knows something changed and flags it.
- windwake12 8y agoAll properly configured applications should be running with least required db credentials, limiting what they can do. That Accenture was monitoring all administrative level queries against the db makes perfect sense in a locked down production environment.
- wyldfire 8y agoYour question is unclear. IBM designed this product to audit for things that look like signs of an attack/probing and if it generated a lot of false alerts, the product would be worthless. How often do you design a database query to get the count of all rows from a table? It sounds like an uncommon use case, but probably one you could mute from the audit if you do it intentionally.
- ceejayoz 8y ago> How often do you design a database query to get the count of all rows from a table? Any time I paginate a list, so it can show the "x of y pages"?
- omouse 8y agoYou aren't keeping the count somewhere else, in a cache, let's say?
- ceejayoz 8y agoSure, but even if you're caching, you're presumably populating that "how many users do we have" widget's cache at least once a day or so.
- ars 8y agoSpecifically in this case, there's no reason to show a list of all of the users in the system. What would you do with a list with millions of rows? A search makes more sense here.
- ceejayoz 8y ago> What would you do with a list with millions of rows? Not much, but that doesn't mean folks won't. https://stackoverflow.com/questions https://stackoverflow.com/questions says "17,335,212 questions " and the pagination goes "1 2 3 4 5 … 346705", if you want a real-world example.
- bradjohnson 8y ago
- jsight 8y agoSince the table in question isn't mentioned in that quote, is it possible that what was really anomalous was a number of similar queries on somewhat unusual tables? Meaning, something that indicated some software was doing queries that were both unusual and somewhat high frequency?
- cmroanirgo 8y agoMy db security is rather lax, and wouldn't notice an anomalous row count, and certainly wouldn't generate a "red flag" as a result. Anyone know of some lightweight foss monitoring tools that are in this arena? (for single server MySQL setups).
- xemdetia 8y agoFor single host MySQL setups there is the mysql audit plugin and related configuration. It's not perfect but if you already have other SIEM/alert infrastructure it should glue in OK. You'll want to test it to make sure it doesn't interfere with expected SLA's.
- bahmboo 8y agoYou point out what alarmed me. An unknown entity had admin access. That's 5 alarm stuff....but they waited 3 days to look into it. Pasted here again because formatting: "As part of our investigation into the alert, we learned that the individual whose credentials were used had not actually made the query,"
- munk-a 8y agoYea... that's like saying "Oh yea, this attacker did have root access and we checked the logs of what he did and apparently he just logged in and then three hours later he edited some log file and logged out... it's weird he didn't do anything while on the server..."
- borumpilot 8y agoThat is just regular DAM (Database Activity Monitoring). I know few companies use it on even fewer DB's but it can really help to identify "rogue access" to your data. It takes (a lot) of time to tone it down to your specific needs and DB access, but once it is up and running it can trigger at the right time (as in the article). There are different kinds of DAM, inline just reading the network traffic, missing the direct access accounts would have from the DB server itself, and the ones that monitor the DB service memory directly. Of course you could start way cheaper / simpler with some proper logging from the DB software itself. It is a growing security area that is a nice (once tamed) additional information source.
- guitarbill 8y agoThis is basically a PR piece on how careful they're being and how "seriously" they're treating it. > Uncovering the full scope of the attack took significant forensic work, the CEO said. However, despite the RAT's presence on the Starwood IT system, at that point, there was no evidence that unauthorized parties had accessed customer data located in Starwood's guest reservation database. > By the next month, in October, the forensic firm also found Mimikatz [...]. The tool was most likely used to help hackers acquire passwords for other Starwood systems and help them move to other parts of the IT network. > Yet again, investigators didn't find evidence that hackers had accessed customer data. > Yet again, there was still no evidence of hackers accessing customer data. With infrastructure so compromised, and so few protective measures, the default assumption has to be some kind of customer data was accessed. Spin at its worst. And while I haven't watched the hearing, you'd expect a post-mortem to include maybe how they gained access, and measures to fix it in future. Even if the CEO didn't elaborate, you'd expect a reporter to ask similar questions, and at least point out nothing was said.
- judge2020 8y agoI also like how there's no mention of revoking the admins' credentials after Guardium detected the anomaly. I would say it's obvious that they changed the password, but then I may be giving them too much credit.
- airstrike 8y ago> there was no evidence that unauthorized parties had accessed customer data located in Starwood's guest reservation database. This has to be one of the most disingenuous phrases one could possibly utter about an event like this. The fact that the attacker left no evidence (or that you failed to identify whatever evidence was left behind) in no way assures people that their data wasn't stolen. It's fallacious and it reeks of negligence and a complete disregard for accountability. I'm never staying at a Marriott again if I can help it. It's not much, but it's better than nothing... Last year I hosted a corporate event at a Marriott in NY and we paid north of $15k.
- wvenable 8y agoDid you read the entire article? Once they had confirmed that customer data was stolen, they informed the authorities and the public. I don't think any company would do that differently.
- time_for_toll 8y agoNo he/she only wanted to boast about the $15k.
- tomcatfish 8y agoIt's the same point the current highest comment made: Even though the system was compromised, they assumed the best case instead of the worst case as would be expected with customer data. The money seems to be a proof that their impact will be larger than just a person who stays at a hotel twice a year.
- wvenable 8y agoI don't think that's how to read it. They probably did not assume that at all. But they are not going to go public with any information until they have confirmation. You're reading a postmortem and not necessary exactly what they were thinking at the time.
- deleted 8y ago
- elliekelly 8y ago> The Marriott CEO said again that investigative efforts have yet to uncover evidence to suggest that hackers gained access to the encryption key used to encrypt the payment card numbers, meaning that most of the compromised payment card numbers are still useless to attackers. So the clearly sophisticated hacker(s) had access to the network for four years yet somehow forgot the private keys to decrypt the millions of records they went through all that effort to steal?
- linsomniac 8y agoI think you are misunderstanding... They got dumps of a couple of databases, which they encrypted, downloaded, and presumably decrypted. Those dumps themselves contained a credit card number that was encrypted by Marriott with a key they don't think the attackers got. That is my read of it anyway.
- gwbas1c 8y agoWhat I really want to know is why wasn't this discovered as part of the due diligence for the acquisition? I thought it was normal to run a security audit of whatever systems a business uses prior to the acquisition?
- rhizome 8y agoLong story short: he's not in prison.
- DaniloDias 8y agoIt had never occurred to me to define a bunch of select * statements in production that would trigger alerts. Brilliant.