4 ms·

>It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How do you read linux fire

>It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How do you read linux firewall documentation with the first example being old school public->DMZ->private? So your web server got hacked? That means what? Your database is across a single port through the firewall, and ACLs mean that any traffic arriving at the part from the DMZ CIDR can only do the limited set of queries that provide single-person-at-time. SSH from hacked box to database? Got a zero-day for the firewall (because 22 isn't open from there), and a zero-day for sshd (because its keys only). I know this and I'm not one of our hundreds of security people. (And as I'm not the security guy, if any actual security guys want to school me here, please do so!) Sure, I agree that business peeps will say "Oh that's too expensive I don't care", but lets not pretend that we don't know how to do this. Be clear, if you are the IT security guy, and you agree to bow down to "business" people (you're a business person) who tell you "No, that's too complicated", then just be clear that you are getting fired when the shit hits the fan.

sillysaurus3 9y ago
Yep. You can do all that. At the end of it, a box somewhere needs to be able to access the data. That box has access. You handwave it as single-person-at-a-time queries. But on any given day, they have to run tens or hundreds of thousands of those queries. Even if people aren't accessing their credit info, there is plenty of automation that does. Meaning if you pop that single person query box, you can scale up the queries by a factor of ten without raising any flags directly. It's much better, certainly. Companies should do that. But it's not obviously a panacea.
- lowbloodsugar 9y ago
  Conceivably, the attackers spent the 75 days downloading the 143,000,000 records, 2,000,000 records per day, and managed to trigger no scaling alarms as a result. Or perhaps there were no alarms. I wonder if we'll ever know.
  - sova 9y ago
    75 days? The report by Equifacks makes it sound like a one day delay from attack to taking the machine offline. Sneaky language.
  - closeparen 9y ago
    TFA says Equifax learned of the breach by detecting anomalous traffic. I'd say there was an alarm, but it didn't do anything to actually stop the attack.
  - AlexCoventry 9y ago
    Given the vagueness of their statements about the 143M records which "may" have been affected, it seems likely that they weren't logging the relevant queries in any useful way, and that is definitely an architectural failure.
- mdpopescu 9y ago
  I worked for a company that was doing at least 100k credit checks each day. A lot of it was batched - we would get a file with 25k records and had to process it and send back the results (usually HTML reports and/or decisions based on them). What I want to say is, it wouldn't surprise me to hear that the three main bureaus process millions of requests per day. Oh, and also - all bureaus were replying to a request in less than a second. Quite impressive.
  - sillysaurus3 9y ago
    Thank you for backing up my claim with data. I was admittedly making an educated guess, but there was a nagging "hundreds of thousands? Are you sure?" feeling. This underscores the importance of figuring out a better architecture. And not just for credit bureaus. It is remarkable how resistant people are to the idea that you need to be patient while the world changes. Everyone is out for blood and gives not a fleeting thought about what would come after such a purge.
technion 9y ago
I feel there's a couple of places where this doesn't reflect reality. The first very likely scenario is given business logic says a web server application needs certain data, where "certain" is "pretty much all of it". So your application server in that DMZ has a database credential it needfully holds. Which in turn is a credential an attacker can use to access "pretty much everything". single person at a time The majority of modern web applications will open a series of database connections. One application server limited to one DB connection for a public application is probably going to break the app. Overall, security people don't get the choice about "bowing" to business people in most of my experiences. I'm not saying scape goating never happens, but this then just be clear that you are getting fired when the shit hits the fan. What do you think the alternative is? Tell them you're quitting early on because they won't do what you say? I can present risks, and offer mitigations. More often than not, some get through easily, some never will, and it's those on the fence you work on constantly improving.
- lowbloodsugar 9y ago
  Heh. I don't mean the database can do only one query at a time. I mean the db is set up so the front end can't just hit it with a SQL query to dump the database. Now, sure, if the front ends query 10% of all records every day, then yeah, it only takes 10 days to get all the data, one query at a time, without triggering any alarms. They had 75 days. I don't know though, on any given day, what percentage of the population are getting credit checked? And did they have any alarms?
  - technion 9y ago
    Thanks for that clarification, it does make more sense. I think there are still practical limits to that. If your application can do: SELECT * FROM users WHERE ID = 10 Why not SELECT * FROM users where ID > 0 Honestly this is something I'd like to be able to put in place in certain areas, but I'm not aware of a sane strategy to do so existing.
    - krallja 9y ago
rectang 9y ago
> just be clear that you are getting fired when the shit hits the fan. How many jobs are out there for people who tell the "business" people what they want to hear, versus people who tell them hard truths about security? How many businesses succeed despite lax security -- gambling irresponsibly, yet winning?
chx 9y ago
If the business people at Equifax didn't spend through the nose on security then they are negligent, quite probably criminally so. You can argue that Target is not in the "we need to secure this treasure trove of data" business and so them leaking 40 million credit cards is just an unfortunate accident. But when you sit on the PII of 150 million people then you need to be top notch security wise. I am sure the eventually inevitable courts will appreciate a company culture where admin/admin is acceptable.

sillysaurus3 9y ago

Yep. You can do all that. At the end of it, a box somewhere needs to be able to access the data. That box has access. You handwave it as single-person-at-a-time queries. But on any given day, they have to run tens or hundreds of thousands of those queries. Even if people aren't accessing their credit info, there is plenty of automation that does. Meaning if you pop that single person query box, you can scale up the queries by a factor of ten without raising any flags directly. It's much better, certainly. Companies should do that. But it's not obviously a panacea.

lowbloodsugar 9y ago
Conceivably, the attackers spent the 75 days downloading the 143,000,000 records, 2,000,000 records per day, and managed to trigger no scaling alarms as a result. Or perhaps there were no alarms. I wonder if we'll ever know.
- sova 9y ago
  75 days? The report by Equifacks makes it sound like a one day delay from attack to taking the machine offline. Sneaky language.
- closeparen 9y ago
  TFA says Equifax learned of the breach by detecting anomalous traffic. I'd say there was an alarm, but it didn't do anything to actually stop the attack.
- AlexCoventry 9y ago
  Given the vagueness of their statements about the 143M records which "may" have been affected, it seems likely that they weren't logging the relevant queries in any useful way, and that is definitely an architectural failure.
mdpopescu 9y ago
I worked for a company that was doing at least 100k credit checks each day. A lot of it was batched - we would get a file with 25k records and had to process it and send back the results (usually HTML reports and/or decisions based on them). What I want to say is, it wouldn't surprise me to hear that the three main bureaus process millions of requests per day. Oh, and also - all bureaus were replying to a request in less than a second. Quite impressive.
- sillysaurus3 9y ago
  Thank you for backing up my claim with data. I was admittedly making an educated guess, but there was a nagging "hundreds of thousands? Are you sure?" feeling. This underscores the importance of figuring out a better architecture. And not just for credit bureaus. It is remarkable how resistant people are to the idea that you need to be patient while the world changes. Everyone is out for blood and gives not a fleeting thought about what would come after such a purge.

technion 9y ago

I feel there's a couple of places where this doesn't reflect reality. The first very likely scenario is given business logic says a web server application needs certain data, where "certain" is "pretty much all of it". So your application server in that DMZ has a database credential it needfully holds. Which in turn is a credential an attacker can use to access "pretty much everything". single person at a time The majority of modern web applications will open a series of database connections. One application server limited to one DB connection for a public application is probably going to break the app. Overall, security people don't get the choice about "bowing" to business people in most of my experiences. I'm not saying scape goating never happens, but this then just be clear that you are getting fired when the shit hits the fan. What do you think the alternative is? Tell them you're quitting early on because they won't do what you say? I can present risks, and offer mitigations. More often than not, some get through easily, some never will, and it's those on the fence you work on constantly improving.

lowbloodsugar 9y ago
Heh. I don't mean the database can do only one query at a time. I mean the db is set up so the front end can't just hit it with a SQL query to dump the database. Now, sure, if the front ends query 10% of all records every day, then yeah, it only takes 10 days to get all the data, one query at a time, without triggering any alarms. They had 75 days. I don't know though, on any given day, what percentage of the population are getting credit checked? And did they have any alarms?
- technion 9y ago
  Thanks for that clarification, it does make more sense. I think there are still practical limits to that. If your application can do: SELECT * FROM users WHERE ID = 10 Why not SELECT * FROM users where ID > 0 Honestly this is something I'd like to be able to put in place in certain areas, but I'm not aware of a sane strategy to do so existing.
  - krallja 9y ago

rectang 9y ago

> just be clear that you are getting fired when the shit hits the fan. How many jobs are out there for people who tell the "business" people what they want to hear, versus people who tell them hard truths about security? How many businesses succeed despite lax security -- gambling irresponsibly, yet winning?

chx 9y ago

If the business people at Equifax didn't spend through the nose on security then they are negligent, quite probably criminally so. You can argue that Target is not in the "we need to secure this treasure trove of data" business and so them leaking 40 million credit cards is just an unfortunate accident. But when you sit on the PII of 150 million people then you need to be top notch security wise. I am sure the eventually inevitable courts will appreciate a company culture where admin/admin is acceptable.