21 ms·

Fresh Horrors from Equifax CEO Richard Smith's Congressional Hearing

290 points by wglb 9y ago

fractal618 9y ago
Nuts
dtran 9y ago
> When asked by multiple lawmakers why Equifax set up this separate site, Smith said the company's main domain was not architected to process the enormous traffic the company knew would come its way after the announcement. In all, Smith said, the independent breach-response site has had 400 million consumer visits, which would have crumpled the main site. Apparently they've never heard of subdomains, static pages, reverse proxies, and <insert many other solutions anyone on HN could come up with off the top of their heads>?
- rconti 9y ago
  Right. I think it's code for "I think domain means website, and our technical folks said our existing corp site didn't have the capacity, so we told them to setup a 'completely separate site'".
  - brandnewlow 9y ago
    Also maybe "Our marketing folks want all the angry links to point to a separate domain we don't care aboout vs. our main web site"
    - mrkurt 9y ago
      This is PR recommendation #1 when you get breached.
    - mikekij 9y ago
      ^ This. SEO basically.
      - KGIII 9y ago
        It strikes me more as a move suggested by a reputation management company, though they may have in-house specialists. While usually, and intentionally, behind the scenes, reputation management companies are often involved in things whee there is massive public outcry.
- austenallred 9y ago
  Ya, that solution alone says a lot about the technical decision-making taking place. Especially given that random numbers input to the site returned that data had been compromised.
- akg_67 9y ago
  Most probably their marketing and legal team came up with the idea. Once the outcry blows over turn off the site and all incoming external links become dead. After a while general public will not remember and wouldn’t associate their main site with this incident through search. Legal might have encouraged to have two fully independent sites. I will not be surprised two sites belong to two separate legal entities. Just a speculation on my part considering how US corporate behave.
  - wmccullough 9y ago
    Without looking it up, I’ll throw my chips on the table that they offshore the majority of their IT.
    - mavelikara 9y ago
      Offshore or outsource?
    - greedy_buffer 9y ago
      Maybe. The separate domain might be explained by lack of trust in their PR company though. See Krebs' article: https://krebsonsecurity.com/2017/09/equifax-breach-response-turns-dumpster-fire/ https://krebsonsecurity.com/2017/09/equifax-breach-response-...
    - drieddust 9y ago
      That's just Horseshit. I have worked on both sides of the table and incompetence is equally widespread. However, vendors are not as bad as they seems to appear and employees are not as good as they seems to appear. Most of the contracts nowadays have a penalty clause and a stated Service Level Agreements (SLA) around performing the vulnerability scan and closing the GAP within a stipulated time frame. Outsourcing companies might not be innovative but they don't like to lose money on failing SLAs. Coming back to the issue of incompetence. I have often seen so called innovator of the just taking the credit for the work done by Vendor. The way typical scenario rolls out is this. Senior management of of the company will throw a challenge to the senior management of Vendor. After a few discussions middle management of will assign the responsibility to their lowest level employees. After a lot of hard work when solution is in sight, all communication channels will be closed. A few weeks later solution will be presented to their own management while keeping the vendor completely out of loop as if vendor was completely useless and they did all the hard work. Vendor will be presented as mindless robot who can just execute the instruction. Hell I have seen employees not even having the courtesy even the reword our solutions. Incentives of employees are aligned with making the vendor look less effective or they themselves will be replaced at some point by their own higher management. I think its not their fault. It is just the environment where painting the vendor in positive color will be detrimental to employee's own job.
- wglb 9y ago
  Seems to me to be dead simple. One dns entry for equifax.com, another pointing off to different hardware for equisecuritybreach.equifax.com. Nothing fancy at all. Make the page have a different look and feel.
  - speedplane 9y ago
    A ten-year-old can do it on godaddy in ten minutes.
- Mouse47 9y ago
  Not me :( I'm just a .NET developer, I have no idea what my code runs on.
  - nameless912 9y ago
    Assuming you're serious, get on that my dude! All the best developers I know have a solid mind for ops; operation of your application should inform its design.
- smsm42 9y ago
  That seems to be indication that decision process there is as broken as security process. Either they did not have a professional that could explain the management why the claim above is baloney and what are the solutions to make it work, or that professional wasn't consulted, or his opinion was dismissed by the management. In any case, the symphony of fail continues.
- 0xbear 9y ago
  On the positive side, their chief security officer can compose one heck of a musical composition.
- ComodoHacker 9y ago
  >the company's main domain was not architected to process the enormous traffic I understand all the sarcasm expressed here, but this actually may be true if they run their own DNS service. Subdomain isn't a solution in this case, but separate domain is.
  - Xylakant 9y ago
    I fail to see why DNS would be an issue. Set a sufficiently large TTL and you should see almost no queries hit your DNS servers once intermediate caches have seen the domain.
    - ComodoHacker 9y ago
      With this amount of traffic you have to load-balance, so you can't just set large TTL.
      - azinman2 9y ago
        But load balancing also applies to their new domain as well. Why would you need a separate hostname?
      - Xylakant 9y ago
        Look, you could just set the new subdomain as a cname with a large TTL and then handle that a-record via a different provider if you really need to. You could also choose to delegate the whole subdomain. Also, load-balancing via multiple a-records does not at all depend on the TTL. DNS is really the easiest part to handle here. Internal policy might prevent delegation, but that’s not a technical problem.
        ComodoHacker 9y ago
        How can I handle a subdomain with a different provider while handling the whole zone myself? I'm not an expert in DNS, so really curious.
- benmmurphy 9y ago
  from a security point of view it does kind of make sense to have non-trusted content hosted under a separate domain. so if they were using a third party to host the breach page it may have been risky to have it hosted under the .equifax.com domain. this is why you have .github.io and googleusercontent domains, etc. though, probably the risk from confusing users is worse than the risk from the third party playing silly games with cookies in this case.
uptown 9y ago
“IRS awards multimillion-dollar fraud-prevention contract to Equifax” The no-bid contract was issued last week, as the company continued facing fallout from its massive security breach. http://www.politico.com/story/2017/10/03/equifax-irs-fraud-protection-contract-243419 http://www.politico.com/story/2017/10/03/equifax-irs-fraud-p...
- aresant 9y ago
  The solution to stopping stupidity like this is to take a 12 month tour of duty at the US Digital Service: https://www.usds.gov/join#tours-of-duty https://www.usds.gov/join#tours-of-duty Yes the bureaucrats, lobbyists, and special interests will continue fucking everything up. But the USDS is set up so that at least competent technologists can throw up some objective defense vs. the government officials who really think Equifax is a viable solution aka "Nobody ever got fired buying an IBM."
  - kelnos 9y ago
    Yes and no. USDS doesn't exactly have domain over all US gov't IT decisions, and aren't even involved or have any influence over a lot of things. Not to mention that the idea of taking a massive pay cut and having to move to DC holds no appeal to me.
    - deleted 9y ago
      [deleted]
    - loeg 9y ago
      I'm also turned off by the idea of switching jobs just to switch again in one year. Even if the pay cut and DC were palatable.
  - make3 9y ago
    I can't imagine being a subordinate to Jared fucking Kushner for a full year
    - 9y ago
sputknick 9y ago
#2 is the most disturbing to me, simply because it's SO SIMPLE. Know your environment, patch it. My guess would be the reason their security scanners didn't alert to the missing patch was that they scanner had not been updated. Again, SO SIMPLE, get updated definitions anytime you scan. #4 I'm fine with, the CEO meeting on IT security quarterly sounds adequate to me. His involvement in IT security should not extend beyond ensuring policy is in place and being acted on. Quarterly is more than adequate to ensure this is being done.
- TallGuyShort 9y ago
  Agreed on #2. Quarterly meetings with CEO about security don't seem so bad in the general case (although in Equifax's case, security is a way bigger deal), but that shouldn't be all. Only 1 person knew about the patch? Even a medium-size enterprise where security is even a moderate concern should have more people than that focused on security 100% of the time.
- wglb 9y ago
  Quarterly is possibly fine, but when sparks are flying in the back end, and fluid is pouring out of the transmission on the running engine, CSO needs to be able to access CEO promptly.
- cookiecaper 9y ago
  First, no one gives a crap about security until something like this happens to them. The incentives are not in place to require it. Time and resources spent on security are time and resources not spent on beating the competitor. The best outcome you can hope for if you _do_ spend on security is that your competitor will have some apocalyptic breach similar to this one, and that doesn't happen most of the time, at least not in a way that results in any accountability or visibility. It's important to understand that because it's the reason why your bosses will never care unless/until they have an experience like this, or until they expect someone important to be looking for an opening (audits, due diligence, etc). Second, there are indeed some clowns and cronies hired at these sort of places, but a lot of the people are decently skilled. I'm speaking generally about the workforce at these larger enterprisey companies, as I don't know anyone at Equifax afaik. Technical competence is less of an issue than intra-managerial political games like "shift the blame" and "silo defense", at least some of the time. Part of "intra-managerial games" is having the least-skilled people in the most authoritative positions, leaving decision authority with those least equipped to evaluate a situation and determine a responsible response. I am sure that there is someone at Equifax who saw the news on this vulnerability and registered it as being something they needed to patch. I wouldn't be surprised at all to learn that this person didn't say anything because they learned long ago that there's nothing to be gained by actually trying to get the patch done in any non-routine manner. I also wouldn't be surprised to learn they did try to raise the alarm and were mocked and/or shut down, both by other technical groups trying to avoid the appearance of an issue with "their" section of the system, and/or by management, who are likely to interpret such attempts as alarmism, if not plain political malfeasance. Hate to say it but we're on the brink of a formalized licensing program for security, servers, etc. I am honestly surprised that the mainstream outlets and politicians have not been touting this yet, with all the high-profile and extremely costly data security breaches that have been occurring over the last few years, but I really expect it to come soon. I wouldn't be surprised at all to see Equifax become the Enron-like scapegoat for such legislation.
ceejayoz 9y ago
> In a moment of true dystopian chaos, the official Equifax Twitter account repeatedly tweeted a phishing link, mistaking it for the breach response page. It wasn't a phishing site, Wired. It was a site designed to point out that their approach is susceptible to phishing. Entering data in the form gave you a scolding. > The Equifax Breach Exposes America's Identity Crisis When asked by multiple lawmakers why Equifax set up this separate site, Smith said that the company's main domain was not architected to process the enormous traffic the company knew would come its way after the announcement. In all, Smith said the independent breach response site has had 400 million consumer visits, which would have crumpled the main site. They could have done a subdomain and pointed it at the site.
- positr0n 9y ago
  > It wasn't a phishing site, Wired. It was a site designed to point out that their approach is susceptible to phishing. Entering data in the form gave you a scolding. I'd argue it was a phishing site. Just fortunately it was controlled by white hats.
  - empath75 9y ago
    It never exfiltrated data, but it could have.
- matt_wulfeck 9y ago
  What a bizarre excuse. Do their IT teams know about subdomains?
  - macintux 9y ago
    In fairness, I could easily see people ignoring the subdomain and just going to the main site. Of course, that probably happened anyway.
  - greedy_buffer 9y ago
    As Brian Krebs reported in the "dumpster fire" article the response site was setup by their PR company. Perhaps they did not want to give away ownership of part of their domain... For security reasons
danjoc 9y ago
"The company announced Monday that the total number of people impacted by its breach is not 143 million—the amount it first disclosed—but in fact 145.5 million. Its ability to casually misplace 2.5 million lives upended by the breach is alarming" I can't really take the author too seriously after reading that.
- pc2g4d 9y ago
  Why not?
  - rawrmaan 9y ago
    It's kind of a ridiculous statement. A counting error is nowhere near is shocking as the initial breach, especially considering it happened after the breach.
    - GuB-42 9y ago
      It is bad because not only they let someone in but they don't know what he did despite having recorded everything (or so they said). It means that maybe more is to come.
      - KGIII 9y ago
        Forensics is hard for even companies that are well versed and adept with tech. Evidence suggests that Equifax is not among that group of companies. The number being revised doesn't surprise me. I'd have been surprised if it hadn't been revised. I'd expect additional revision, as the investigation continues and the breach more fully understood. For the record, that doesn't show I support them, nor am I making excuses for them. It's just the opposite. I believe them to be very, very inept - to the point of being incompetent, illsuited, and inexcusable. I expect additional revision, not because that is justified but because they simply suck that bad. I can't think of a breach handled quite this poorly, and that includes my biases from the OPM breach. I'd say the OPM breach was far worse but handled marginally better. Equifax had to work hard to take the crown, but they have managed to edge them out by a narrow margin.
tejaswiy 9y ago
> Second, Smith blamed a scanning system used to spot this sort of oversight that did not identify the customer-dispute portal as vulnerable. Smith said forensic investigators are still looking into why the scanner failed. Do these scanners ever work? Without naming names, the only reason we used this at a previous company was to kind of handwave around hey we have this tool doing regular security checks. > The first excuse Smith gave was "human error." He says there was a particular (unnamed) individual who knew that the portal needed to be patched but failed to notify the appropriate IT team. How is this one individual responsible for tracking all IT security vulnerabilities in all the technologies that equifax uses across its stack? How do they not have an admin team whose job it is to do these things?
- mikekij 9y ago
  Apparently the right user:security engineer ratio is 143,000,000:1. /s
- tehwebguy 9y ago
  Maybe he was talking about himself?
- jeremyjh 9y ago
  We do not know anything about the number of people who could have known about this issue and escalated it, but it does not surprise me at all that there is one primary person who is responsible for it. In fact, if there were not exactly one person who is responsible for the security of the application that would be a different sort of problem as diffuse responsibility can be neglected quite easily in large organizations. Still, at my own company - A LOT of people were focused on the timely remediation of this issue and there was no chance of it being left to one person, even for development environments of internal facing applications. The scanners do work, we used them at my company and they found this specific vulnerability. It surprised some development teams because there are libraries that use struts without really advertising it, so even applications that don't "use struts" had a transitive dependency on it, it was running in their container and showed up on the lists that were circulating.
- wglb 9y ago
cody8295 9y ago
To think it could have all been avoided with simple encryption. Seriously how does a company like Equifax not encrypt this kinda stuff?
- paulddraper 9y ago
  It was encrypted. (HTTPS/TLS)
  - wglb 9y ago
    That protects the data between the browser and the web server (or SSL offloader) from eavesdropping and tampering. It does not protect against the case where the attacker gains authorization, over HTTPS, to administrative access at the web server.
    - paulddraper 9y ago
      Right. So what hypothetical encryption was cody8295 referring to?
      - FuckOffNeemo 9y ago
        As stated above, the data was not encrypted at rest.
        paulddraper 9y ago
        Encrypting at rest wouldn't have solved anything. They didn't steal the disk. They gained access to the web server, which would have access to the unencrypted data.
  - FuckOffNeemo 9y ago
    The article specifically states that encryption in transit is provided but not at rest, which is what the OP is referencing too.
- 9y ago
cpitman 9y ago
#2 isn't surprising. A ton of shops only patch on a quarterly basis, and even then they only consistently patch the OS. Only when there is a vendor (like my employer, Red Hat) sending out notices that there is a critical vulnerability do people move any faster. Which is twice as bad for application dependencies. These don't even have quarterly patch cycles. Dependencies may be updated when a new release is deployed, which may be a couple times a year or never. One of my favorite questions for clients is "who is responsible for patching applications with no development team anymore?" This is an underappreciated benefit of CICD. An automated process lets you get a central group to take ownership of third party libraries and their security, and then let them trigger all applications to rebuild and release. Especially with containers, this is essential.
- gsylvie 9y ago
  THIS: > "who is responsible for patching applications with no development team anymore?" Struts 2.2.3 (oldest vulnerable version) was released on May 7, 2011. Personally I find it easy to imagine an app depending on that, going into prod, and just chugging away, forgotten for 6 years on some obscure corner of an enterprise's public (or internal) web assets.
  - greedo 9y ago
    Getting business lines to sign off and invest the resources to update applications can be a huge challenge. Shiny new applications get all the attention, but maintaining and updating legacy applications is not prioritized.
  - speedplane 9y ago
    I personally have no idea how large companies manage this. Every time I perform a full Docker update on all of the libraries and applications my app is running on, something breaks. Sure, tests catch it, but it takes a developer and time to figure out how to fix it. No idea how huge non-developer organizations handle this.
ams6110 9y ago
I don't see any of these things as "fresh horrors." They were either already known, or entirely predictible given what is already known. I'm sure the record count will be revised upwards several more times.
konceptz 9y ago
“cybersecurity experts from the law firm King & Spalding“ Did it strike anyone as disheartening that legal was the first call?
- wglb 9y ago
  Actually it is not. There are good reasons to call outside counsel first.
- twunde 9y ago
  Not really. A specialist law firm can give you a risk assessment, give you an overview of all relevant regulations that you need to follow, recommend security firms, connect you to any federal agencies that can help or need to know and all discussed in layman's terms (or at least exec-speak). If your CSO is good, then it's a second opinion. If he or she is not, then it's the only way to get the help you need.
yalogin 9y ago
Why isn’t Equifax offering free credit monitoring and identity protection for all their customers? I had to go and put a fraud alert on mine but I feel this should have happened the moment they knew about the hack. Even now there is no talk of that. Of course I wouldn’t have confidence in what they do but am hoping the other two are not totally incompetent.
- MertsA 9y ago
  You aren't their customer.
LyalinDotCom 9y ago
No one important will go to jail. If someone important is prosecuted and convicted they will be let off very very lightly. No companies will learn any lessons from this, as companies are already doing everything they can within skillset limits of their employees and budgets, or they are not and don't give a shit. No one really cares about the consumer in all of this, they will do the minimum not to look bad or face further legal issues but in the end people will invest lots of hours dealing with any problems that might arise. Who will pay for those hours? lol im just kidding, once again they dont give a shit really, good luck to us. Nothing will change in the industry for companies like Equifax, this is a very small nitch market and we cant seriously hope for any government mandated reforms in today's political climate. This will blow over soon and no one will remember it after the next breach or mass shooting or a war that Trump will start. Sorry to say all this but sometimes people need to say the truth.
- existencebox 9y ago
  I don't disagree with any particular assertion you make. Let me ask the obvious question. What do we do then? (If this doesn't spark discussion I'll gladly contribute my own ideas, which involve finding methods of partially-civil disobedience to even the scales of inconvenience a little bit. I honestly believe a large issue in the current system is that there's no real pressure being held against any of the people with real power. Why would someone who can bring in tens of millions _FROM FAILING AS BADLY AS ONE CAN FAIL_ in their job, give a damn about anything but optimizing that number and minimizing time/effort spent to achieve that? With senate and house incumbency rates hovering around 90%, why would a politician ever deviate from the status quo? The wellbeing of the people is so far separated from any of their driving incentives, we must find ways to realign that.)
  - Joeri 9y ago
    What caused things to turn around in the 30’s was people getting so fed up they voted in a different crew who actually fixed things. Voting (and voter outreach) is still the best path to reform.
    - 9y ago
deleted 9y ago
[deleted]
jaf7 9y ago
> (In a moment of true dystopian chaos, the official Equifax Twitter account repeatedly tweeted a phishing link, mistaking it for the breach-response page.) These institutions are cancerous devolutions in their current form. They are no better than predatory lenders. What has motivated their current structure seems cynical at the core.
thisisit 9y ago
Most of the time person giving the orders is wholly inadequate to understand what they are doing. At one of my previous jobs, the system I was working on was hacked years ago. You would think, this will give people some semblance of what not to do. No one seemed to have heard of the concept known as PII. Salaries of the entire company was sent out daily attached as a text file in an email to a very large group of people. The claim to do nothing was? It contains employee ids so it's kind of masked. And "many" people don't understand the data anyways. Trying to explain the "powers that be" why this was against industry standard fell on deaf years. I am sure if things do go wrong they will blame that "one guy" who was unable to "convince" them of the right thing to do.
paulajohnson 9y ago
#3 bugs me. It gets asked every time there is a breach, but its irrelevant. Encryption at rest is only useful if the only way to access the data is to type in the key. But for Equifax there are going to be hundreds or thousands of accesses per second. If you encrypt the data then you have added no protection at all because you still have a huge pipeline out through an always-on decryption mechanism. Any attacker is going to access the data through that mechanism and ignore the encryption completely.
- danpalmer 9y ago
  It depends on how they get access, and it can prevent some types of access. For example, they can’t steal backups, and they may need access through a web app server that has the database key rather than being able to go directly to the database. It’s not a catch-all solution but it is correct security practice and they should have been doing it.
- smsm42 9y ago
  Depends on vulnerability. If you have RCE (like I understand Equifax had) then encryption at rest is useless, the code can access the data, obviously, so the RCE code can too (maybe with a little more work, but ultimately it is not a barrier anymore).

290 points by wglb 9y ago

Nuts

> When asked by multiple lawmakers why Equifax set up this separate site, Smith said the company's main domain was not architected to process the enormous traffic the company knew would come its way after the announcement. In all, Smith said, the independent breach-response site has had 400 million consumer visits, which would have crumpled the main site. Apparently they've never heard of subdomains, static pages, reverse proxies, and <insert many other solutions anyone on HN could come up with off the top of their heads>?

rconti 9y ago
Right. I think it's code for "I think domain means website, and our technical folks said our existing corp site didn't have the capacity, so we told them to setup a 'completely separate site'".
- brandnewlow 9y ago
  Also maybe "Our marketing folks want all the angry links to point to a separate domain we don't care aboout vs. our main web site"
  - mrkurt 9y ago
    This is PR recommendation #1 when you get breached.
  - mikekij 9y ago
    ^ This. SEO basically.
    - KGIII 9y ago
      It strikes me more as a move suggested by a reputation management company, though they may have in-house specialists. While usually, and intentionally, behind the scenes, reputation management companies are often involved in things whee there is massive public outcry.
austenallred 9y ago
Ya, that solution alone says a lot about the technical decision-making taking place. Especially given that random numbers input to the site returned that data had been compromised.
akg_67 9y ago
Most probably their marketing and legal team came up with the idea. Once the outcry blows over turn off the site and all incoming external links become dead. After a while general public will not remember and wouldn’t associate their main site with this incident through search. Legal might have encouraged to have two fully independent sites. I will not be surprised two sites belong to two separate legal entities. Just a speculation on my part considering how US corporate behave.
- wmccullough 9y ago
  Without looking it up, I’ll throw my chips on the table that they offshore the majority of their IT.
  - mavelikara 9y ago
    Offshore or outsource?
  - greedy_buffer 9y ago
    Maybe. The separate domain might be explained by lack of trust in their PR company though. See Krebs' article: https://krebsonsecurity.com/2017/09/equifax-breach-response-turns-dumpster-fire/ https://krebsonsecurity.com/2017/09/equifax-breach-response-...
  - drieddust 9y ago
    That's just Horseshit. I have worked on both sides of the table and incompetence is equally widespread. However, vendors are not as bad as they seems to appear and employees are not as good as they seems to appear. Most of the contracts nowadays have a penalty clause and a stated Service Level Agreements (SLA) around performing the vulnerability scan and closing the GAP within a stipulated time frame. Outsourcing companies might not be innovative but they don't like to lose money on failing SLAs. Coming back to the issue of incompetence. I have often seen so called innovator of the just taking the credit for the work done by Vendor. The way typical scenario rolls out is this. Senior management of of the company will throw a challenge to the senior management of Vendor. After a few discussions middle management of will assign the responsibility to their lowest level employees. After a lot of hard work when solution is in sight, all communication channels will be closed. A few weeks later solution will be presented to their own management while keeping the vendor completely out of loop as if vendor was completely useless and they did all the hard work. Vendor will be presented as mindless robot who can just execute the instruction. Hell I have seen employees not even having the courtesy even the reword our solutions. Incentives of employees are aligned with making the vendor look less effective or they themselves will be replaced at some point by their own higher management. I think its not their fault. It is just the environment where painting the vendor in positive color will be detrimental to employee's own job.
wglb 9y ago
Seems to me to be dead simple. One dns entry for equifax.com, another pointing off to different hardware for equisecuritybreach.equifax.com. Nothing fancy at all. Make the page have a different look and feel.
- speedplane 9y ago
  A ten-year-old can do it on godaddy in ten minutes.
Mouse47 9y ago
Not me :( I'm just a .NET developer, I have no idea what my code runs on.
- nameless912 9y ago
  Assuming you're serious, get on that my dude! All the best developers I know have a solid mind for ops; operation of your application should inform its design.
smsm42 9y ago
That seems to be indication that decision process there is as broken as security process. Either they did not have a professional that could explain the management why the claim above is baloney and what are the solutions to make it work, or that professional wasn't consulted, or his opinion was dismissed by the management. In any case, the symphony of fail continues.
0xbear 9y ago
On the positive side, their chief security officer can compose one heck of a musical composition.
ComodoHacker 9y ago
>the company's main domain was not architected to process the enormous traffic I understand all the sarcasm expressed here, but this actually may be true if they run their own DNS service. Subdomain isn't a solution in this case, but separate domain is.
- Xylakant 9y ago
  I fail to see why DNS would be an issue. Set a sufficiently large TTL and you should see almost no queries hit your DNS servers once intermediate caches have seen the domain.
  - ComodoHacker 9y ago
    With this amount of traffic you have to load-balance, so you can't just set large TTL.
    - azinman2 9y ago
      But load balancing also applies to their new domain as well. Why would you need a separate hostname?
    - Xylakant 9y ago
      Look, you could just set the new subdomain as a cname with a large TTL and then handle that a-record via a different provider if you really need to. You could also choose to delegate the whole subdomain. Also, load-balancing via multiple a-records does not at all depend on the TTL. DNS is really the easiest part to handle here. Internal policy might prevent delegation, but that’s not a technical problem.
      - ComodoHacker 9y ago
        How can I handle a subdomain with a different provider while handling the whole zone myself? I'm not an expert in DNS, so really curious.
benmmurphy 9y ago
from a security point of view it does kind of make sense to have non-trusted content hosted under a separate domain. so if they were using a third party to host the breach page it may have been risky to have it hosted under the .equifax.com domain. this is why you have .github.io and googleusercontent domains, etc. though, probably the risk from confusing users is worse than the risk from the third party playing silly games with cookies in this case.

uptown 9y ago

“IRS awards multimillion-dollar fraud-prevention contract to Equifax” The no-bid contract was issued last week, as the company continued facing fallout from its massive security breach. http://www.politico.com/story/2017/10/03/equifax-irs-fraud-protection-contract-243419 http://www.politico.com/story/2017/10/03/equifax-irs-fraud-p...

aresant 9y ago
The solution to stopping stupidity like this is to take a 12 month tour of duty at the US Digital Service: https://www.usds.gov/join#tours-of-duty https://www.usds.gov/join#tours-of-duty Yes the bureaucrats, lobbyists, and special interests will continue fucking everything up. But the USDS is set up so that at least competent technologists can throw up some objective defense vs. the government officials who really think Equifax is a viable solution aka "Nobody ever got fired buying an IBM."
- kelnos 9y ago
  Yes and no. USDS doesn't exactly have domain over all US gov't IT decisions, and aren't even involved or have any influence over a lot of things. Not to mention that the idea of taking a massive pay cut and having to move to DC holds no appeal to me.
  - deleted 9y ago
    [deleted]
  - loeg 9y ago
    I'm also turned off by the idea of switching jobs just to switch again in one year. Even if the pay cut and DC were palatable.
- make3 9y ago
  I can't imagine being a subordinate to Jared fucking Kushner for a full year
  - 9y ago

sputknick 9y ago

#2 is the most disturbing to me, simply because it's SO SIMPLE. Know your environment, patch it. My guess would be the reason their security scanners didn't alert to the missing patch was that they scanner had not been updated. Again, SO SIMPLE, get updated definitions anytime you scan. #4 I'm fine with, the CEO meeting on IT security quarterly sounds adequate to me. His involvement in IT security should not extend beyond ensuring policy is in place and being acted on. Quarterly is more than adequate to ensure this is being done.

TallGuyShort 9y ago
Agreed on #2. Quarterly meetings with CEO about security don't seem so bad in the general case (although in Equifax's case, security is a way bigger deal), but that shouldn't be all. Only 1 person knew about the patch? Even a medium-size enterprise where security is even a moderate concern should have more people than that focused on security 100% of the time.
wglb 9y ago
Quarterly is possibly fine, but when sparks are flying in the back end, and fluid is pouring out of the transmission on the running engine, CSO needs to be able to access CEO promptly.
cookiecaper 9y ago
First, no one gives a crap about security until something like this happens to them. The incentives are not in place to require it. Time and resources spent on security are time and resources not spent on beating the competitor. The best outcome you can hope for if you _do_ spend on security is that your competitor will have some apocalyptic breach similar to this one, and that doesn't happen most of the time, at least not in a way that results in any accountability or visibility. It's important to understand that because it's the reason why your bosses will never care unless/until they have an experience like this, or until they expect someone important to be looking for an opening (audits, due diligence, etc). Second, there are indeed some clowns and cronies hired at these sort of places, but a lot of the people are decently skilled. I'm speaking generally about the workforce at these larger enterprisey companies, as I don't know anyone at Equifax afaik. Technical competence is less of an issue than intra-managerial political games like "shift the blame" and "silo defense", at least some of the time. Part of "intra-managerial games" is having the least-skilled people in the most authoritative positions, leaving decision authority with those least equipped to evaluate a situation and determine a responsible response. I am sure that there is someone at Equifax who saw the news on this vulnerability and registered it as being something they needed to patch. I wouldn't be surprised at all to learn that this person didn't say anything because they learned long ago that there's nothing to be gained by actually trying to get the patch done in any non-routine manner. I also wouldn't be surprised to learn they did try to raise the alarm and were mocked and/or shut down, both by other technical groups trying to avoid the appearance of an issue with "their" section of the system, and/or by management, who are likely to interpret such attempts as alarmism, if not plain political malfeasance. Hate to say it but we're on the brink of a formalized licensing program for security, servers, etc. I am honestly surprised that the mainstream outlets and politicians have not been touting this yet, with all the high-profile and extremely costly data security breaches that have been occurring over the last few years, but I really expect it to come soon. I wouldn't be surprised at all to see Equifax become the Enron-like scapegoat for such legislation.

ceejayoz 9y ago

> In a moment of true dystopian chaos, the official Equifax Twitter account repeatedly tweeted a phishing link, mistaking it for the breach response page. It wasn't a phishing site, Wired. It was a site designed to point out that their approach is susceptible to phishing. Entering data in the form gave you a scolding. > The Equifax Breach Exposes America's Identity Crisis When asked by multiple lawmakers why Equifax set up this separate site, Smith said that the company's main domain was not architected to process the enormous traffic the company knew would come its way after the announcement. In all, Smith said the independent breach response site has had 400 million consumer visits, which would have crumpled the main site. They could have done a subdomain and pointed it at the site.

positr0n 9y ago
> It wasn't a phishing site, Wired. It was a site designed to point out that their approach is susceptible to phishing. Entering data in the form gave you a scolding. I'd argue it was a phishing site. Just fortunately it was controlled by white hats.
- empath75 9y ago
  It never exfiltrated data, but it could have.
matt_wulfeck 9y ago
What a bizarre excuse. Do their IT teams know about subdomains?
- macintux 9y ago
  In fairness, I could easily see people ignoring the subdomain and just going to the main site. Of course, that probably happened anyway.
- greedy_buffer 9y ago
  As Brian Krebs reported in the "dumpster fire" article the response site was setup by their PR company. Perhaps they did not want to give away ownership of part of their domain... For security reasons

danjoc 9y ago

"The company announced Monday that the total number of people impacted by its breach is not 143 million—the amount it first disclosed—but in fact 145.5 million. Its ability to casually misplace 2.5 million lives upended by the breach is alarming" I can't really take the author too seriously after reading that.

pc2g4d 9y ago
Why not?
- rawrmaan 9y ago
  It's kind of a ridiculous statement. A counting error is nowhere near is shocking as the initial breach, especially considering it happened after the breach.
  - GuB-42 9y ago
    It is bad because not only they let someone in but they don't know what he did despite having recorded everything (or so they said). It means that maybe more is to come.
    - KGIII 9y ago
      Forensics is hard for even companies that are well versed and adept with tech. Evidence suggests that Equifax is not among that group of companies. The number being revised doesn't surprise me. I'd have been surprised if it hadn't been revised. I'd expect additional revision, as the investigation continues and the breach more fully understood. For the record, that doesn't show I support them, nor am I making excuses for them. It's just the opposite. I believe them to be very, very inept - to the point of being incompetent, illsuited, and inexcusable. I expect additional revision, not because that is justified but because they simply suck that bad. I can't think of a breach handled quite this poorly, and that includes my biases from the OPM breach. I'd say the OPM breach was far worse but handled marginally better. Equifax had to work hard to take the crown, but they have managed to edge them out by a narrow margin.

tejaswiy 9y ago

> Second, Smith blamed a scanning system used to spot this sort of oversight that did not identify the customer-dispute portal as vulnerable. Smith said forensic investigators are still looking into why the scanner failed. Do these scanners ever work? Without naming names, the only reason we used this at a previous company was to kind of handwave around hey we have this tool doing regular security checks. > The first excuse Smith gave was "human error." He says there was a particular (unnamed) individual who knew that the portal needed to be patched but failed to notify the appropriate IT team. How is this one individual responsible for tracking all IT security vulnerabilities in all the technologies that equifax uses across its stack? How do they not have an admin team whose job it is to do these things?

mikekij 9y ago
Apparently the right user:security engineer ratio is 143,000,000:1. /s
tehwebguy 9y ago
Maybe he was talking about himself?
jeremyjh 9y ago
We do not know anything about the number of people who could have known about this issue and escalated it, but it does not surprise me at all that there is one primary person who is responsible for it. In fact, if there were not exactly one person who is responsible for the security of the application that would be a different sort of problem as diffuse responsibility can be neglected quite easily in large organizations. Still, at my own company - A LOT of people were focused on the timely remediation of this issue and there was no chance of it being left to one person, even for development environments of internal facing applications. The scanners do work, we used them at my company and they found this specific vulnerability. It surprised some development teams because there are libraries that use struts without really advertising it, so even applications that don't "use struts" had a transitive dependency on it, it was running in their container and showed up on the lists that were circulating.
wglb 9y ago

cody8295 9y ago

To think it could have all been avoided with simple encryption. Seriously how does a company like Equifax not encrypt this kinda stuff?

paulddraper 9y ago
It was encrypted. (HTTPS/TLS)
- wglb 9y ago
  That protects the data between the browser and the web server (or SSL offloader) from eavesdropping and tampering. It does not protect against the case where the attacker gains authorization, over HTTPS, to administrative access at the web server.
  - paulddraper 9y ago
    Right. So what hypothetical encryption was cody8295 referring to?
    - FuckOffNeemo 9y ago
      As stated above, the data was not encrypted at rest.
      - paulddraper 9y ago
        Encrypting at rest wouldn't have solved anything. They didn't steal the disk. They gained access to the web server, which would have access to the unencrypted data.
- FuckOffNeemo 9y ago
  The article specifically states that encryption in transit is provided but not at rest, which is what the OP is referencing too.
9y ago

cpitman 9y ago

#2 isn't surprising. A ton of shops only patch on a quarterly basis, and even then they only consistently patch the OS. Only when there is a vendor (like my employer, Red Hat) sending out notices that there is a critical vulnerability do people move any faster. Which is twice as bad for application dependencies. These don't even have quarterly patch cycles. Dependencies may be updated when a new release is deployed, which may be a couple times a year or never. One of my favorite questions for clients is "who is responsible for patching applications with no development team anymore?" This is an underappreciated benefit of CICD. An automated process lets you get a central group to take ownership of third party libraries and their security, and then let them trigger all applications to rebuild and release. Especially with containers, this is essential.

gsylvie 9y ago
THIS: > "who is responsible for patching applications with no development team anymore?" Struts 2.2.3 (oldest vulnerable version) was released on May 7, 2011. Personally I find it easy to imagine an app depending on that, going into prod, and just chugging away, forgotten for 6 years on some obscure corner of an enterprise's public (or internal) web assets.
- greedo 9y ago
  Getting business lines to sign off and invest the resources to update applications can be a huge challenge. Shiny new applications get all the attention, but maintaining and updating legacy applications is not prioritized.
- speedplane 9y ago
  I personally have no idea how large companies manage this. Every time I perform a full Docker update on all of the libraries and applications my app is running on, something breaks. Sure, tests catch it, but it takes a developer and time to figure out how to fix it. No idea how huge non-developer organizations handle this.

ams6110 9y ago

I don't see any of these things as "fresh horrors." They were either already known, or entirely predictible given what is already known. I'm sure the record count will be revised upwards several more times.

konceptz 9y ago

“cybersecurity experts from the law firm King & Spalding“ Did it strike anyone as disheartening that legal was the first call?

wglb 9y ago
Actually it is not. There are good reasons to call outside counsel first.
twunde 9y ago
Not really. A specialist law firm can give you a risk assessment, give you an overview of all relevant regulations that you need to follow, recommend security firms, connect you to any federal agencies that can help or need to know and all discussed in layman's terms (or at least exec-speak). If your CSO is good, then it's a second opinion. If he or she is not, then it's the only way to get the help you need.

yalogin 9y ago

Why isn’t Equifax offering free credit monitoring and identity protection for all their customers? I had to go and put a fraud alert on mine but I feel this should have happened the moment they knew about the hack. Even now there is no talk of that. Of course I wouldn’t have confidence in what they do but am hoping the other two are not totally incompetent.

MertsA 9y ago
You aren't their customer.

LyalinDotCom 9y ago

No one important will go to jail. If someone important is prosecuted and convicted they will be let off very very lightly. No companies will learn any lessons from this, as companies are already doing everything they can within skillset limits of their employees and budgets, or they are not and don't give a shit. No one really cares about the consumer in all of this, they will do the minimum not to look bad or face further legal issues but in the end people will invest lots of hours dealing with any problems that might arise. Who will pay for those hours? lol im just kidding, once again they dont give a shit really, good luck to us. Nothing will change in the industry for companies like Equifax, this is a very small nitch market and we cant seriously hope for any government mandated reforms in today's political climate. This will blow over soon and no one will remember it after the next breach or mass shooting or a war that Trump will start. Sorry to say all this but sometimes people need to say the truth.

existencebox 9y ago
I don't disagree with any particular assertion you make. Let me ask the obvious question. What do we do then? (If this doesn't spark discussion I'll gladly contribute my own ideas, which involve finding methods of partially-civil disobedience to even the scales of inconvenience a little bit. I honestly believe a large issue in the current system is that there's no real pressure being held against any of the people with real power. Why would someone who can bring in tens of millions _FROM FAILING AS BADLY AS ONE CAN FAIL_ in their job, give a damn about anything but optimizing that number and minimizing time/effort spent to achieve that? With senate and house incumbency rates hovering around 90%, why would a politician ever deviate from the status quo? The wellbeing of the people is so far separated from any of their driving incentives, we must find ways to realign that.)
- Joeri 9y ago
  What caused things to turn around in the 30’s was people getting so fed up they voted in a different crew who actually fixed things. Voting (and voter outreach) is still the best path to reform.
  - 9y ago

deleted 9y ago

[deleted]

jaf7 9y ago

> (In a moment of true dystopian chaos, the official Equifax Twitter account repeatedly tweeted a phishing link, mistaking it for the breach-response page.) These institutions are cancerous devolutions in their current form. They are no better than predatory lenders. What has motivated their current structure seems cynical at the core.

thisisit 9y ago

Most of the time person giving the orders is wholly inadequate to understand what they are doing. At one of my previous jobs, the system I was working on was hacked years ago. You would think, this will give people some semblance of what not to do. No one seemed to have heard of the concept known as PII. Salaries of the entire company was sent out daily attached as a text file in an email to a very large group of people. The claim to do nothing was? It contains employee ids so it's kind of masked. And "many" people don't understand the data anyways. Trying to explain the "powers that be" why this was against industry standard fell on deaf years. I am sure if things do go wrong they will blame that "one guy" who was unable to "convince" them of the right thing to do.

paulajohnson 9y ago

#3 bugs me. It gets asked every time there is a breach, but its irrelevant. Encryption at rest is only useful if the only way to access the data is to type in the key. But for Equifax there are going to be hundreds or thousands of accesses per second. If you encrypt the data then you have added no protection at all because you still have a huge pipeline out through an always-on decryption mechanism. Any attacker is going to access the data through that mechanism and ignore the encryption completely.

danpalmer 9y ago
It depends on how they get access, and it can prevent some types of access. For example, they can’t steal backups, and they may need access through a web app server that has the database key rather than being able to go directly to the database. It’s not a catch-all solution but it is correct security practice and they should have been doing it.
smsm42 9y ago
Depends on vulnerability. If you have RCE (like I understand Equifax had) then encryption at rest is useless, the code can access the data, obviously, so the RCE code can too (maybe with a little more work, but ultimately it is not a barrier anymore).