7 ms·

Writing that report must have been hell with - I bet - all the attention from Sr. management.

by Izmaki 4y ago

sillysaurusx 4y ago
The actual process of writing those reports is generally pretty laid back. It was hell mostly because it consumes all your time. Only about 50% of your time is spent hacking, if that. The rest is poured into that damn report. It goes through multiple rounds of review, and every small detail is caught and corrected. Which is how it should be. It’s just the opposite of fun. The report is literally the product. The implications of that didn’t sink in until a few months on the job.
- rfoo 4y ago
  Not to mention you HAVE to put some really lame recommendation here. "Application binary is not obfuscated", seriously, wtf?
  - hsnewman 4y ago
    Since it's not obfuscated the binary can be decompileed easily and any coding issues can be identified more easily.
    - saagarjha 4y ago
      This is a good thing.
    - deleted 4y ago
      [deleted]
  - Spooky23 4y ago
    The point of an assessment like this is to identify potential risks. That one is categorized as “informational”, and isn’t a defect. These types of recommendations are useful in context. You may have a moderate finding or series of findings that in context are more significant because of some informational finding.
- tptacek 4y ago
  Nobody spends 50% of their time writing reports. If anything, it's the exact opposite: people do their entire doc the last Friday of the test. It's enough of a problem that carefully scoped tests sometimes include doc days in the SOW. I used to get in arguments with the person we hired to run one of our offices for demanding that people stay late on Fridays to finish docs that customers weren't even going to look at until the following week. This "multiple rounds of review" thing is news to me too. To answer the original comment: the ordinary process for a project like this with a public report is, you deliver the whole project as if there wasn't a public report, with an internal report, and then the public report is based on the internal one (probably post-remediation). So actually delivering the project is usually no different than any other project.
  - Izmaki 4y ago
    This is my experience as well. The first paragraph that is.
  - sillysaurusx 4y ago
    Tom, I literally worked at your company. Matasano. Let’s just say the culture changed after you left. Is that so hard to believe? You probably didn’t spend 50% of your time writing reports, but Andy and the rest of my coworkers did. Working at Matasano was one of the biggest disappointments of my career. It was basically the exact opposite of everything you pitched it to be. But I should’ve known better than to be deceived by marketing. The actual hacking was sometimes fun, though. But again, that was maybe 50% of the time, if you’re lucky. Would you like me to break down an actual engagement timeline as a refresher? An engagement lasts a week, max. Usually it’s two days. That means day one is hacking, day two is reporting. The weeklong engagements are where the scales sometimes tip. But even by day three, you had better start writing up your findings, or else the report won’t be ready by Friday, and the client (or more specifically Wally, our manager) will not be pleased.
    - tptacek 4y ago
      Yes, it's hard to believe. You worked at Matasano/NCC for like a minute immediately after I left. What you are describing is nothing like the ordinary workload at NCC; the modal NCC engagement is 2 people, 2 weeks --- in fact, that's the modal engagement across the entire industry, not just at NCC. "A week, max". Sheesh. I'm not interested in defending NCC; I have no interest in NCC (other than the friends of mine who still work there, I guess) and haven't since 2014. But I'm an inveterate message board nerd and I'm going to damn sure correct false things said about pentesting on HN. In this instance, I'm going to go ahead and say I have better information about this than you do. It's "Thomas", by the way.
- bink 4y ago
  I'm going to stay away from the tire fire of accusations between you and the former NCC boss. But I do want to say that having run a pen testing firm for many, many years these claims aren't hard to believe. It does depend on what you consider to be time spent working on a report, however. 50% time actually hacking even sounds kinda high depending on how the organization is structured. We had some engineers that just absolutely sucked at writing documentation but were wizzes at hacking so they got away with passing some of the documentation off on others (dramatically increasing the other engineers workloads, but sometimes that's ok). We also had some folks that were just great at formatting and dealing with client requests for docs and they could take workload off of engineers and make everyone happy. An engagement ranged from 3 days to 6 weeks, with the average around 3 weeks. A typical internal pen test would be 2 weeks onsite followed by a week for writing the report and handling feedback (and travel and meetings and blah blah blah). It wasn't unusual for a client to request that all references to X be changed to Y or the word "should" be changed to "shall" or whatever other thing client management cared about. That time can add up quickly and make your job miserable. Every single pen testing firm goes through a phase where they realize they are "wasting" weeks of effort on writing their reports. And they all decide they're going to automate as much as they can to improve their profit margins and employee happiness. They all usually come up with a snarky name for the new tool that does this automation. Most of the time the generation tool saves pain when it comes to formatting tables, table of contents, glossaries, etc., but wastes time wrestling with client expectations that their report be unique or focus on unusual areas. Then the engineer has to figure out how to modify the auto generated report without breaking any future changes to finding severity or writeup. Some people would also consider time spent presenting the report to a board of directors as time spent "working on a report" and I wouldn't really disagree. It's not unusual to send an engineer and a manager to do a presentation at a remote site after the engagement finishes. That'll include 1-2 days of travel time and the time the group spends trying to "up sell" the client on further engagements. It's been some years since I've been in that field but back in my day a break down of a 3 week engagement could look something like this: First week is the fun part. You probably spend 30-40 billable hours on actual discovery, scanning, testing, hacking. And if you're smart you spend 3-5 hours on the early design of the report. The client might want a focus on the API or on WiFi or whatever and you arrange things so the report is organized properly early on and the team can input their findings as needed (or input them to the automated system which might not have a dedicated category for this focus area). Much of the second week could be considered part of writing the report. You've ideally root'd most of the infra in the first week and now you're using tools and manual inspection to find every single vuln you can in the networks. You're using automated tools to find common mis-configurations and missing patches. Those tools are (hopefully) connected to your automated report generating system somehow so you don't need to format them manually. But there's always new stuff on each engagement that'll need some custom write-ups and severity determination will need to be adjusted depending on the client's security posture and risk profile. The third week is 1-2 days getting the report ready, though that can extend further for unique engagements. Then there's submitting it to a tech editor, submitting it to management, submitting it to the client and dealing with corrections from all of them. It's extremely rare for a client not to have comments and changes that need to be addressed and that's another 2-8 hours. Pen testing is difficult -- the travel, the presentations, the sales, the report writing. I'm not saying what this commenter is claiming is true, but the job is not how most people imagine it to be. I'll also say that that's not a criticism of NCC at all. All firms I've worked for, managed, or talked with are like this.
- wglb 4y ago
  I worked at Matasano for two years and did many tens of assessments. None even remotely approached 50% of the time spent on the assessment. I left there before the Litany of Failure was completed. I did security assessments on my own and wrote something similar to Litany of Failure (I think it is better) and the thing is that you should be documenting as you go along so it is not such a painful crush at the end.

Writing that report must have been hell with - I bet - all the attention from Sr. management.

by Izmaki 4y ago

Writing that report must have been hell with - I bet - all the attention from Sr. management.

The actual process of writing those reports is generally pretty laid back. It was hell mostly because it consumes all your time. Only about 50% of your time is spent hacking, if that. The rest is poured into that damn report. It goes through multiple rounds of review, and every small detail is caught and corrected. Which is how it should be. It’s just the opposite of fun. The report is literally the product. The implications of that didn’t sink in until a few months on the job.

rfoo 4y ago
Not to mention you HAVE to put some really lame recommendation here. "Application binary is not obfuscated", seriously, wtf?
- hsnewman 4y ago
  Since it's not obfuscated the binary can be decompileed easily and any coding issues can be identified more easily.
  - saagarjha 4y ago
    This is a good thing.
  - deleted 4y ago
    [deleted]
- Spooky23 4y ago
  The point of an assessment like this is to identify potential risks. That one is categorized as “informational”, and isn’t a defect. These types of recommendations are useful in context. You may have a moderate finding or series of findings that in context are more significant because of some informational finding.
tptacek 4y ago
Nobody spends 50% of their time writing reports. If anything, it's the exact opposite: people do their entire doc the last Friday of the test. It's enough of a problem that carefully scoped tests sometimes include doc days in the SOW. I used to get in arguments with the person we hired to run one of our offices for demanding that people stay late on Fridays to finish docs that customers weren't even going to look at until the following week. This "multiple rounds of review" thing is news to me too. To answer the original comment: the ordinary process for a project like this with a public report is, you deliver the whole project as if there wasn't a public report, with an internal report, and then the public report is based on the internal one (probably post-remediation). So actually delivering the project is usually no different than any other project.
- Izmaki 4y ago
  This is my experience as well. The first paragraph that is.
- sillysaurusx 4y ago
  Tom, I literally worked at your company. Matasano. Let’s just say the culture changed after you left. Is that so hard to believe? You probably didn’t spend 50% of your time writing reports, but Andy and the rest of my coworkers did. Working at Matasano was one of the biggest disappointments of my career. It was basically the exact opposite of everything you pitched it to be. But I should’ve known better than to be deceived by marketing. The actual hacking was sometimes fun, though. But again, that was maybe 50% of the time, if you’re lucky. Would you like me to break down an actual engagement timeline as a refresher? An engagement lasts a week, max. Usually it’s two days. That means day one is hacking, day two is reporting. The weeklong engagements are where the scales sometimes tip. But even by day three, you had better start writing up your findings, or else the report won’t be ready by Friday, and the client (or more specifically Wally, our manager) will not be pleased.
  - tptacek 4y ago
    Yes, it's hard to believe. You worked at Matasano/NCC for like a minute immediately after I left. What you are describing is nothing like the ordinary workload at NCC; the modal NCC engagement is 2 people, 2 weeks --- in fact, that's the modal engagement across the entire industry, not just at NCC. "A week, max". Sheesh. I'm not interested in defending NCC; I have no interest in NCC (other than the friends of mine who still work there, I guess) and haven't since 2014. But I'm an inveterate message board nerd and I'm going to damn sure correct false things said about pentesting on HN. In this instance, I'm going to go ahead and say I have better information about this than you do. It's "Thomas", by the way.
bink 4y ago
I'm going to stay away from the tire fire of accusations between you and the former NCC boss. But I do want to say that having run a pen testing firm for many, many years these claims aren't hard to believe. It does depend on what you consider to be time spent working on a report, however. 50% time actually hacking even sounds kinda high depending on how the organization is structured. We had some engineers that just absolutely sucked at writing documentation but were wizzes at hacking so they got away with passing some of the documentation off on others (dramatically increasing the other engineers workloads, but sometimes that's ok). We also had some folks that were just great at formatting and dealing with client requests for docs and they could take workload off of engineers and make everyone happy. An engagement ranged from 3 days to 6 weeks, with the average around 3 weeks. A typical internal pen test would be 2 weeks onsite followed by a week for writing the report and handling feedback (and travel and meetings and blah blah blah). It wasn't unusual for a client to request that all references to X be changed to Y or the word "should" be changed to "shall" or whatever other thing client management cared about. That time can add up quickly and make your job miserable. Every single pen testing firm goes through a phase where they realize they are "wasting" weeks of effort on writing their reports. And they all decide they're going to automate as much as they can to improve their profit margins and employee happiness. They all usually come up with a snarky name for the new tool that does this automation. Most of the time the generation tool saves pain when it comes to formatting tables, table of contents, glossaries, etc., but wastes time wrestling with client expectations that their report be unique or focus on unusual areas. Then the engineer has to figure out how to modify the auto generated report without breaking any future changes to finding severity or writeup. Some people would also consider time spent presenting the report to a board of directors as time spent "working on a report" and I wouldn't really disagree. It's not unusual to send an engineer and a manager to do a presentation at a remote site after the engagement finishes. That'll include 1-2 days of travel time and the time the group spends trying to "up sell" the client on further engagements. It's been some years since I've been in that field but back in my day a break down of a 3 week engagement could look something like this: First week is the fun part. You probably spend 30-40 billable hours on actual discovery, scanning, testing, hacking. And if you're smart you spend 3-5 hours on the early design of the report. The client might want a focus on the API or on WiFi or whatever and you arrange things so the report is organized properly early on and the team can input their findings as needed (or input them to the automated system which might not have a dedicated category for this focus area). Much of the second week could be considered part of writing the report. You've ideally root'd most of the infra in the first week and now you're using tools and manual inspection to find every single vuln you can in the networks. You're using automated tools to find common mis-configurations and missing patches. Those tools are (hopefully) connected to your automated report generating system somehow so you don't need to format them manually. But there's always new stuff on each engagement that'll need some custom write-ups and severity determination will need to be adjusted depending on the client's security posture and risk profile. The third week is 1-2 days getting the report ready, though that can extend further for unique engagements. Then there's submitting it to a tech editor, submitting it to management, submitting it to the client and dealing with corrections from all of them. It's extremely rare for a client not to have comments and changes that need to be addressed and that's another 2-8 hours. Pen testing is difficult -- the travel, the presentations, the sales, the report writing. I'm not saying what this commenter is claiming is true, but the job is not how most people imagine it to be. I'll also say that that's not a criticism of NCC at all. All firms I've worked for, managed, or talked with are like this.
wglb 4y ago
I worked at Matasano for two years and did many tens of assessments. None even remotely approached 50% of the time spent on the assessment. I left there before the Litany of Failure was completed. I did security assessments on my own and wrote something similar to Litany of Failure (I think it is better) and the thing is that you should be documenting as you go along so it is not such a painful crush at the end.