3 ms·
It's an interesting tactic to set up a strawman and then beat it up. Where am I to start when the reply will mostly be "That's not true" or "I didn't say that"?
by sillysaurusx 4y ago
It's an interesting tactic to set up a strawman and then beat it up. Where am I to start when the reply will mostly be "That's not true" or "I didn't say that"? This is devolving into being boring for the audience, but you're (for some reason) attacking my reputation. You're also backpedaling; I thought you were going to tell stories? I'd like to hear them. Or did you check with someone, and they said "Um, actually, Shawn wasn't that bad"?
> I think these claims are pretty much risible. The report you're talking about is autogenerated from a bug tracker. The client sees your bugs. If you spend 50% of your time writing them, you're cheating the client.
You keep saying the report is autogenerated because Litany existed. This is a bit like claiming that scientific papers are autogenerated because LaTeX exists. Yes, it does a lot of heavy lifting. No, it doesn't change the fact that you start with a template and then rework it into the proper form. As any grad student will tell you, that work takes a lot of time. My experience at Matasano was similar. I bet Drew, Andy, Damien, Dmitri, and a few other folks would at least say that reporting occupied a significant chunk of time.
From where I'm sitting, the claim seems unremarkable. Look at how long this thread's security assessment is. Most of the words are boilerplate, but you can't simply ship boilerplate. And yeah, the reports went through multiple rounds of back-and-forth before they got shipped, during which every detail was combed over.
> What I really think happens is that you misinterpret things people tell you and blow them into weird directions. Somebody told you that "the PDF is what clients are paying for". Well, no shit. The PDF is the only deliverable from the project. It's where the bugs are written down.
It wasn't "someone." Damien was one of the most experienced pentesters at Matasano. He led several redteam projects, and taught me a lot of interesting tricks for sneaking your way into a network.
> I wasn't there for whatever you got told, but it sounds to me like the subtext of it was probably "so it doesn't matter much what you do on a project as long as the client ends up with a report that they can use to claim they've had an assessment done". That's a cynical thing to say, but definitely a thing that gets said. It's also, strictly speaking, true.
This is a strawman. What he was saying was that we need to do a good job on the report, in addition to the hacking. I don't know why you'd spin it into some cynical thing, but at least you're consistent.
> The actual figure of merit from a software pentest is the list of bugs, full stop. Yes, the list is delivered in PDF. Don't let that confuse you.
We can debate who's the one confused, but at this point it's pretty clear that our experiences were dramatically different. What possible benefit would it be to me to sit here and lie about it? Not only would you (and everyone else) call me out, but I'd have to keep making up more and more elaborate falsehoods.
Sounds exhausting. I'm just reporting what I saw, and what I did.
I don't see a productive way to continue this. Your position is that "nobody spends 50% of their time on reports." I'll concede that maybe it's closer to 40%. But it's certainly not 10%, or whatever small amount you're hinting at. And as you pointed out, my last month was filled with 100% documentation-writing.
Let's agree to disagree and move on.
- tptacek 4y agoI'm comfortable with what the thread says about how we came at this disagreement. As you've acknowledged upthread: your claim that documentation is 50% of the time on a pentest doesn't hold up. I believe it took 50% of your time, because you say it did, but like you said upthread: it's was an exceptional case. Maybe: 1. You spent more time on doc than was actually required 2. You worked a bunch of SARs, which are short doc-focused projects (and not pentests) 3. You were given a bunch of retests, which are short doc-focused projects; this almost fits, since it's some of the first work that's given to new team members after they're done shadowing on projects, except that it would be weird for there to be so much retest work that you could do them back-to-back for a long period of time (most clients don't purchase retests) 4. You worked ENPTs (external netpens), which are short projects and have a huge doc component of "fitting tool results into a report". But (a) your office didn't do a lot of those (Matasano in general didn't do a lot of netpen work) and (b) it wasn't low-seniority work; as I recall, the people who did netpens specialized in it. It could be some combination of all these factors. Meanwhile: Litany is nothing at all like LaTeX (though after I left LaTeX staged a coup and replaced the commercial PDF library it had been using). Litany is a bug tracker. You enter a finding title, a URL/location, and a description --- which Litany autogenerates for common bug classes --- and when you're done with the project you push a button and get a complete PDF. This would have been the primary way all reporting was done during your tenure. Interestingly, one of the few major disputes between Matasano and iSec Partners (the big two constituent firms in NCC US, iSec bigger by a factor of almost 2) was report generation. iSec's actually had a LaTeX template consultants would use to generate reports; they wrote them by hand. "I refuse to use Litany" (and lose control over my report formatting) was supposedly such a big thing that they had to rename Litany when they re-introduced it across the firm; it's now called Limb. Which is a tragedy, because Litany of Failure is one of the all-time great product names (credit, I believe, to Craig Brozefsky). Some of the people you've mentioned in this thread have reached out to me. I stand by literally everything I've said. It's OK to be wrong, and you are. The rest of it: d0136ada7858ad143675c2c3302db6343869190878191c8fc6a1547f0f23530e Just in case this ever comes up again, I'll at least commit to not having changed my story. :) Long story short: don't pay for software pentests that spend 50% of their time in doc. You're paying for the testing, not the report, even if the report is all you ultimately care about.