14 ms·

An Example of Forensic Science at Its Worst: US v. Brig. Gen. Jeffrey Sinclair

114 points by jzdziarski 12y ago

na85 12y ago
The author states they were shocked at how Sinclair seemed to get off easy, with just a slap on the wrist. I'm not sure why that's shocking. It's been painfully obvious for more than a decade that there exists a certain class of people in the US who are above the law, e.g. James Clapper.
- pyre 12y ago
  Well, it looked like the powers-that-be wanted to make an example out of him. So I could see it be surprising that he got a slap on the wrist after having those "guns" pointed at him.
- maxerickson 12y ago
  I read that line to mean they were shocked that he received as much punishment as he did. The phrase just before that talks about damaging the prosecution's case.
arjie 12y ago
I wonder if anyone has ever been wrongfully convicted because of poor forensic software. Those programmers are incredibly unethical people. I'm familiar with the food-on-the-table argument, but wow, possibly getting innocent people locked up just because you don't know what you're doing is something. It can't feel good to receive the double whammy of knowing you're incompetent and that you're incompetent enough to ruin lives. Would I do it? I don't know. Through good fortune, I've never been, as a programmer, in the situation where I worried about losing my job and being unable to find one. So maybe those of us who feel we would never do something like this (work on critical health software while ignorant, or on forensic software while clueless) just don't know that it is easy when your livelihood is threatened. Still really unethical, though.
- ObviousScience 12y ago
  I used to work in pressure equipment control software. Quite literally, the software malfunctioning (ignoring physical safety mechanisms which should prevent it) could literally cause an explosion that would destroy the lab, send the equipment rocketing in to the air if not properly bolted in to concrete, and would kill anyone that happened to be in the area when it happened. The pressure differential of the shockwave was enough to seriously injure you with sudden compression and was strong enough to tare limbs off. Even with physical failsafes, that only reduced the damage to a sudden jet of highly pressurized, unbreathable air, which was dangerous in its own ways. I would have quit that job (even if it meant going back to painting houses or tutoring high school kids or serving coffee) if it had for a moment required that I not take every opportunity to make sure the software was as safe and predictable as it needed to be. This included consulting with staff from Boeing on good engineering practices for mission critical safety gear, using languages outside of the most popular ones (ie, not using popular ones and instead using languages with good tooling for safety and reliability), etc. I don't think it's acceptable we hold people who make forensic software to a different standard than those who make airplanes or industrial safety systems - they're ruining lives just the same. > I wonder if anyone has ever been wrongfully convicted because of poor forensic software. Not only this, they've been convicted by digital expert witnesses just being lazy in their analysis and only looking at the surface level. One of my favorite digital forensic instructors from my time in college had almost helped send a guy to prison for child pornography by half-assing his analysis before he decided that he was going to do the very best job he could even if he hated the guys guts, took a second look at the evidence before submitting his report, and discovered that it was likely a coworker who had looked up the child porn on the accused guy's work computer.
- AlyssaRowan 12y ago
  My sources say: yes. There exists software which is designed to confuse them and convincingly plant evidence; and while that software is imperfect, they are powerless against that.
  - jonstewart 12y ago
    I've been a digital forensics software developer for a decade. There's plenty of buggy software, yes, but I've never heard of software that deliberately plants evidence. FLETC and other training classes instill in investigators the necessity of manually verifying and testing their results for their reports and/or court testimony. Most cases involve child pornography. Most of the time its known child pornography (i.e., there's a known victim, there are likely other folks who have been convicted of possessing the images). Most of the time the suspect takes a plea bargain. It's the rare case that involves figuring out who-did-what-when events on a computer based on trace artifacts. So, the good news is that they are not the norm; the bad news is that only the best examiners (on either prosecution or defense) can typically handle them.
patio11 12y ago
Ironically, the amount of pressure from the top involved in this (and related) cases caused several prosecutions of sexual misconduct in the military to be dismissed. You read that right. There's a doctrine called "unlawful command influence" where, if the posture of the Commander on Chief on down suggests that there is a "right verdict" independent of the facts, then the military courts should dismiss cases rather than having military officers be, essentially, ordered to convict. A certain US president who is a Constitutional law scholar has run into this at least three times and professes to be really surprised every time it happens. see generally: http://www.nytimes.com/2013/07/14/us/obama-remark-is-complicating-military-trials.html?pagewanted=all&_r=0 http://www.nytimes.com/2013/07/14/us/obama-remark-is-complic...
- lotsofmangos 12y ago
  We have something similar going on in the UK with Cameron making commentary and publicly taking sides on court proceedings. http://www.bbc.co.uk/news/uk-england-london-25350419 http://www.bbc.co.uk/news/uk-england-london-25350419 http://www.bbc.co.uk/news/uk-politics-28014035 http://www.bbc.co.uk/news/uk-politics-28014035
  - hga 12y ago
    While that's not good, it's very different thing. Those two examples were of normal, civilian courts, proceeding of an independent judiciary; a President or Prime Minister has the "power of the bully pulpit", but no formal power. Our minimalist Constitution goes so far as to say Federal judge compensation "shall not be diminished during their Continuance in Office", i.e. the other branches of government can't dock their pay. In military courts, the judges and jury are in the chain of command, so as patio11 put it, "the military courts should dismiss cases rather than having military officers be, essentially, ordered to convict." Hence the formal name "unlawful command influence".
- magicalist 12y ago
  To be fair, the Uniform Code of Military Justice is wholly separate from the Constitution and the modern form (including the unlawful command influence clause[1]) has only really existed since 1950. He probably should have been more wary after commenting on the Chelsea Manning case, but it's worth noting that there have been many judges who haven't dismissed cases after his remarks, so at the very least the propriety of his statement (which was fairly tame and limited to calling for discharge once convicted) is debatable. [1] http://www.au.af.mil/au/awc/awcgate/ucmj.htm#837.%20ART.%2037.%20UNLAWFULLY%20INFLUENCING%20ACTION%20OF%20COURT http://www.au.af.mil/au/awc/awcgate/ucmj.htm#837.%20ART.%203...
Radim 12y ago
A sad read on many levels. "I investigated a little. As it turns out, this particular manufacturer first copies the iOS file system onto the Windows partition that their software is running on, and then pulls the timestamp information off of the copy of the data." "Irresponsible, to say the least. But this is the quality of the forensics software assisting our government and military. Poorly written, over-priced assumptionware." Also, the OP's professional frustration is endearing -- typical woes of an expert who "knows and cares", observing snakeoil barons become rich on bullshit :-)
- wahsd 12y ago
  I think you should understand that America's economy has a long an illustrious history of "snakeoil barons becoming rich on bullshit". It's kind of the very basis of our whole system.
cmb99 12y ago
An Example of Pathetically Self-Aggrandizing Twaddle, Nearly at Its Worst.
- kghose 12y ago
  You seem to have called a spade a spade and some folks don't seem to like that. Yes, that page (the article and the byline) does seem a little anxious to establish the author's credentials. But that should be taken separately from the content. I'm surprised that the article got so many up-votes considering it has so little technical detail.
DanielBMarkham 12y ago
"...This, and many other types of artifacts are often either completely overlooked by numerous commercially sold, expensive-as-hell tools, or in the case of at least one tool – seemingly made up data. All of these came into play in this case and would later play a role in its outcome..." So -- prosecutors and investigators don't understand the tools they use to search phones, judges give them wide leeway in reporting such "evidence", AND some commercial tools make up data to return? Does this not deeply concern anybody in any of the branches of the government? I hope I'm not being alarmist, but that statement, if true, seems to me to be incredibly damming of the entire criminal justice system. It's like saying the prosecutors determine who's guilty simply by selecting them for trial. The rest of the work is simply acquiring and configuring the appropriate "tools" (Note that it doesn't say they do this on purpose. I guess that's something.)
Symmetry 12y ago
Sadly, forensic science can actually be a lot worse than the example here. http://www.huffingtonpost.com/2011/09/01/michael-west-fabricating-bite-marks_n_944228.html http://www.huffingtonpost.com/2011/09/01/michael-west-fabric...
kghose 12y ago
So, I read through the article and could not find what was concretely wrong, technically, with the data. Does he state that any where? Thanks.
- zenocon 12y ago
  Me neither. It seemed like a fascinating read, but I wish he would have trimmed the amount of soapbox talk about his own credentials and bad software engineers, and instead provided the facts about the case (at least what changed the case and could have been revealed).
- justincormack 12y ago
  The details are of course confidential. He says: "There are reasons I’m not going to dig into the details of the case. Certain people that were involved could easily be pinpointed by revealing technical details that could be pieced together with news reports, and help build a story in your mind that would probably be inaccurate. The details aren’t so important as the errors that were made. All you need to know from a technical perspective is right here: some of the types of information that these commercial tools were (and likey still are) misreporting is significant. Evidence and timestamps of a device erasure event. Evidence of a backup restore event. Application usage dates. Application deletion events and timestamps. File access times. This, and many other types of artifacts are often either completely overlooked by numerous commercially sold, expensive-as-hell tools, or in the case of at least one tool – seemingly made up data. All of these came into play in this case and would later play a role in its outcome."
  - zo1 12y ago
    "The details are of course confidential. He says:" I kept having to skip paragraphs to get to the juicy bits. Eventually, I gave up and closed the tab. I don't much like this guy's writing style, as it constantly goes off on tangents that don't really add anything to the story, much less the baity title.
- jonstewart 12y ago
  Yeah, he doesn't really discuss it. It's just the usual mix of crypto-anarcho/libertarian/fundamentalist paranoia I've come to expect. He may indeed not be able to reveal certain details of the case. However, he'd certainly be capable of reproducing whatever bugs he found in the unnamed software (likely EnCase or FTK) and discussing them. Instead of science, though, he merely provides lip service thereto, along with political commentary, irrelevant asides, and run-on sentences.
zibit 12y ago
The OP should start hacking: http://www.sleuthkit.org/ http://www.sleuthkit.org/
sp332 12y ago
Jeffery Sinclair is the name of the first commander of Babylon 5. I had to check that it was a real name, but it seems that it is also the name of a real brigadier general.
RexRollman 12y ago
This article makes me wonder: does forensic software undergo any kind of 3rd party testing for accuracy? Or are we literally just taking someone's word that it is producing accurate results?
- jonstewart 12y ago
  There's some, but it's far too little to be comprehensive. Some law enforcement labs have entire QA departments dedicated to validating internal procedures, including how particular tools should be used.
rayiner 12y ago
What this really highlights is that lay people trust computer forensics people to be experts, and often that trust is misplaced. Unfortunately, this isn't the only example of misplaced trust. In several places, Zdziarski laments that computer forensics doesn't live up to the standards of proper "science." Yet, his trust in the rest of forensic "science" is also misplaced. Pretty much everything besides DNA forensics, maybe fingerprints, is junk: http://lst.law.asu.edu/FS09/pdfs/Koehler4_3.pdf http://lst.law.asu.edu/FS09/pdfs/Koehler4_3.pdf (page 4). Indeed, the National Science Foundation has been looking into the field and has been shocked by how unfounded it is: http://www.nsf.gov/pubs/2013/nsf13120/nsf13120.jsp http://www.nsf.gov/pubs/2013/nsf13120/nsf13120.jsp ("While the report acknowledges that 'the forensic science disciplines have produced valuable evidence that has contributed to the successful prosecution and conviction of criminals as well as to the exoneration of innocent people,' it cites a need for systematic research to validate the various disciplines’ underlying assumptions and methodologies, adding that the 'forensic science ... communities will be improved by opportunities to collaborate with the broader science and engineering communities.') All that is just a polite way of saying: "holy shit your techniques lack real scientific foundations; you guys need to hire real scientists."
- Zigurd 12y ago
  On top of the junky aspects of forensic "science" are many bad practices. 1600+ cases in Massachusetts were called into question due to a crime lab employee who told police and prosecutors what they wanted to hear. She was a hero until she wasn't. She was convicted of a crime and got a short prison term. The investigation found she "acted alone." Is she an outlier?
bitsteak 12y ago
John's blog post sure has a hell of a lot of conjecture and opinions in it for someone so hellbent on discrediting closed source forensic tools for doing the same. John would have a lot more credibility if he would shut his mouth and stuck to reporting unbiased technical information like he asks for from everyone else.
munin 12y ago
the courts and their interaction with science has always been tenuous, the story of arson investigators, fire science and criminal justice is similarly terrifying http://www.newyorker.com/magazine/2009/09/07/trial-by-fire http://www.newyorker.com/magazine/2009/09/07/trial-by-fire
contingencies 12y ago
TLDR; "computer says no" all over again.

114 points by jzdziarski 12y ago

na85 12y ago

The author states they were shocked at how Sinclair seemed to get off easy, with just a slap on the wrist. I'm not sure why that's shocking. It's been painfully obvious for more than a decade that there exists a certain class of people in the US who are above the law, e.g. James Clapper.

pyre 12y ago
Well, it looked like the powers-that-be wanted to make an example out of him. So I could see it be surprising that he got a slap on the wrist after having those "guns" pointed at him.
maxerickson 12y ago
I read that line to mean they were shocked that he received as much punishment as he did. The phrase just before that talks about damaging the prosecution's case.

arjie 12y ago

I wonder if anyone has ever been wrongfully convicted because of poor forensic software. Those programmers are incredibly unethical people. I'm familiar with the food-on-the-table argument, but wow, possibly getting innocent people locked up just because you don't know what you're doing is something. It can't feel good to receive the double whammy of knowing you're incompetent and that you're incompetent enough to ruin lives. Would I do it? I don't know. Through good fortune, I've never been, as a programmer, in the situation where I worried about losing my job and being unable to find one. So maybe those of us who feel we would never do something like this (work on critical health software while ignorant, or on forensic software while clueless) just don't know that it is easy when your livelihood is threatened. Still really unethical, though.

ObviousScience 12y ago
I used to work in pressure equipment control software. Quite literally, the software malfunctioning (ignoring physical safety mechanisms which should prevent it) could literally cause an explosion that would destroy the lab, send the equipment rocketing in to the air if not properly bolted in to concrete, and would kill anyone that happened to be in the area when it happened. The pressure differential of the shockwave was enough to seriously injure you with sudden compression and was strong enough to tare limbs off. Even with physical failsafes, that only reduced the damage to a sudden jet of highly pressurized, unbreathable air, which was dangerous in its own ways. I would have quit that job (even if it meant going back to painting houses or tutoring high school kids or serving coffee) if it had for a moment required that I not take every opportunity to make sure the software was as safe and predictable as it needed to be. This included consulting with staff from Boeing on good engineering practices for mission critical safety gear, using languages outside of the most popular ones (ie, not using popular ones and instead using languages with good tooling for safety and reliability), etc. I don't think it's acceptable we hold people who make forensic software to a different standard than those who make airplanes or industrial safety systems - they're ruining lives just the same. > I wonder if anyone has ever been wrongfully convicted because of poor forensic software. Not only this, they've been convicted by digital expert witnesses just being lazy in their analysis and only looking at the surface level. One of my favorite digital forensic instructors from my time in college had almost helped send a guy to prison for child pornography by half-assing his analysis before he decided that he was going to do the very best job he could even if he hated the guys guts, took a second look at the evidence before submitting his report, and discovered that it was likely a coworker who had looked up the child porn on the accused guy's work computer.
AlyssaRowan 12y ago
My sources say: yes. There exists software which is designed to confuse them and convincingly plant evidence; and while that software is imperfect, they are powerless against that.
- jonstewart 12y ago
  I've been a digital forensics software developer for a decade. There's plenty of buggy software, yes, but I've never heard of software that deliberately plants evidence. FLETC and other training classes instill in investigators the necessity of manually verifying and testing their results for their reports and/or court testimony. Most cases involve child pornography. Most of the time its known child pornography (i.e., there's a known victim, there are likely other folks who have been convicted of possessing the images). Most of the time the suspect takes a plea bargain. It's the rare case that involves figuring out who-did-what-when events on a computer based on trace artifacts. So, the good news is that they are not the norm; the bad news is that only the best examiners (on either prosecution or defense) can typically handle them.

patio11 12y ago

Ironically, the amount of pressure from the top involved in this (and related) cases caused several prosecutions of sexual misconduct in the military to be dismissed. You read that right. There's a doctrine called "unlawful command influence" where, if the posture of the Commander on Chief on down suggests that there is a "right verdict" independent of the facts, then the military courts should dismiss cases rather than having military officers be, essentially, ordered to convict. A certain US president who is a Constitutional law scholar has run into this at least three times and professes to be really surprised every time it happens. see generally: http://www.nytimes.com/2013/07/14/us/obama-remark-is-complicating-military-trials.html?pagewanted=all&_r=0 http://www.nytimes.com/2013/07/14/us/obama-remark-is-complic...

lotsofmangos 12y ago
We have something similar going on in the UK with Cameron making commentary and publicly taking sides on court proceedings. http://www.bbc.co.uk/news/uk-england-london-25350419 http://www.bbc.co.uk/news/uk-england-london-25350419 http://www.bbc.co.uk/news/uk-politics-28014035 http://www.bbc.co.uk/news/uk-politics-28014035
- hga 12y ago
  While that's not good, it's very different thing. Those two examples were of normal, civilian courts, proceeding of an independent judiciary; a President or Prime Minister has the "power of the bully pulpit", but no formal power. Our minimalist Constitution goes so far as to say Federal judge compensation "shall not be diminished during their Continuance in Office", i.e. the other branches of government can't dock their pay. In military courts, the judges and jury are in the chain of command, so as patio11 put it, "the military courts should dismiss cases rather than having military officers be, essentially, ordered to convict." Hence the formal name "unlawful command influence".
magicalist 12y ago
To be fair, the Uniform Code of Military Justice is wholly separate from the Constitution and the modern form (including the unlawful command influence clause[1]) has only really existed since 1950. He probably should have been more wary after commenting on the Chelsea Manning case, but it's worth noting that there have been many judges who haven't dismissed cases after his remarks, so at the very least the propriety of his statement (which was fairly tame and limited to calling for discharge once convicted) is debatable. [1] http://www.au.af.mil/au/awc/awcgate/ucmj.htm#837.%20ART.%2037.%20UNLAWFULLY%20INFLUENCING%20ACTION%20OF%20COURT http://www.au.af.mil/au/awc/awcgate/ucmj.htm#837.%20ART.%203...

Radim 12y ago

A sad read on many levels. "I investigated a little. As it turns out, this particular manufacturer first copies the iOS file system onto the Windows partition that their software is running on, and then pulls the timestamp information off of the copy of the data." "Irresponsible, to say the least. But this is the quality of the forensics software assisting our government and military. Poorly written, over-priced assumptionware." Also, the OP's professional frustration is endearing -- typical woes of an expert who "knows and cares", observing snakeoil barons become rich on bullshit :-)

wahsd 12y ago
I think you should understand that America's economy has a long an illustrious history of "snakeoil barons becoming rich on bullshit". It's kind of the very basis of our whole system.

cmb99 12y ago

An Example of Pathetically Self-Aggrandizing Twaddle, Nearly at Its Worst.

kghose 12y ago
You seem to have called a spade a spade and some folks don't seem to like that. Yes, that page (the article and the byline) does seem a little anxious to establish the author's credentials. But that should be taken separately from the content. I'm surprised that the article got so many up-votes considering it has so little technical detail.

DanielBMarkham 12y ago

"...This, and many other types of artifacts are often either completely overlooked by numerous commercially sold, expensive-as-hell tools, or in the case of at least one tool – seemingly made up data. All of these came into play in this case and would later play a role in its outcome..." So -- prosecutors and investigators don't understand the tools they use to search phones, judges give them wide leeway in reporting such "evidence", AND some commercial tools make up data to return? Does this not deeply concern anybody in any of the branches of the government? I hope I'm not being alarmist, but that statement, if true, seems to me to be incredibly damming of the entire criminal justice system. It's like saying the prosecutors determine who's guilty simply by selecting them for trial. The rest of the work is simply acquiring and configuring the appropriate "tools" (Note that it doesn't say they do this on purpose. I guess that's something.)

Symmetry 12y ago

Sadly, forensic science can actually be a lot worse than the example here. http://www.huffingtonpost.com/2011/09/01/michael-west-fabricating-bite-marks_n_944228.html http://www.huffingtonpost.com/2011/09/01/michael-west-fabric...

kghose 12y ago

So, I read through the article and could not find what was concretely wrong, technically, with the data. Does he state that any where? Thanks.

zenocon 12y ago
Me neither. It seemed like a fascinating read, but I wish he would have trimmed the amount of soapbox talk about his own credentials and bad software engineers, and instead provided the facts about the case (at least what changed the case and could have been revealed).
justincormack 12y ago
The details are of course confidential. He says: "There are reasons I’m not going to dig into the details of the case. Certain people that were involved could easily be pinpointed by revealing technical details that could be pieced together with news reports, and help build a story in your mind that would probably be inaccurate. The details aren’t so important as the errors that were made. All you need to know from a technical perspective is right here: some of the types of information that these commercial tools were (and likey still are) misreporting is significant. Evidence and timestamps of a device erasure event. Evidence of a backup restore event. Application usage dates. Application deletion events and timestamps. File access times. This, and many other types of artifacts are often either completely overlooked by numerous commercially sold, expensive-as-hell tools, or in the case of at least one tool – seemingly made up data. All of these came into play in this case and would later play a role in its outcome."
- zo1 12y ago
  "The details are of course confidential. He says:" I kept having to skip paragraphs to get to the juicy bits. Eventually, I gave up and closed the tab. I don't much like this guy's writing style, as it constantly goes off on tangents that don't really add anything to the story, much less the baity title.
jonstewart 12y ago
Yeah, he doesn't really discuss it. It's just the usual mix of crypto-anarcho/libertarian/fundamentalist paranoia I've come to expect. He may indeed not be able to reveal certain details of the case. However, he'd certainly be capable of reproducing whatever bugs he found in the unnamed software (likely EnCase or FTK) and discussing them. Instead of science, though, he merely provides lip service thereto, along with political commentary, irrelevant asides, and run-on sentences.

zibit 12y ago

The OP should start hacking: http://www.sleuthkit.org/ http://www.sleuthkit.org/

sp332 12y ago

Jeffery Sinclair is the name of the first commander of Babylon 5. I had to check that it was a real name, but it seems that it is also the name of a real brigadier general.

RexRollman 12y ago

This article makes me wonder: does forensic software undergo any kind of 3rd party testing for accuracy? Or are we literally just taking someone's word that it is producing accurate results?

jonstewart 12y ago
There's some, but it's far too little to be comprehensive. Some law enforcement labs have entire QA departments dedicated to validating internal procedures, including how particular tools should be used.

rayiner 12y ago

What this really highlights is that lay people trust computer forensics people to be experts, and often that trust is misplaced. Unfortunately, this isn't the only example of misplaced trust. In several places, Zdziarski laments that computer forensics doesn't live up to the standards of proper "science." Yet, his trust in the rest of forensic "science" is also misplaced. Pretty much everything besides DNA forensics, maybe fingerprints, is junk: http://lst.law.asu.edu/FS09/pdfs/Koehler4_3.pdf http://lst.law.asu.edu/FS09/pdfs/Koehler4_3.pdf (page 4). Indeed, the National Science Foundation has been looking into the field and has been shocked by how unfounded it is: http://www.nsf.gov/pubs/2013/nsf13120/nsf13120.jsp http://www.nsf.gov/pubs/2013/nsf13120/nsf13120.jsp ("While the report acknowledges that 'the forensic science disciplines have produced valuable evidence that has contributed to the successful prosecution and conviction of criminals as well as to the exoneration of innocent people,' it cites a need for systematic research to validate the various disciplines’ underlying assumptions and methodologies, adding that the 'forensic science ... communities will be improved by opportunities to collaborate with the broader science and engineering communities.') All that is just a polite way of saying: "holy shit your techniques lack real scientific foundations; you guys need to hire real scientists."

Zigurd 12y ago
On top of the junky aspects of forensic "science" are many bad practices. 1600+ cases in Massachusetts were called into question due to a crime lab employee who told police and prosecutors what they wanted to hear. She was a hero until she wasn't. She was convicted of a crime and got a short prison term. The investigation found she "acted alone." Is she an outlier?

bitsteak 12y ago

John's blog post sure has a hell of a lot of conjecture and opinions in it for someone so hellbent on discrediting closed source forensic tools for doing the same. John would have a lot more credibility if he would shut his mouth and stuck to reporting unbiased technical information like he asks for from everyone else.

munin 12y ago

the courts and their interaction with science has always been tenuous, the story of arson investigators, fire science and criminal justice is similarly terrifying http://www.newyorker.com/magazine/2009/09/07/trial-by-fire http://www.newyorker.com/magazine/2009/09/07/trial-by-fire

contingencies 12y ago

TLDR; "computer says no" all over again.