11 ms·
Mark Felt-Tipped: Uncovering top-secret information by counting pixels
- mc32 9y agoWhy a numeral exactly and not an u defined quantity like "some". Is there additional context to eliminate the possibility of imprecise quantifiers?
- netsharc 9y agoHere's what "four" and "some" look like: https://i.imgur.com/UNSpv6v.png https://i.imgur.com/UNSpv6v.png . Different letter widths and kerning make this identification possible. I wonder how brute-forceable it is. Given a blank with length of half a line, can I just throw 20-30 combination of the alphabet and punctuation, and figure out which combinations match, and the anagram for it? Actually, brute-forcing using words from a dictionary (English words plus names of the involved people (Americans, and obviously Russians)) would make it go even faster.
- OkGoDoIt 9y agoIn theory kerning differences would make letter ordering significant, so this becomes a permutation problem not a combination problem, and therefore much harder to brute force.
- incidentnormal 9y agoDid you mean this becomes a combination problem not a permutation problem?
- OkGoDoIt 9y agoIt's been a while since my last discreet math class so take this with a grain of salt, but in a combination you only care about selecting the correct members of a set, so order does not mater. In a permutation problem you care about selecting the correct members as well as their order, so it is a significantly larger solution space which expands factorially.
- alkonaut 9y agoIs kerning used also across spaces? I.e is the space between two words dependent on which letter the first word ends with, and which letter the second word starts with? If not, it still seems tractable using dictionary words.
- blattimwind 9y ago> Is kerning used also across spaces? Yes.
- alkonaut 9y agoOk. Makes it much more difficult. Still, assuming grammatically valid sequences of dictionary words, kerning would be known both within and between words. These texts however probably contain lots of abbreviations, footnote symbols, numbers, brackets etc, that make it likely to be a lot harder than just regular prose from dictionary words.
- blattimwind 9y agoBut valid sentence fragments are not random draws from an alphabet, they're drawn from a dictionary, which makes it much easier.
- mc32 9y agoThanks for pointing tjat out. So if they are to leave the original layout --not padding the spacing or subtracting spacing asf. then they should at a minimum use monospaced typefaces.
- lemoncucumber 9y agoIt stands to reason that a precise quantifier would be much more likely to be considered sensitive enough to redact than an imprecise one.
- methodin 9y agoWhy would they redact an undefined quantity?
- ballenf 9y agoExactly, these aren't randomly selected words for redaction. The fact that someone thought the information was worth hiding means gives a lot of information regarding its possible contents.
- mlady 9y agoUsing this methodology more generally, it would be interesting to use NLP to identify the part of speech that is redacted to narrow the word search space. In this example, the POS would be an adjective, and since the subject noun is plural, it would be more likely the adjective is a number
- EGreg 9y agoWhat?
- probably_wrong 9y agoNLP: Natural language processing. Part of speech (POS): the elements that, when combined, form a sentence (adjectives, verb, substantive, etc). What the OP suggests is to run existing tools to identify what POS are missing (as in "in this blank space you could only have a substantive and two to three adjectives"), and use that to reduce the search space.
- abpavel 9y agoOr better yet, rank the resulting sentences for each of the possible fits using existing speech API, setting a cutoff to filter out nonsensical results. This might even yield surprises.
- FLUX-YOU 9y agoYou can't really predict if there's going to be an aside in the redacted sentence though.
- FLUX-YOU 9y agoYou might get some out of predicting the first and last words and word type of a redaction (based on the words next to the redaction), but it's only cutting down on brute force space. That makes short redactions more dangerous for declassification than entire paragraphs as a general rule because you have no context to start from, but that's probably common sense to people doing redactions.
- pja 9y agoNote to self: print all documents in mono-spaced Courier New if I intend to ever redact any contents.
- Smushman 9y agoMixed fonts...
- kabdib 9y agoClip out letters from magazines and paste them onto craft paper. Plenty of precedence. "iF YOu wAnT tO SEe yOUR deMOCRacY aGAIn lEAvE 5 miLLioN dOLlArs uNDer THE bRidGe AT ..."
- tempestn 9y agoThat always seemed bizarre to me. I get that printers, typewriters, and handwriting can all be identified, but why not just grab a common bic pen and hold it in your fist like a four year old, then print? I don't see how that could be matched to anything, and surely there's less room to screw up than when you're pasting out of a bunch of magazines and newspapers.
- kabdib 9y agoAgreed. I always figured this was a meme that made it easier for the police to track you down (just do some forensics on the magazine paper, fonts, etc., and locate someone with matching subscriptions). "Hey boss, we got a subscriber to Mad Bomber Monthly, Torture World, and BYTE." (flipFlipFLip) "One of these three guys?"
- Smushman 9y agoIf I drank milk, it would have shot out of my nose.
- nasredin 9y agoPlease take money! We will throw in another 5 if you also take your closet-case VP with you!
- Smushman 9y agoLong thought this should be possible... I am very impressed you figured out a way to do it. You could iterate the strategy further by creating a full font from the existing text, identifying spacing and etc for each next letter/character, calculate for word wrap lengths, and get it even closer.
- pacaro 9y agoIt’s probably most effective to identify the software and font that produced the original, exact positioning of text is complex: you’d need to identify kerning pairs, spacing rules within lines, sub pixel positioning (where appropriate), etc.
- cypherpunks01 9y agoLooks like a job for dynamic programming! It's a classic packing problem - searching for phrases of exact pixel width, where each of the unknown number of letters contributes a different number of pixels.
- senatorobama 9y agoIs it even tractable?
- mhb 9y agoWhy were digits ruled out? Like "100"?
- mlady 9y agomaybe there's some standard style guide that rules that out? I would be interested in seeing how many "matches" digits would yield
- astura 9y agoI don't know if it's an official style somewhere or not, but I was always taught 1-9 you spell out (one, two, three,...) And 10+ you write digits (10, 11, 12, 13,...). With a few exceptions, like if two numbers are next to each other.
- matt4077 9y agoA two-digit number (i. e. "13") would be too short for the redaction. A three- or four-digit number is highly unlikely ( because it would be treated as a single conspiracy, not as separate investigations of individuals). Plus, the sentence that follows after the colon seems to be a list of the individuals, and its length points at a single-digit number.
- eschutte2 9y agoIn the example shown, it's obvious it's four, since you can see the right edge of the letter sticking out past the black mark.
- mhb 9y agoOr n6, nn6, n7 or nn7.
- skosch 9y agoI am working on automating this process and trying to find similar hits for the remaining redactions. I was thinking that too, but it characterizes Carter Page as a "former campaign foreign policy advisor". The other names may have similar designations, making that a hopeless game. We do know, from an unredacted footnote, that one of them was Michael Flynn. So that makes 2/4.
- mattnewton 9y agoNo points for the others either: Manafort and Gates have also been indicted publically, and names and titles for each probably fit in the redacted section.
- andrewstellman 9y agoMatt Tait (infosec researcher at the Strauss Center at UT Austin) had some insight into this: https://twitter.com/pwnallthethings/status/967523196181336064 https://twitter.com/pwnallthethings/status/96752319618133606... – it's worth reading his first few comments, he has some good thoughts on using fixed with fonts in redacted documents.
- nabla9 9y agoConsider the opposite problem: What if you want to create document that retains redacted information. Some kind of Reed–Solomon type encoding in the typography that would allow retrieving the whole document even after it has been redacted and copied.
- d4mi3n 9y agoIf that's the case, I think the easiest thing to do would be to transcribe the document and include annotations inline about where content was omitted. e.g. Go from "XXXX people connected with.." to "[REDACTED] people connected with". So long as it's noted that redaction happened and one doesn't have the source document to glean things from, that would probably be sufficient to block the things the author of this article is doing. What I'm wondering, though, is if this is permissible. Being able to see how much of a document was redacted gives us some important context: 1. We know we're looking at the "original" report. This might not be the case with an approved transcription of some sort. 2. It's easy for a layman to get a sense of how much information is being withheld. Knowing what and how much was redacted seems like it's pretty important for maintaining trust and transparency with an organization that has legitimate reasons to withhold some amount of information.
- jobigoud 9y agoYou could use steganography to store the redacted parts.
- mulmen 9y agoWouldn't that change the non-redacted parts of the document? Or would there be a cover page with a story about a boy and his dog on every FOIA request?
- CapacitorSet 9y agoRandom thought: using serifs to encode additional bits with a Reed-Solomon would allow for, say, one bit per letter (either we use the serif or the sans-serif version). Assuming latin characters, it would allow for t/2 = 1/16 (6,25%) of the original text - a bit lower if we consider that letters like "o" do not have serifs, somewhat higher if we remove easily-guessable words like "the", "an" and so on. That's a good idea, I might do a quick Show HN about that.
- rafael859 9y agoTom Murphy did the same thing yesterday (on the same part of the text no less), but he got five instead of four! [1] [1] https://twitter.com/tom7/status/967568358861430785 https://twitter.com/tom7/status/967568358861430785
- mattnewton 9y agoI trust four more, because his alignment seems to be off in the beginning, and because the space that follows is too short for descriptions of 5 people assuming that it follows the same format as Carter page (association then full name).
- lifeformed 9y agoPlus there is that little nub that sticks out on the right side of the redaction, where the end of the "r" in "four" fits in perfectly.
- Bartweiss 9y agoI was surprised that wasn't mentioned; in addition to spacing issues it looks like the redaction literally just stopped too soon.
- ohiovr 9y agoHow high of a resolution were these scans?
- ixtli 9y agoI wonder whether or not there should be a rule on HN about linking to classified information. I am an extremely harsh critic of the american security state and the IC in general, but lower level employees can be fired or even charged with crimes for viewing things they're not supposed to and I don't think its right to disregard the effect on the individual. Obviously, in this case, it's only a single word, but i've seen leaked docs here before.
- jadedhacker 9y agoI understand your concerns, but we can't not publicize the information, that would violate the public interest. These leaks have little effect if they are not broadly distributed. However, out of concern for individual workers in the bureaucracy, I would support a tag on such stories to warn them before clicking. EDIT: On reading the other comments here, I don't think even a tag is warranted for this story as no leaking has occurred.
- lifeformed 9y agoI guess a "NSFW" tag technically fits - it's not something 100% safe to view at work.
- jadedhacker 9y agoI would prefer something like (LEAK) or (CLASSIFIED LEAK). NSFW means that many ordinary people that aren't subject to the logical contradictions of the classification system won't look because they think it's porn or something.
- astura 9y agoNSFW isn't enough, if you possess a security clearance it's not safe to view at home either as 1) you can't access classified information if you don't have a need-to-know and 2) you aren't authorized to access classified information from an unclassified computer.
- timb07 9y agoNothing has been leaked here. The source document has been publicly released. Insufficient care has been taken in redacting parts of the document. Responsibility for that falls to whoever redacted the document, not for whoever was able to deduce the redacted content.
- paulcole 9y agoBlacking out text with a sharpie isn’t tradecraft.
- timb07 9y agoMy understanding is that the text should be cut out with scissors instead.
- guidovranken 9y agoIt's not different for encrypted data that obscures the content but not its size. I've written before about how you can sometimes infer the size of a password in a HTML form submitted over an otherwise fully secure, correctly implemented SSL (HTTPS) connection. I've also shown how you you can guess the general location on the earth encoded in GPS coordinates from the size of the textual representation of those coordinates alone. It's not rocket science to make this deduction, but it is, I think, nonetheless an under-appreciated aspect of consumer-grade encryption. See here if you want to read my findings https://guidovranken.files.wordpress.com/2015/12/https-bicycle-attack.pdf https://guidovranken.files.wordpress.com/2015/12/https-bicyc...
- dspillett 9y ago> I've written before about how you can sometimes infer the size of a password in a HTML form submitted over an otherwise fully secure, correctly implemented SSL (HTTPS) connection. Good catch. Interesting, but obvious once you think about it. Not really a danger for properly chose passwords/passphrases (the search space would still be too large for brute forcing) but it would highlight where a brute force attempt is worth trying. I might have to add an extra (hidden) input to all our authentication pages programatically filled with random characters up to a length of 1024 minute the length of the entered password.
- Terr_ 9y agoFor extra paranoia, pad to a byte limit so that UTF-8 doesn't leak anything from it's fluctuating length. IMO if you don't need the entropy, it's probably easier to pad with some kind of "clearly not legal" character which is still visible when debugging, such as newlines.
- dspillett 9y ago> IMO if you don't need the entropy You almost certainly don't, but my default for any security related value is "properly random" except where the randomness itself might give clues. That way around is less likely to result in a "kicking oneself" situation in the future!
- weaksauce 9y agoFootnote seven in the next paragraph lists the four people anyway and also un-redacts that redaction.
- cypherpunks01 9y agoAny sense of the redacted word in that next paragraph?
- weaksauce 9y agotough to say but the most likely answer is that it's either four with extra space redacted or the word "several" or maybe "seven" as that would be 3 more and not a stretch to imagine there are 3 more in this admin that would have been under investigation?
- Moodles 9y agoThis reminds me of the New York times redaction fail: https://www.techdirt.com/articles/20140128/08542126021/new-york-times-suffers-redaction-failure-exposes-name-nsa-agent-targeted-network-uploaded-pdf.shtml https://www.techdirt.com/articles/20140128/08542126021/new-y...
- mattnewton 9y agoSomeone else tried that (posted further down) https://twitter.com/tom7/status/967568358861430785 https://twitter.com/tom7/status/967568358861430785 I think it is also unlikely given the space that follows too which would read more naturally if it was name + title, but it could be if the page was just off too much (I think the article here does a better job aligning than this guy did from looking at the beginning of the sentence.)
- cypherpunks01 9y agoThe Washington Post actually ran with this redaction analysis, in "What we learned from the Democratic response to the Nunes memo — and what we didn’t" posted a couple hours ago: Article: https://www.washingtonpost.com/news/politics/wp/2018/02/25/what-we-learned-from-the-democratic-response-to-the-nunes-memo-and-what-we-didnt/ https://www.washingtonpost.com/news/politics/wp/2018/02/25/w... Handy GIF: https://img.washingtonpost.com/pbox.php?url=https://www.washingtonpost.com/news/politics/wp-content/uploads/sites/11/2018/02/Redact.gif&op=noop https://img.washingtonpost.com/pbox.php?url=https://www.wash... "By September 2016, the FBI had opened investigations into four members of Trump’s campaign team. The Democratic memo says the information compiled by Steele into his infamous “dossier” of 17 raw intelligence reports didn’t get to the FBI’s counterintelligence team until the middle of September. By that point, we can conclude thanks to a sloppy redaction (noted by former intelligence officer Matt Tait) and an unredacted footnote that Page, Papadopoulos, former Trump campaign chairman Paul Manafort and Michael Flynn, who would go on to be Trump’s national security adviser, were all already under investigation." Matt Tait links to: https://twitter.com/pwnallthethings/status/967523196181336064 https://twitter.com/pwnallthethings/status/96752319618133606...
- zaroth 9y agoThe FBI actually first interviewed Steele in July 2016 about 25 days before they opened the investigation. “Simpson said Steele first shared his concerns with the FBI during the first week of July 2016 and in a subsequent meeting with the Rome official two months later when Steele provided the official ‘a full briefing’ of his findings” https://www.usatoday.com/story/news/politics/2018/01/09/dossier-author-told-fbi-had-source-inside-trump-organization/1017938001/ https://www.usatoday.com/story/news/politics/2018/01/09/doss...
- flatline 9y agoThis would appear to have been refuted by the latest release: https://www.politico.com/story/2018/02/24/democratic-memo-gop-fbi-trump-campaign-423446 https://www.politico.com/story/2018/02/24/democratic-memo-go...
- crafty 9y agoInteresting, but operating systems & graphics applications conform typefaces to different subpixel grids. 17pt Times in Pixelmator on a Mac is not 17pt Times on Photoshop on Windows, and certainly not 17pt Times on a scanned document printed on an undisclosed substrate by an unknown printer. Similarly, typeface tracking adds a significant margin of error to this interpretation method. Metrics can vary by application and some applications will automatically provide optical tracking adjustment for a better aesthetic. It's for these reasons that I'm convinced that anything other than trivial (in terms of LOE) & obvious disclosures won't be made using this methodology, and we can't universally trust the results.
- lokopodium 9y agoThey achieved 100% match on the unredacted text with Word. Empirically speaking, they've proven their software generates the same output as FBI's.
- colinbartlett 9y agoCould you use some kind of kerning randomizer to prevent this?
- philip1209 9y agoIf it is felt pen-redacted and then scanned, wouldn't there be some detectable difference between the parts of the paper with and without underlying ink?
- anfilt 9y agoYes if you have the orginal. However, the scans usually adjust the contrast so much that saved image is not grayscale or colour. Its either black or white.
- mikeash 9y agoSeems like if you truly care about hiding the redacted information, the only choice is to manually retype the document with the redacted text replaces with something that has no information content, like the literal text “REDACTED.” Why don’t they do this?
- skygazer 9y agoProbably to maintain the sense of boundedness of the withheld. How big is the secret they're hiding? Is it of unlimited or limited length? Your technique would obscure the sense of scale.
- mikeash 9y agoYou could add multiple instances of “REDACTED” to show approximately how much stuff is missing I’d thats important.
- Bucephalus355 9y agoAlso a good time to mention “Van Eck Radiation”. This is something all CRT screens, and to a lesser extent, LCDs, emit. If you pick up this radiation and know the model of the screen being used, you essentially have access to a live visual of a person’s monitor. Also worth mentioning that just like the Secret Service has an ink database on all the printer types in the world, the NSA is supposed to have a database of what different keyboards sound like. This means that simply by recording the sound of you typing, they can infer keystrokes / characters. Obviously the easiest way to record this is by hacking your phone, which is right next to you. https://en.m.wikipedia.org/wiki/Van_Eck_phreaking https://en.m.wikipedia.org/wiki/Van_Eck_phreaking
- unimpressive 9y agoActually, the exploit with the keyboards is both more interesting and more sophisticated than that. As described in Silence On The Wire, essentially how it works is that English letters are not randomly distributed. If you hear any given keystroke, you know that the most likely letter to be pressed knowing nothing else is the letter 'e'. This on its own isn't very helpful, but you don't type every letter on your keyboard at the same speed. It takes you ever so slightly longer to type 'z' than 'f'. You of course also only type one key at a time, as a consequence it's possible to merely follow a procedure something like this to recover English text given audio of it being typed on a standard keyboard: - Assign a prior probability of letter frequencies based on a corpus of the language text you're analyzing. - Separate the different keystrokes in the audio file into a series of times between keystrokes. (i.e, have your program recognize one keystroke as distinct from another) - Based on the subtle timing differences between keystrokes, assign various lengths to different timings. - Using the prior probability of the letter frequencies, assign the different time lengths to different characters based on their frequencies. - You now have a straightforward mapping between the distance between two keystrokes and the character typed, which should allow you to decode the typed text. Further calibration can probably be had by considering a word dictionary and using fuzzy matching to detect how often words are decoded incorrectly and what the correct decoding would be.
- smrq 9y ago
- tzahola 9y agoCan we use this approach to recover the missing 18 minutes of the Watergate tapes?
- gravypod 9y agoI wonder if someone could train a DNN to uncensor images of similarly redacted information
- eccbits 9y agoWhile this is fun technologically, is there an onligation for the author to think of the negative impact? Certainly this isn't hekling the US.
- sgtcodfish 9y agoI'm not sure there's any obligation on the author of this article; if there's any, it's on the party that authorised the document to be released in the state it's in. You can be sure that the high-level "bad guys" (for anybody's definition of bad) did the same thing the author did. That said, it seems remarkably heavy-handed to censor in this case, assuming that "four" is the missing text and if the 4 names match those given later - Papadoupolous, Flynn, Page and Manafort. We obviously know about all of those already - although I suppose there could be some use in concealing when exactly investigations were opened into them.
- aaronbrethorst 9y agoThe title of the post is a pun: Watergate's "Deep Throat" was a man named Mark Felt, who was the Associate Director of the FBI. He felt jilted over his lack of promotion to the Directorship, and responded by leaking to Woodward and Bernstein. https://www.vanityfair.com/news/2013/11/watergate-leak-mark-felt-career-ladder https://www.vanityfair.com/news/2013/11/watergate-leak-mark-...
- astrodust 9y agoDisappointed the Marker Felt font didn't make a cameo.
- salgernon 9y agoSo, given the very small list of possible names associated with the subject of this paragraph, I should think the number of permutations would be in the low hundreds. I’d be willing to bet the larger block falls to this analysis by morning in Washington. (Not that that is necessarily the most important block, but it probably has the bounded scope.)
- kw71 9y agoWhy can't this be presented without requiring JavaScript???
- jwilk 9y agoHere's a copy that works without JS: https://gist.github.com/anonymous/37200394d975e1a86ee9d19bef0cf2e9 https://gist.github.com/anonymous/37200394d975e1a86ee9d19bef...
- Angostura 9y agoThe joke's on him. The number is actually numeric, and I work it out to be 8423
- StavrosK 9y agoYou can see the right tip of the "r" past the redacted part.
- cantrevealname 9y agoHere on Hacker News four years ago we figured out the redacted name of a country in a similar way. The Intercept[1] reported that the National Security Agency was secretly recording the audio of every phone call in the Bahamas and in one other unnamed country. Looking at the length of country name in the source document, we figured out that it was Afghanistan[2] based on the length of the blacked-out area and that it couldn't word wrap to a second line. [1] https://firstlook.org/theintercept/article/2014/05/19/data-pirates-caribbean-nsa-recording-every-cell-phone-call-bahamas/ https://firstlook.org/theintercept/article/2014/05/19/data-p... [2] https://news.ycombinator.com/item?id=7768839 https://news.ycombinator.com/item?id=7768839
- arca_vorago 9y agoJust fyi the government became aware of this issue in about ~2010 and changed the policy on how redactions were done, changing from blackout to whitebox with black edges that were overlayed on the text with small size variations to prevent the attack. Old documents though that have already been released are likely to be ripe for this kind of analysis.
- onychomys 9y agoThis one was released over the weekend, though, so apparently somebody didn't get the memo.
- Bartweiss 9y agoI'm guessing Congressional memo redactions, especially these days, aren't quite up to the standards of intelligence agencies that have to redact and release for FOIA on a daily basis.