5 ms·

Cool to see this here. It’s funny because we do so many huge, complex, multiyear projects at Free Law Project, but this is the most viral any of our work has e

Cool to see this here. It’s funny because we do so many huge, complex, multiyear projects at Free Law Project, but this is the most viral any of our work has ever gone! Anyway, I made X-ray to analyze the millions of documents we have in CourtListener so that we can try to educate people about the issue. The analysis was fun. We used S3 batch jobs to analyze millions of documents in a matter of minutes, but we haven’t done the hard part of looking at the results and reporting them out. One day.

thangalin 9mo ago
https://www.argeliuslabs.com/deep-research-on-pdf-redaction-failures-and-security-risks-exploits-and-best-practices/ https://www.argeliuslabs.com/deep-research-on-pdf-redaction-... > Information Leaking from Redaction Marks: Even when content is properly removed, the redaction marks themselves can leak some information if not done carefully. For example, if you have a black box exactly covering a word, the length of that black box gives a clue to the word’s length (and potentially its identity). Does X-ray employ glyph spacing attacks and try to exploit font metric leaks?
- mlissner 9mo ago
  No, we worked with researchers that developed that kind of system, but didn't broadcast our work b/c the research was too sensitive. Seems the cat is out the bag now though. I think the combination of AI and font-metrics is going to be wild though. You ought to be able to make a system that can figure out likely words based on the unredacted ones and the redaction's size. I haven't seen any redaction system yet that protects against this.
  - thangalin 9mo ago
    > I haven't seen any redaction system yet that protects against this. The linked article suggests widening redacted areas more than needed with some randomization applied to the width. Strikes me that that wouldn't do much except add a few more possible solutions.
    - vlovich123 9mo ago
      Yeah, the more robust protection is to widen to a constant. But in the general case that could require reflowing the pdf. But honestly single word redactions are really probably useless with cheap AI that can highly accurately fill in the gaps
      - rgmerk 9mo ago
        Depends what you're trying to hide. If the redaction is a person's name, and there's nothing else to give the person's identity away, single word redaction probably works reasonably well, AI or no AI.
        godelski 9mo ago
        > If the redaction is a person's name I'm not sure if you're aware, but peoples names are variable in length. We are talking about a system that can identify single character differences. So that does reduce the search space, especially since names are not all possible letter permutations. Combine that with the fact that it isn't uncommon to see partial first letters show up. You can even see some instances in the Epstein files. Of course, you can also take this further. Even if you can't recover names you can get meta information about how many parties are involved by recognizing different length redactions correspond to different entities. While same length redaction doesn't guarantee same entity it is a hint.
        mycall 9mo ago
        It is also common for authors to misspell names (proper nouns) in an attempt to determine who leaks docs (and to force non-matches for FOIA requests).
        mhast 9mo ago
        If you want to fingerprint text you can also do it by small insignificant changes to text which doesn't change the meaning. If you have a number such locations with alternatives then you can make a number of identifiable versions by combining alternates.
  - vlovich123 9mo ago
    I thought glyph spacing attacks are an old idea; like I recall reading about such ideas 10-20 years ago unless I’m misremembering. Can you clarify why it was considered “too sensitive” if the whole point of this effort is to showcase these attacks?
    - mlissner 9mo ago
      It’s a fine line. Most redactions are for the good, to protect someone or something. For example even in the Epstein files, where some redactions are being abused, most redactions are protecting victims. If there’s a way to undo huge amounts of redactions, that’d certainly be a net negative. Sort of like if encryption were suddenly broken, you wouldn’t publish a paper saying so. Our goal has always been to educate about the problem so that it can be addressed. We didn’t have resources to push on the font metrics approach, so we stayed mostly quiet about it.
      - btreecat 9mo ago
        > If there’s a way to undo huge amounts of redactions, that’d certainly be a net negative. Sort of like if encryption were suddenly broken, you wouldn’t publish a paper saying so. I can't state emphatically enough how this is not the right mental playbook. If you have found a vulnerability, it's likely someone else has too. By sitting on it, you only create more future victims. Disclosure will lead to fixing this issue, mitigating it's precense, or switching tools/workflows, possibly a combination of. Sitting on it only ensures that folks who think they are protected, actually aren't.
        mlissner 9mo ago
        We’re familiar with vulnerability disclosure philosophies, but what if the problem can’t be fixed because there’s no forward secrecy for the hundreds of millions of documents that are already out there? It’s tricky stuff and we have limited resources, unfortunately.
        opello 9mo ago
  - NoboruWataya 9mo ago
    This is going to be a disaster IMO because AI will just hallucinate what it thinks is the most probable redacted word and people will take that as gospel.
    - PunchyHamster 9mo ago
      "don't redact or we will hallucinate something worse and make people believe it as gospel" is nice deterrent
      - rafram 9mo ago
        We don’t need a “deterrent” against things being redacted in publicly released documents. We can have transparency without the whole world finding out the names of victims and witnesses, people’s phone numbers and SSNs, etc., every time a document is released.
  - hahn-kev 9mo ago
    Maybe we should all just use mono-space fonts for everything
hsbauauvhabzb 9mo ago
Presumably with font kerning and pixel perfect recreation of the source, it would be possible to guess the word very accurately. The strings oioioi and oooiii will have different widths in some fonts because character organisation matters a lot.
- setopt 9mo ago
  I suppose it gets a bit more complex again if you enable stuff like microtype, but even then you can probably measure how much inter-letter and inter-word spacing has been adjusted by just scanning other text in the same line. I think the conclusion is honestly that PDF is an outdated format for keeping records that might have to be redacted in the future, like court documents. Something reflowable like epub could have the text replaced with constant-space black squares instead no hints leaked as someone mentioned in a parallel comment.
  - hsbauauvhabzb 9mo ago
    I’ve never heard anyone suggest PDF is a good format, and while I don’t know the spec, I imagine based on the acrobat cve list it’s an absolute clusterfuck.

thangalin 9mo ago

https://www.argeliuslabs.com/deep-research-on-pdf-redaction-failures-and-security-risks-exploits-and-best-practices/ https://www.argeliuslabs.com/deep-research-on-pdf-redaction-... > Information Leaking from Redaction Marks: Even when content is properly removed, the redaction marks themselves can leak some information if not done carefully. For example, if you have a black box exactly covering a word, the length of that black box gives a clue to the word’s length (and potentially its identity). Does X-ray employ glyph spacing attacks and try to exploit font metric leaks?

mlissner 9mo ago
No, we worked with researchers that developed that kind of system, but didn't broadcast our work b/c the research was too sensitive. Seems the cat is out the bag now though. I think the combination of AI and font-metrics is going to be wild though. You ought to be able to make a system that can figure out likely words based on the unredacted ones and the redaction's size. I haven't seen any redaction system yet that protects against this.
- thangalin 9mo ago
  > I haven't seen any redaction system yet that protects against this. The linked article suggests widening redacted areas more than needed with some randomization applied to the width. Strikes me that that wouldn't do much except add a few more possible solutions.
  - vlovich123 9mo ago
    Yeah, the more robust protection is to widen to a constant. But in the general case that could require reflowing the pdf. But honestly single word redactions are really probably useless with cheap AI that can highly accurately fill in the gaps
    - rgmerk 9mo ago
      Depends what you're trying to hide. If the redaction is a person's name, and there's nothing else to give the person's identity away, single word redaction probably works reasonably well, AI or no AI.
      - godelski 9mo ago
        > If the redaction is a person's name I'm not sure if you're aware, but peoples names are variable in length. We are talking about a system that can identify single character differences. So that does reduce the search space, especially since names are not all possible letter permutations. Combine that with the fact that it isn't uncommon to see partial first letters show up. You can even see some instances in the Epstein files. Of course, you can also take this further. Even if you can't recover names you can get meta information about how many parties are involved by recognizing different length redactions correspond to different entities. While same length redaction doesn't guarantee same entity it is a hint.
        mycall 9mo ago
        It is also common for authors to misspell names (proper nouns) in an attempt to determine who leaks docs (and to force non-matches for FOIA requests).
        mhast 9mo ago
        If you want to fingerprint text you can also do it by small insignificant changes to text which doesn't change the meaning. If you have a number such locations with alternatives then you can make a number of identifiable versions by combining alternates.
- vlovich123 9mo ago
  I thought glyph spacing attacks are an old idea; like I recall reading about such ideas 10-20 years ago unless I’m misremembering. Can you clarify why it was considered “too sensitive” if the whole point of this effort is to showcase these attacks?
  - mlissner 9mo ago
    It’s a fine line. Most redactions are for the good, to protect someone or something. For example even in the Epstein files, where some redactions are being abused, most redactions are protecting victims. If there’s a way to undo huge amounts of redactions, that’d certainly be a net negative. Sort of like if encryption were suddenly broken, you wouldn’t publish a paper saying so. Our goal has always been to educate about the problem so that it can be addressed. We didn’t have resources to push on the font metrics approach, so we stayed mostly quiet about it.
    - btreecat 9mo ago
      > If there’s a way to undo huge amounts of redactions, that’d certainly be a net negative. Sort of like if encryption were suddenly broken, you wouldn’t publish a paper saying so. I can't state emphatically enough how this is not the right mental playbook. If you have found a vulnerability, it's likely someone else has too. By sitting on it, you only create more future victims. Disclosure will lead to fixing this issue, mitigating it's precense, or switching tools/workflows, possibly a combination of. Sitting on it only ensures that folks who think they are protected, actually aren't.
      - mlissner 9mo ago
        We’re familiar with vulnerability disclosure philosophies, but what if the problem can’t be fixed because there’s no forward secrecy for the hundreds of millions of documents that are already out there? It’s tricky stuff and we have limited resources, unfortunately.
        opello 9mo ago
- NoboruWataya 9mo ago
  This is going to be a disaster IMO because AI will just hallucinate what it thinks is the most probable redacted word and people will take that as gospel.
  - PunchyHamster 9mo ago
    "don't redact or we will hallucinate something worse and make people believe it as gospel" is nice deterrent
    - rafram 9mo ago
      We don’t need a “deterrent” against things being redacted in publicly released documents. We can have transparency without the whole world finding out the names of victims and witnesses, people’s phone numbers and SSNs, etc., every time a document is released.
- hahn-kev 9mo ago
  Maybe we should all just use mono-space fonts for everything

hsbauauvhabzb 9mo ago

Presumably with font kerning and pixel perfect recreation of the source, it would be possible to guess the word very accurately. The strings oioioi and oooiii will have different widths in some fonts because character organisation matters a lot.

setopt 9mo ago
I suppose it gets a bit more complex again if you enable stuff like microtype, but even then you can probably measure how much inter-letter and inter-word spacing has been adjusted by just scanning other text in the same line. I think the conclusion is honestly that PDF is an outdated format for keeping records that might have to be redacted in the future, like court documents. Something reflowable like epub could have the text replaced with constant-space black squares instead no hints leaked as someone mentioned in a parallel comment.
- hsbauauvhabzb 9mo ago
  I’ve never heard anyone suggest PDF is a good format, and while I don’t know the spec, I imagine based on the acrobat cve list it’s an absolute clusterfuck.