4 ms·

Pure: A static analysis file format checker

98 points by emidoots 4y ago

jwilk 4y ago
Backstory: https://news.ycombinator.com/item?id=23760809 https://news.ycombinator.com/item?id=23760809
- _vvhw 4y ago
  Thanks for the link to that comment of mine. Wow, that's a blast from the past! Here's also a recent HN discussion about Pure, in the context of buffer bleeds and memory safe languages, with tptacek: https://news.ycombinator.com/item?id=31852820 https://news.ycombinator.com/item?id=31852820
tyingq 4y ago
Seems interesting to build something looking for things like zip file bombs, unexpected sparseness, overflow/underflow, and so on with C.
ducktective 4y ago
Is there something like this for formats other than ZIP like jpeg or pdf?
- jwilk 4y ago
  For tar: https://github.com/ferivoz/alquitran https://github.com/ferivoz/alquitran
- _vvhw 4y ago
  I wanted to do this for those formats in particular, as well as Office files. Fastmail were going to sponsor some of this work to extend Pure, and I was chatting with the folks at Hey, but I couldn't get enough other sponsors to fund the work. The idea is to use Pure to have a "Zero-Day Defense Mode" button for your email provider that you can push, if you want static analysis on 90% of the file formats coming and going, so that you can have stronger assurance on email attachments that you open. For example, I believe that Pure would have prevented last year's zero-click traversal against Apple Mail. There are also so many ways to use file format anomalies to take out AV engines, so Pure can act as a first line of defense, before hostile data gets to them. For example, protecting them from the David Fifield zip bomb. What I find most interesting with these checks, is that they're looking for explicit bleeds or explicit overflow bounds. So the signal to noise ratio is high. Mostly, if people are interested, I would really love to move this all to Zig, to benefit from the checked arithmetic and spatial safety, since otherwise everything is all single-threaded run-to-completion.
  - deleted 4y ago
    [deleted]
- tgv 4y ago
  Idk, but there's an interesting project, wuffs: a programming language specifically aimed at efficiently and safely parsing document structure. There is a PNG decoder [2]. 1. https://github.com/google/wuffs https://github.com/google/wuffs 2. https://news.ycombinator.com/item?id=26714831 https://news.ycombinator.com/item?id=26714831
- _vvhw 4y ago
  PDF is also a minefield, for how many ways there are to mess with the format: https://blog.didierstevens.com/2008/04/29/pdf-let-me-count-the-ways/ https://blog.didierstevens.com/2008/04/29/pdf-let-me-count-t...
fbanon 4y ago
What does "static analysis" mean in this context? In program analysis it means that you're deducing properties of the program without executing it.
- frabert 4y ago
  I think it's meant in the sense of "can detect the exploits without having to actually attempt decompressing anything"
  - _vvhw 4y ago
    Exactly, although, at present, Pure will also do dynamic analysis on the compressed stream once the metadata checks out, to verify that the compressed stream checks out with the metadata, i.e. that there's no semantic gap between the designated size and the actual stream size. This guards against one category of ZIP bomb.
_vvhw 4y ago
What a surprise to open up HN today and see Pure staring back at me! Here's the HN thread from 2019 where rattray encouraged me just to go for it and open-source the original JavaScript version: https://news.ycombinator.com/item?id=20352439 https://news.ycombinator.com/item?id=20352439 That led to a contract with Microsoft, which then led to my work at Coil on TigerBeetle, a distributed database written in Zig: https://www.youtube.com/watch?v=rNmZZLant9o https://www.youtube.com/watch?v=rNmZZLant9o
robertlagrant 4y ago
This looks really interesting; can an end user make use of it directly? Or is it more for providers of things?
- _vvhw 4y ago
  You can run it as a CLI directly on any ZIP file that you pass by path. I do want to port it to Zig though, to get rid of the NPM install bootstrapping, have a single binary, and also gain Zig's checked arithmetic and spatial safety.

98 points by emidoots 4y ago

jwilk 4y ago

Backstory: https://news.ycombinator.com/item?id=23760809 https://news.ycombinator.com/item?id=23760809

_vvhw 4y ago
Thanks for the link to that comment of mine. Wow, that's a blast from the past! Here's also a recent HN discussion about Pure, in the context of buffer bleeds and memory safe languages, with tptacek: https://news.ycombinator.com/item?id=31852820 https://news.ycombinator.com/item?id=31852820

tyingq 4y ago

Seems interesting to build something looking for things like zip file bombs, unexpected sparseness, overflow/underflow, and so on with C.

ducktective 4y ago

Is there something like this for formats other than ZIP like jpeg or pdf?

jwilk 4y ago
For tar: https://github.com/ferivoz/alquitran https://github.com/ferivoz/alquitran
_vvhw 4y ago
I wanted to do this for those formats in particular, as well as Office files. Fastmail were going to sponsor some of this work to extend Pure, and I was chatting with the folks at Hey, but I couldn't get enough other sponsors to fund the work. The idea is to use Pure to have a "Zero-Day Defense Mode" button for your email provider that you can push, if you want static analysis on 90% of the file formats coming and going, so that you can have stronger assurance on email attachments that you open. For example, I believe that Pure would have prevented last year's zero-click traversal against Apple Mail. There are also so many ways to use file format anomalies to take out AV engines, so Pure can act as a first line of defense, before hostile data gets to them. For example, protecting them from the David Fifield zip bomb. What I find most interesting with these checks, is that they're looking for explicit bleeds or explicit overflow bounds. So the signal to noise ratio is high. Mostly, if people are interested, I would really love to move this all to Zig, to benefit from the checked arithmetic and spatial safety, since otherwise everything is all single-threaded run-to-completion.
- deleted 4y ago
  [deleted]
tgv 4y ago
Idk, but there's an interesting project, wuffs: a programming language specifically aimed at efficiently and safely parsing document structure. There is a PNG decoder [2]. 1. https://github.com/google/wuffs https://github.com/google/wuffs 2. https://news.ycombinator.com/item?id=26714831 https://news.ycombinator.com/item?id=26714831
_vvhw 4y ago
PDF is also a minefield, for how many ways there are to mess with the format: https://blog.didierstevens.com/2008/04/29/pdf-let-me-count-the-ways/ https://blog.didierstevens.com/2008/04/29/pdf-let-me-count-t...

fbanon 4y ago

What does "static analysis" mean in this context? In program analysis it means that you're deducing properties of the program without executing it.

frabert 4y ago
I think it's meant in the sense of "can detect the exploits without having to actually attempt decompressing anything"
- _vvhw 4y ago
  Exactly, although, at present, Pure will also do dynamic analysis on the compressed stream once the metadata checks out, to verify that the compressed stream checks out with the metadata, i.e. that there's no semantic gap between the designated size and the actual stream size. This guards against one category of ZIP bomb.

_vvhw 4y ago

What a surprise to open up HN today and see Pure staring back at me! Here's the HN thread from 2019 where rattray encouraged me just to go for it and open-source the original JavaScript version: https://news.ycombinator.com/item?id=20352439 https://news.ycombinator.com/item?id=20352439 That led to a contract with Microsoft, which then led to my work at Coil on TigerBeetle, a distributed database written in Zig: https://www.youtube.com/watch?v=rNmZZLant9o https://www.youtube.com/watch?v=rNmZZLant9o

robertlagrant 4y ago

This looks really interesting; can an end user make use of it directly? Or is it more for providers of things?

_vvhw 4y ago
You can run it as a CLI directly on any ZIP file that you pass by path. I do want to port it to Zig though, to get rid of the NPM install bootstrapping, have a single binary, and also gain Zig's checked arithmetic and spatial safety.