14 ms·
Wuffs: Wrangling Untrusted File Formats Safely
- dang 2y agoRelated: Wuffs the Language - https://news.ycombinator.com/item?id=26731305 https://news.ycombinator.com/item?id=26731305 - April 2021 (75 comments) Wuffs’ PNG image decoder - https://news.ycombinator.com/item?id=26714831 https://news.ycombinator.com/item?id=26714831 - April 2021 (138 comments)
- newman314 2y agoDoes anyone know of a tool that can do this for PDFs instead?
- warkdarrior 2y agoAs soon as someone writes a Javascript interpreter in Wuffs..
- jchw 2y agoDo you ever need a JS interpreter to parse a PDF? That's horrifying. I understand PDF has a bunch of limbs, but I always assumed the JS stuff was at least separate from the parsing. (I am familiar with the PDF format at a lower level but I never touched any of the weird features.)
- timschmidt 2y agoI wrote an SVG that's all javascript, no elements. All the graphics are generated dynamically at runtime by the javascript. It's SVG standards compliant, but only opens correctly in browsers, not in inkscape or other desktop publishing apps. I work a lot in OpenSCAD, and had a need to design some custom graph paper. So I found the subset of SVG which was similar to OpenSCAD. :)
- jszymborski 2y agoFrankly, I wouldn't begrudge a website for not correctly parsing an svg I composed entirely with javascript. It's annoying you can't just "flatten" or "bake" such an svg like yours into one composed entirely of elements (unless one exists?)
- csande17 2y agoOften you can open the SVG in a browser and then use the developer tools to copy out the resulting nodes as "flat" SVG source code. Chrome even includes a --dump-dom flag you can use to do this on the command line, although I haven't tested it with an SVG.
- jszymborski 2y agoClever!
- Joel_Mckay 2y agoThere are PDF readers that do not support the scripting format extensions. Note this does not prevent unscrupulous companies abusing dominant market positions to voluntarily embed machine and serial hash watermarks. To be clear: formats like pdf, ps, webp, svg, and tiff are so badly implemented in some ecosystems... they can't _ever_ be assumed safe input formats. Thus, at some point people need to spin up an actual VM to transcode a "web" version, and scrub each stage of the rendering pipeline like a virus or header injection is already present. "I never play where nice things are, and don't break things" (Eliza Mowry Blven, The Humanitarian Review, Volume 3, March, 1905) Cheers =3
- tialaramex 2y agoI worked with TIFF pretty extensively, it's a mess but I don't see why a WUFFS TIFF codec can't be fine. What makes you say you need "an actual VM to transcode" a TIFF ?
- Joel_Mckay 2y agoThe complex formats of tiff and tga specifications makes it nearly impossible to span all the edge-cases with unit-tests. A VM can be in a known-state snapshot, process pre/post signature logged/compared with a scripted debugger, and binary input/output stripped of non-compliant metadata/blobs at each stage of the pipeline if the process behaves as expected. I've yet to find a better method than Honeypots to sustainably mitigate the complex leaky dependency mess on traditional architectures. It has been my experience that "all software is terrible, but some of it is useful". It may just be my bias, but I see code smell getting worse in recent decades... Have a nice day, =3 https://www.youtube.com/watch?v=aCbfMkh940Q https://www.youtube.com/watch?v=aCbfMkh940Q
- tialaramex 2y agoSo, there's actually no particular reason and if somebody cares to write one then yup, TIFF codec in WUFFS would in fact be safer and faster than your uh, approach.
- deleted 2y ago[deleted]
- ThePowerOfFuet 2y agohttps://dangerzone.rocks/ https://dangerzone.rocks/
- trustno2 2y agopdfs are really really hard. the only viewer that parses them semi-correctly is ... Acrobat Reader. try to ever read any code for PDFs and see all the horrors. Google gave up and just bought the code from foxit.
- kjksf 2y agoGoogle was never trying to write PDF reader from scratch so they never "gave up". They just bought foxit code to save years of development when they wanted to ship PDF reader in Chrome. Your comment about "the only viewer that is semi-correct" is also wildly off the mark. Parsing correctly written PDF files is hard but multiple engines can do it correctly. Parsing real life PDFs is much harder then correctly implementing PDF spec because lots of PDFs are just broken. They generators create invalid PDF files and then PDF readers have to spend heroic efforts to somehow make sense of this brokenness. Adobe does it better than most because... well it would be embarrassing if they didn't. They invented the format, they make money from their tools, they were doing it the longest, they have the largest archive of broken PDFs for testing etc. It's hard to expect that e.g. an open-source project with one or two developers can match that. I work on SumatraPDF so I know.
- trustno2 2y agoOK. In one of my previous jobs, I needed to auto-fill PDF forms on BE (among other things with PDFs), the only thing that worked reliably across PDFs was Acrobat. I did not try SummatraPDF. edit: it seems Summatra doesn't support PDF forms? Either AcroForms or XFA forms?
- yjftsjthsd-h 2y agoThis is one of my favorite attempts at better programming language safety, because it compiles down to C that can then be shipped like normal C, so you don't get the ecosystem friction like with ex. Rust.
- vlovich123 2y agoIt’s an interesting idea for sure but it isn’t a general purpose language, so the problem domains it can solve is very very different vs what Rust is trying to do.
- tialaramex 2y agoNigel has said that emitting "unsafe" Rust is a reasonable thing for a hypothetical WUFFS 1.0 to be able to do as an alternative to C. As with good "unsafe" Rust written by humans WUFFS would know exactly why what it's doing is fine, it's just that the Rust compiler can't necessarily see that, hence the need to label it "unsafe". Today C makes most sense given the WUFFS language is still in flux. [Edited to fix a serious typo]
- nequo 2y agoWhat would be the primary benefit of emitting Rust rather than C? Both would be considered safe (assuming Wuffs generates correct code), and Rust could access the C code via FFI. Is there something I’m missing?
- vlovich123 2y agoNominally it can safely elide bounds checks via unsafe that it has proved are actually safe within the constraints of Wuffs, which is what it does for C (+ the language is built for more easy translation to vectorizated than something like llvm is able to do for general purpose languages). So basically higher performance. FFI nominally has a runtime and compile time cost - whether that matters for you in particular will depend on your needs, but being able to publish a very simple crate without a build.rs to manage can have an attraction.
- deleted 2y ago[deleted]
- tedunangst 2y agoRelated, in the sense of solving the same problem in a different manner: https://rlbox.dev/ https://rlbox.dev/
- Ono-Sendai 2y agoWuffs is great. I use it in Substrata (https://substrata.info/ https://substrata.info/) for loading PNGs. It is both faster and safer than LibPNG. It's something around 2x faster than LibPNG in my tests (depending on the PNG file), see timings here: https://github.com/google/wuffs/issues/13#issuecomment-1732575707 https://github.com/google/wuffs/issues/13#issuecomment-17325... So generally Wuffs is great and you should use it to decode your PNGs. There are some downsides: not all of the obscure bit depths and formats that PNG supports are loaded as-is, some are converted to more standard formats. Also the Wuffs documentation is a bit hard to understand. It's a litle bit of a mission getting PNG decoding working. You can see my code for that here though: https://github.com/glaretechnologies/glare-core/blob/2c7174ca95c68bfc4cea982ec12ac60db0bfd61d/graphics/PNGDecoder.cpp#L175 https://github.com/glaretechnologies/glare-core/blob/2c7174c...
- repsilat 2y agoThe "mango" lib [1] claims to be even faster for PNGs. Actively maintained but doesn't have as much buzz, I think the devs haven't advertised it as much on places like this. Also, it has the funniest testimonials. 1: https://github.com/t0rakka/mango https://github.com/t0rakka/mango
- yjftsjthsd-h 2y agoSpeed isn't the only thing that matters; is mango as safe as wuffs in the face of untrusted input?
- edflsafoiewq 2y agoWhere does the extra speed come from?
- nicoburns 2y agoMy understanding is that libpng is unoptimised and 5-10x faster is possible.
- pornel 2y ago
- YoshiRulz 2y agoSuperior in every sense to that Magicka garbage they released a couple months ago. I'm excited to see its via-Rust codegen.
- jay-barronville 2y agoWuffs is cool, but you can get similar results writing normal C library code, compiling it into a .wasm binary via Clang, and then running the .wasm binary through the `wasm2c` tool of the WebAssembly Binary Toolkit [0]. I personally prefer this method, although Wuffs will usually produce faster code. [0]: https://github.com/WebAssembly/wabt/tree/44837a7236e85c048de3214fd2ae4c37d3a034ce/wasm2c https://github.com/WebAssembly/wabt/tree/44837a7236e85c048de...
- krick 2y agoIt is not obvious to me why this should guarantee safety.
- jay-barronville 2y ago`wasm2c` fully implements the WebAssembly sandbox execution environment [0][1] and has the passing tests to prove it. To be a bit more specific, the .wasm binary you generate initially already has the WebAssembly semantics baked in (obviously) and `wasm2c` creates a portable C translation of the WebAssembly while also ensuring that the execution environment is sandboxed (e.g., the code traps when attempting out-of-bounds memory accesses). [0]: https://webassembly.org https://webassembly.org [1]: https://github.com/WebAssembly/wabt/issues/2289#issuecomment-1709207053 https://github.com/WebAssembly/wabt/issues/2289#issuecomment...
- eviks 2y agoHow much faster (say, for something like an image codec)
- jay-barronville 2y agoThis might not be what you want to hear (and I might get downvoted for it), but it’s what I consider the best answer: Implement something minimal but useful (and realistic) using both methods and benchmark them yourself. Even if I told you some of the numbers I’ve seen in my experiments and usage, it wouldn’t be wise to trust them or let them taint your opinion.
- refibrillator 2y agoCan Wuffs provide stronger safety guarantees than techniques like WasmBoxC? My understanding is that compiling unsafe C to WASM and back would also guarantee safety with respect to buffer overflows, integer arithmetic overflows and null pointer dereferences. It’s nice not annotating code to explicitly prove invariants to the compiler like you would in say Wuffs or Rust, but I suppose that’s what limits performance.
- klabb3 2y agoDoesn’t wasm have a memory model as well? So unless you sandbox certain parts of it you can still in theory have access across different C functions, within the same wasm module? What seems nice about wuffs is that it has no side effects and a clear project scope. Deserialization is so riddled with severe issues that it does kind of warrant its own DSL. OTOH, some legacy formats will probably never be ported.
- CJefferson 2y agoTechnically, while WASM promises you put data in and get data out, you can still have memory corruption (as it has a flat memory), so I could make a (for example) gif with some color palette, then later overflow and rewrite the palette. Not fatal, but perhaps annoying.
- azakai 2y agoYes, Wuffs can do better than WasmBoxC because it does more than sandboxing of the code. It also checks things like integer overflows which can lead to exploits that are technically not memory safety issues, but still potentially dangerous. But the tradeoff is that you need to rewrite your code for Wuffs, while WasmBoxC can sandbox anything that compiles to wasm and prevent it from corrupting the outside, including existing code in C, C++, Zig, unsafe Rust, etc. etc.
- who-shot-jr 2y agoCould you use this to make sure users uploading files to your website are correct (i.e only jpegs and valid image data)? But in a fast and safe way, or is this overkill?
- pornel 2y agoYes, you could. But be careful to make sure that there's no more data left after the decoder finishes, because it's possible to append a ZIP file (or acropcalypse) at the end of any other valid image file data, and decoders usually stop at the end of the image and don't parse past its end, so won't complain about extra data.
- deleted 2y ago[deleted]
- dividuum 2y agoNot sure that’s possible. I’m pretty sure it is not safe to assume „parses in wuffs“ -> „is safe in any other decoder“. I’m using wuffs to check user upload (see my recent response in another thread) but I still generate out linear RGBA and work with that. I still consider the original JPEG data hostile.
- Alifatisk 2y agowhile true { ... } endwhile Please, let the end brackets should be enough.
- zelphirkalt 2y agoI have a question about this. Why is wuffs considered to be safe? In this thread I saw a code example from cpp using wuffs, which seems to the a C library accessed from cpp in that example. Why should I trust that C library to be safe or safer than other libs?
- yoochan 2y agoCool project, I'm eagerly waiting for a jpegxl decoder to be written in wuffs so that Google won't have any reason not to support the format... I would be glad to have a try but the language specification and documentation is almost non existent.