5 ms·
Unit Testing PDF Generation
- t344344 4y agoPDF may have generation date etc, much better to use OCR and compare strings.
- crazygringo 4y agoNo, at the end of the day the proposed approach of rendering to an image and comparing pixels is best. Things can go wrong graphically that OCR won't catch, like an entire background color is missing or an image is missing. If you're worried about a generation date in the margin, then compare inside of a bounding box that includes most of the page but not that margin. Or just use a fixed date for the test, even better -- since otherwise you've got to be careful about running the test within a few seconds of midnight anyways.
- ks2048 4y agoThe example here is drawing a red rectangle, so OCR won’t do anything.
- jimjimjim 4y agoThis can get very difficult. Especially with pages that are more than just text and images. Lines, interactive content, optional layers, annotations, embedded content, blend mode transparencies. All of this and more make things complex. The real problem is that reading a pdf is vastly more complex than writing a pdf. The spec (1000+ pages) is open to interpretation and different readers interpret it differently. A page that might render perfectly in adobe may look different when viewed in firefox or chrome or ghostscript.
- flandish 4y agoIsn’t testing the physical generation of a pdf more aligned with “integration” test not unit testing? Testing the api that makes the pdf is ok, but testing like this post suggests, with bitwise comparison is integration testing, no?
- izacus 4y agoIs naming these tests a seriously useful thing to bikeshed on?
- mardifoufs 4y agoHaving well defined terms, and using them well, is essential to any type of engineering. I don't know why aiming for precise terminology is only controversial in software engineering.
- PaulStatezny 4y agoThere is a distinct and meaningful difference between unit tests and integration tests. flandish is not bikeshedding. Unit tests are about testing a single unit in isolation. Integration tests are about testing the integration of multiple units. With unit tests, the industry's general attitude is that there should be no side effects, such as reading/writing to databases or the disk. Side effects are generally embraced for integration tests, on the other hand. As a result, unit tests are mostly useful for "pure" functions, ones where the output is 100% derived from the input, regardless of any state external to the function. (Such as database records.) However, a large portion of the industry hasn't realized this and so you get millions of lines of dependency-injected unit tests that really don't provide much value in terms of catching actual bugs. (If these tests were integration tests, they'd catch actual bugs 10x more often.) A unit test for generating a PDF will not actually involve writing a PDF for disk. An integration test, however, might. So as I said, this isn't bikeshedding. ;-)
- leni536 4y agoWell, the blog post could have just called it "test", and nobody would bikeshed it.
- quectophoton 4y agoYeah the blog author accidentally triggered one of the fundamental laws of internet: "The best way to get the right answer on the internet is not to ask a question; it's to post the wrong answer." Also known as Poe's Law.
- gbro3n 4y agoThis is more of an integration test than a unit test. And if you're going to test for a pixel perfect image match, why not check for full equality with a pre-existing PDF file, byte for byte? And then what are you testing? That something has changed? You'd likely know that the output was going to change, so to fix the broken test you need to use the failure result to create the new comparison file, and if your always going to use the failure output as an input for correcting the test, what is the point. "Don't test that the code is like the code" is a similar principle.
- gbro3n 4y ago*you're
- DavidSJ 4y agoThis sort of tests can be useful when you change things under the hood in such a way that the output shouldn’t have changed.
- mattgreenrocks 4y agoThese are typically called smoke tests, and can be valuable for regression testing of third party libraries you depend on. An alternate approach: generate the PDF, then run it through a PDF reader library to scrape the text out and ensure it is there.
- izacus 4y agoYour approach will completely miss big changes like missing pictures, broken layout, missing background and other breakages in rendering. Also missing text which isn't embedded as a text layer.
- mattgreenrocks 4y agoOf course. It was meant for argument, not as an omnibus to comprehensively testing PDFs. :)
- systems_glitch 4y ago
- zubspace 4y agoWe do something similar, but from my experience small changes, like fonts or lines rendering a tad different after library changes can be quite frequent. Usually small changes you can't really see, only if you compare them as 2 layers in paint.net or something. Adding something like an error margin for all pixels or subsections sometimes makes sense, but this can be tricky. Downscaling the image and comparing grayscale values with a small error margin is another option. It all depends on how accurate your tests have to be.
- izacus 4y agoWell, but those changes are triggered by something aren't they? So when you upgrade your font lib or pdf rendering library, you're warned that you're now generating different output and can update the golden set. Your dependencies aren't changing without a cause are they?
- deleted 4y ago[deleted]
- zubspace 4y agoYeah sure, it just starts to be a problem when you're having dozens of tests failing because of small rendering changes which can be ignored. Someone still has to look at all the test output, compare it to the old state and update the tests with the new state. In our case this happened quite a lot. This is not an issue at first, but the more you use tests like this and the more people work with your code, false positives start to drag you down.
- deleted 4y ago[deleted]
- eddsh1994 4y agoIt's interesting how different peoples use of testing terminology is across teams/companies/professions. Vocab is standardized by various ISO's, ASQ, and ISTQB so we could all share the same language, then we don't have to debate about what integration/unit/smoke/component/regression/golden/snapshot testing means