5 ms·

I have been using this in a CI pipeline to maintain a business-critical PDF generation (healthcare) app (started circa 2010 I think), here is the RSpec helpers

I have been using this in a CI pipeline to maintain a business-critical PDF generation (healthcare) app (started circa 2010 I think), here is the RSpec helpers I'm using: https://gist.github.com/thbar/d1ce2afef68bf6089aeae8d9ddc05ddf https://gist.github.com/thbar/d1ce2afef68bf6089aeae8d9ddc05d... The code contains git-stored reference PDFs, and the test suite re-generate them and assert that nothing has changed. Helped a lot to audit visual changes, or PDF library upgrades!

pmarreck 2y ago
could you not just compare the source (or perhaps even the hash) of the PDF and assert on that?
- alexdoesh 2y ago
  Hashes can change regularly due to metadata. Source checks may also require some filtration or preprocessing before comparison. Visual comparison is the best option here, especially if you have a complex document with multiple third-party components that may change both the hash and source but keep the visual appearance the same.
  - thibaut_barrere 2y ago
    In this case, we indeed have multiple components (although not third-party), and being able to refactor those without risk is quite nice.
- jabroni_salad 2y ago
  If the document is a legally required disclosure (like a bank's fee schedule for example) then you need to grade that document directly rather than its source code. PDFs are horrible and there is a lot that can go wrong with making them between writing and publishing.
- ydant 2y ago
  I use some custom tools for PDF comparison (visual, textual, and perceptual hash) for my personal records/accounting purposes. A number of the financial and medical institutions I deal with re-generate PDFs every time you request them, but the content is 99-100% identical. Sometimes just a date changes. So I use a perceptual hash and content comparison to automate detecting truly new documents vs. ones that are only slightly changed.
- thibaut_barrere 2y ago
  the source sometimes changed for small internal reasons in the library generating the PDF (prawn). So just comparing the source would not give a clear cut answer. A visual comparison has helped quite nicely over time.
- knallfrosch 2y ago
  What should I do when the assertion fails – inspect the PDF with my sad little caveman eyeball?
  - ggrosskopf 2y ago
    In my own tests where I inspect PDF differences in python, I iterate through the pages, if the number of pages is the same, I convert each of them with PIL to bitmap, get the diff (ImageChops.difference is black for everything same and colored for diffs) and find the content of the diff with `getbbox`. This gives me the coordinates of the rectangle where changes appeared, I then use those to also print the page with a colored rectangle and print out the crops. I give out the original page, the original rectangle, the original page with colored rectangle, the new page and the new rectangle, the diff cropped and uncropped only after which I start using my caveman eyeballs I also pixelate it a bit and have a brightness cutoff for the diff to see if the diff actually matters and i also try if re-cropping a bit so shifting by a limited amount of pixels makes it look like an ignorable difference because everything just moved to the left a bit but that is optional. I also recommend exporting the new pdf from the CI/CD tool to be put back into the test as reference. Even between Linux distros and versions small changes in fonts and stuff like that make a difference
tylerflick 2y ago
Are you using singed digests in the PDFs?

pmarreck 2y ago

could you not just compare the source (or perhaps even the hash) of the PDF and assert on that?

alexdoesh 2y ago
Hashes can change regularly due to metadata. Source checks may also require some filtration or preprocessing before comparison. Visual comparison is the best option here, especially if you have a complex document with multiple third-party components that may change both the hash and source but keep the visual appearance the same.
- thibaut_barrere 2y ago
  In this case, we indeed have multiple components (although not third-party), and being able to refactor those without risk is quite nice.
jabroni_salad 2y ago
If the document is a legally required disclosure (like a bank's fee schedule for example) then you need to grade that document directly rather than its source code. PDFs are horrible and there is a lot that can go wrong with making them between writing and publishing.
ydant 2y ago
I use some custom tools for PDF comparison (visual, textual, and perceptual hash) for my personal records/accounting purposes. A number of the financial and medical institutions I deal with re-generate PDFs every time you request them, but the content is 99-100% identical. Sometimes just a date changes. So I use a perceptual hash and content comparison to automate detecting truly new documents vs. ones that are only slightly changed.
thibaut_barrere 2y ago
the source sometimes changed for small internal reasons in the library generating the PDF (prawn). So just comparing the source would not give a clear cut answer. A visual comparison has helped quite nicely over time.
knallfrosch 2y ago
What should I do when the assertion fails – inspect the PDF with my sad little caveman eyeball?
- ggrosskopf 2y ago
  In my own tests where I inspect PDF differences in python, I iterate through the pages, if the number of pages is the same, I convert each of them with PIL to bitmap, get the diff (ImageChops.difference is black for everything same and colored for diffs) and find the content of the diff with `getbbox`. This gives me the coordinates of the rectangle where changes appeared, I then use those to also print the page with a colored rectangle and print out the crops. I give out the original page, the original rectangle, the original page with colored rectangle, the new page and the new rectangle, the diff cropped and uncropped only after which I start using my caveman eyeballs I also pixelate it a bit and have a brightness cutoff for the diff to see if the diff actually matters and i also try if re-cropping a bit so shifting by a limited amount of pixels makes it look like an ignorable difference because everything just moved to the left a bit but that is optional. I also recommend exporting the new pdf from the CI/CD tool to be put back into the test as reference. Even between Linux distros and versions small changes in fonts and stuff like that make a difference

tylerflick 2y ago

Are you using singed digests in the PDFs?