9 ms·
Is JPEG 2000 a preservation risk? (2013)
- deleted 4y ago[deleted]
- boxerab 4y agoIt certainly was a risk in 2013, when the open source libraries were slow and riddled with bugs. Today, there are two stable, well maintained and fast open source libraries available: https://github.com/uclouvain/openjpeg https://github.com/uclouvain/openjpeg and https://github.com/GrokImageCompression/grok https://github.com/GrokImageCompression/grok JPEG 2000 is a niche codec, but in the niches that it occupies it is the gold standard, even after 20 years: 1. digital cinema (part of the digital cinema standard) and broadcast 2. medical imaging 3. satellite imagery For memory institutions, it still vies with Tiff as the top preservation codec.
- gwern 4y agoLooks like the answer was 'yes', in something of a self-fulfilling prophecy. When was the last time you used a JPEG 2000? https://caniuse.com/jpeg2000 https://caniuse.com/jpeg2000 puts browser adoption today at 18% (!).
- ewalk153 4y agoThere seems to be a functional and open reference implementation for JPEG 2000. https://github.com/uclouvain/openjpeg https://github.com/uclouvain/openjpeg
- simonh 4y agoThat is mentioned in the article, along with its deficiencies.
- greenknight 4y agoFun fact, whenever you go to the movies, films (that are projected digitally) are encoding in JPEG2000 (wrapped in a MXF in XYZ Colourspace). We also use it for our 3D renders for archviz, and is supported in our editing applications aswell... lower filesize compared to PNG, and better quality compared to JPG, and also supports alphas.
- dylan604 4y agoSadly, I was going to reply with "daily" because of this
- wyager 4y agoDo they really use straight up XYZ? Where can I read more about this format?
- KerrAvon 4y agoAppears to be this https://en.wikipedia.org/wiki/Digital_Cinema_Package https://en.wikipedia.org/wiki/Digital_Cinema_Package
- chrisseaton 4y agoI think people mis-estimate file sizes these days - two hours of completely uncompressed 4K video is under 5 TB. Fits on a basic consumer external hard drive no problem. Send it by mail. https://toolstud.io/video/filesize.php?width=3840&height=2160&framerate=24&timeduration=2&timeduration_unit=hours&compression=3000000&specificbitrate=100&specificbitrate_unit=1000000 https://toolstud.io/video/filesize.php?width=3840&height=216...
- matja 4y agoWell, two hard drives in RAID-0 if you want to play it back in real-time without copying it first to faster storage, because a single SATA drive cannot sustain 597.2MB/sec. Unless your definition of "basic consumer external hard drive" is an NVMe SSD over 10Gbps USB 3.2 or better.
- chrisseaton 4y agoAs the thread says - it’s compressed. I’m giving uncompressed sizes as an upper bound.
- giantrobot 4y ago
- prvc 4y agoAre they not common in pdf documents, for example those distributed by The Internet Archive? I suspect this is why so many of them render so slowly, even on fast systems.
- gwern 4y agoI don't think it's common. I've wondered for a long time why every time I try to process or edit an IA book scan - and only IA book scans - they come out bizarrely mangled and broken and inconsistent across viewers (and have mostly given up using them and scanning books myself), slow and surprisingly bulky compared to normal JBIG2-compressed JPEG PDFs. OP's observations about the unreliability and inconsistency of JPEG2k across tools would explain all of this...
- userbinator 4y agoYes. It's very noticeable when you're reading an ebook with J2K page images, just due to how long it takes each page to render. In contrast, regular JPEG of the same dimensions is practically instantaneous. PDFs are probably where the majority of users will encounter J2K.
- prvc 4y agoDo there currently exist tools for transcoding the images to something which renders faster?
- deleted 4y ago[deleted]
- snickerbockers 4y ago> OpenJPEG C 49,892 > libjasper C 26,458 > PIL C 12,493, Python: 9,229 But does PIL actually implement any of those image formats, or is it simply wrapping other libraries in the python/c api?
- mikewarot 4y agoI decided to use JPEG 2000 to save disk space on my photo archive a decade ago... huge mistake. [edit] I thought I had the originals backed up on CD, it turns out I didn't. DigiKam and many other programs don't support them natively. At some point I'm going to have to write a script and convert them all to normal jpegs.
- melissalobos 4y agoWhy was it a huge mistake? There are decent libraries that support it. Did it just not work with your cloud setup or OS libraries?
- simonh 4y agoCheck the article’s discussion of rendering bugs and implementation incompatibilities.
- melissalobos 4y agoSure those were encountered there, but at least in my experience those are easily avoided. I have had to make test suites for this, and in my experience if you use reasonable libraries things work well.
- alophawen 4y agoAre you suggesting the parent sets up test suites for a cloud solution and call it a day? How would that guarantee safe storage, given the previous discussion of all the broken implementations.
- mindslight 4y agoI was thinking about using JPEG2000 to archive scan masters, because I hate the look of original JPEG artifacts. In the end I decided to go with FLIF to eliminate the "preservation risk" - by being lossless, I can run the output through sha256 to make sure I've gotten all the original bits back. In the event that I regret this decision, I can simply transcode and not feel too bad (as opposed to transcoding from lossless and losing even more quality).
- cjbgkagh 4y agoAFAIK The fast J2K libraries are closed source and expensive so you have to worry about licensing a 3rd party component which is a PITA.
- simondotau 4y agoI think a distinction needs to be made between “risk” and “potential inconvenience”. Given that there are free, open source implementations of the JPEG 2000 decoder, There is no plausible preservation risk. Even if, in 150 years, the contemporary version of ImageMagick has dropped support for JPEG 2000, it would be relatively trivial to spin up an ancient virtual machine and run a 100-year-old version of Linux. Or, you know, pay an intern to port some old code to whatever modern language they’re using then. Rust, probably.
- simonh 4y agoThe article addresses this. The open source implementations are not complete so there are compliant files they cannot render. Also they have rendering bugs which mean some content does not render properly, and worse some files they generate render fine now, but are not compliant and if the code is fixed those portions of the files may stop rendering correctly. Some of these issues only manifest at particular zoom levels, so you are never be sure what you are seeing is correct. This us all part of the problem with the spec being so complex.
- usrn 4y agoSoftware rots man. It can be really hard to run some open source software from the 80s and 90s. This is actually one of the big arguments for Open Source: Access to the software is way less interesting than the support/community/ecosystem around it; there's no reason not to open source your software and not doing it actually harms the users.
- dagmx 4y agoYeah, Pixar literally had to spin up VMs of their old software stack when they had to remaster the Toy Story films because it was less work than getting it running again on their current stack
- simondotau 4y agoActually, it seems to me VMs are an ideal way to preserve things like that. Perhaps it should be standard procedure for the archivists of such big films to build and preserve a fully functioning, self-sufficient rendering environment in the form of a virtual machine.
- aaron695 4y agoThere is no such thing as preservation risk for digital formats. This has been studied. Do whatever. The only thing that matters is publishing it in the first place and backing up on hardware that's not at preservation risk. This is great example of fake science (Is it just "science" now?) keep the underclass constantly anxious about things that don't exist so they keep paying academics wages to do nothing but supply fear and disruption.
- Double_a_92 4y agoDid you even read the article? What's the point of perfectly backing up a file on whatever hardware, if the format of that file becomes obsolete and hard to read in the future?
- lajt 4y agoWe're building an iPad reader app that uses Apple's PDFKit, but we encountered JPEG2000s in the PDFs of the client's library, and PDFKit doesn't render them correctly. The frustrating thing is we're stuck with the issue since it's such a niche case that I don't think Apple will ever fix it. There is a 3rd-party commercial library PSPDF for iOS that renders it right, but I don't think they want to pay for it.
- jerryjerryjerry 4y agoDoes it still use wavelet transform?
- philovivero 4y agoLike many others, I don't think I will bother talking about JPEG2000 as a preservation risk. I guess it is, given the bugs and difficult implementation. But for me, the problem is just how unwieldy the files are. Nothing seems to work with them well, except ImageMagick, to convert the files to some more-usable format. I do a lot of Blender renders, and for a while I thought I'd use 12-bit JPEG2000 so that I could capture more details in the shadows for subsequent photo editing. Big mistake. I ended up having to write a series of very complicated scripts to convert all to regular JPEG for 99% of whatever I'm doing with the images. GIMP doesn't work well with them. My own image viewer, based on Kivy, doesn't work at all with them. MacOS generates thumbnails on them at maybe 1/10th the speed of comparable JPEGs. In the end I just decided if I really want to work with the darks in an image I will go find the original .blend and re-render the scene. Now I render straight to regular JPEG and I save massive amounts of time.
- klodolph 4y agoThe arithmetic coder in JPEG2000 is often the culprit for slow encoding / decoding. GIMP doesn't work well high-precision files, although it's gotten better. Photoshop can deal with these, even very old versions of Photoshop. Use OpenEXR or HDR output formats and do the tone mapping as a separate step. Also note that plain JPEG supports 12-bit in the first place.
- Nition 4y agoThe Wikipedia page for JPEG 2000, for the past 16 years, has used the same sample image[0] which arguably makes JPEG 2000 look worse than JPEG at the same file size. I wonder how many people that one image alone has put off. [0]https://upload.wikimedia.org/wikipedia/commons/5/51/JPEG_JFIF_and_2000_Comparison.png https://upload.wikimedia.org/wikipedia/commons/5/51/JPEG_JFI...
- faragon 4y agoJPEG 2000 is used a lot in raster maps (satellite images, with jp2/ecw/sid containers)