11 ms·

We asked camera companies why their RAW formats are all different and confusing

• vr46 2y ago
  I did push for all my digital images to be DNG, and they are, up to around 2018, and two out of four cameras use it natively - Pentax, Leica - while the other two use their own formats - Canon, Fuji. The reason I’m less fussy now is because the combination of edits, metadata and image data in a single file didn’t necessarily help me when I switched from Lightroom to Capture One. I would love to be able to update the files to use newer RAW processors and better IQ, but I lose the Lightroom edit information in C1. That makes sense as they do things differently. But I hoped that with DNG there was a universal format for handling edits. My JPEGs remain the definitive version of the images but I would love to be able to recover all those original edits again in C1, or any other editing program.
• Scaevolus 2y ago
  Ultimately, RAW formats aren't that complex, and camera firmware is mostly developed in countries that don't have strong open source software traditions. Look at the decoders for each format that darktable supports here: https://github.com/darktable-org/rawspeed/tree/develop/src/librawspeed/decoders https://github.com/darktable-org/rawspeed/tree/develop/src/l... It's some binary parsing, reading metadata, maybe doing some decompression-- a thousand lines of C++ on average for each format. These aren't complex codecs like HEVC and only reach JPEG complexity by embedding them as thumbnails! Cameras absolutely could emit DNG instead, but that would require more development friction: coordination (with Adobe), potentially a language barrier, and potentially making it harder to do experimental features. Photographers rarely care, so it doesn't appreciably impact sales. Raw processing software packages have generally good support available soon after new cameras are released.
  • buildbot 2y ago
    I think that might be why a lot of camera makers don't care to use DNG - it's easier to make their own format and easy enough for others to reverse engineer it. One thing that open source libraries do tend to miss is that very important extra metadata - for example, Phase One IIQ files have an embedded sensor profile or full on black frame that is not yet encoded into the raw data like it typically is for a NEF or DNG from many cameras. It does seem rawspeed handles this from a quick scan of the code. It can get more tricky - Sinar digital backs have an extra dark frame file (and flat frame!) that is not part of the RAW files, and that is not handled by any open source library to my knowledge - though I did write a basic converter myself to handle it: https://github.com/mgolub2/iatodng_rs https://github.com/mgolub2/iatodng_rs I'm not sure how DNG would be able to handle having both a dark and flat frame without resorting to applying them to the raw data and saving only the processed (still unbayered) data.
    • zokier 2y ago
      Can't you just throw such frames into additional IFDs or SubIFDs?
      • buildbot 2y ago
        Maybe? I’m not familiar enough with DNG to say - possibly that wasn’t a thing when Phase One first started using IIQ? I doubt it was around when Sinar was - in fact the last two (esprit 65 & S30|45 ) Sinar backs do use DNG as an option!
    • Aaargh20318 2y ago
      > One thing that open source libraries do tend to miss is that very important extra metadata - for example, Phase One IIQ files have an embedded sensor profile or full on black frame that is not yet encoded into the raw data like it typically is for a NEF or DNG from many cameras. In astronomy/astrophotography the FITS format[1] is commonly used, which supports all these things and is, as the name suggests, extremely flexible. I wonder why it never caught on in regular photography. 1: https://en.wikipedia.org/wiki/FITS https://en.wikipedia.org/wiki/FITS
      • buildbot 2y ago
        Oh interesting! This seems like it would be a good fit ;) Especially for really old setups that had RGB color wheels and multiple exposures, exactly like a multispectral astro image might. Phase one also has a multispectral capture system for cultural heritage, which just shoots individual IIQs to my knowledge… It would work great too for multiple pixel shift shots. Possibly, the engineers just didn’t know about it when they were asked to write the firmware? It’s funny, I think most RAW formats are just weird TIFFs to some degree, so why not use this instead.
        • davidkwast 2y ago
          Yes. TIFF would "fit the bil" here. It deals with multspectral satellite images. It supports 32 and 64 bits floats and 16bits integers.
  • spookie 2y ago
    These formats aren't complex because they really are supposed to be raw (-: But yeah, it would be preferable to have them use the digital negative (DNG) format, but why bother when the community makes the work for them? Reminds me of how Bethesda does things.
    • formerly_proven 2y ago
      Traditional Nikon NEF is pretty simple. It's just a tiff. Lossy compression is just gamma-encoding with a LUT (stored in the file). I think most traditional raws are principally similar. More complex compression schemes like ticoraw are fairly recent. What's complex is the metadata. All the cameras have different AF, WB and exposure systems.
      • spookie 2y ago
        yeah metadata really is a mess
  • actionfromafar 2y ago
    The contents are simple. How to interpret the contents, is not simple. That is why you see internet advice advocating for keeping old raw files around, because Lightroom and Photoshop sometimes gets updates which can cram out better results from old raw files. (Edit: I mean, if you want to get a basic debayered RGB image from a raw, that's not too hard. But if you want to cram out the most, there are a lot of devils in a lot of details. Things like estimating how many green pixels are not actually green, but light-spill from what should have been red pixels is just the beginning.)
    • mxfh 2y ago
      Yet that's processing level stuff, not format stuff. Even unlikely that the manufacturer made the best possible result from the sensor input as is.
  • rickdeckard 2y ago
    > Cameras absolutely could emit DNG instead, but that would require more development friction: coordination (with Adobe), [..] Technically speaking, implementing DNG would be another development activity on top of a RAW export, because RAW also has a purpose in development and tuning of the camera and its firmware. It is supposed to be raw data from the sensor with some additional metrics streamed in, just sufficiently standardized to be used in the camera-vendors' toolchain for development. It just "happens" to be also available to select for the end-user after product-launch. Supporting DNG would mean adding an extra feature and then hiding the RAW-option again. I can imagine it's hard to make this a priority in a project plan, since most of the objectives are already achieved by saving in RAW
    • Narretz 2y ago
      > It just "happens" to be also available to select for the end-user after product-launch RAW (any format) is an essential requirement for many photographers. You just can't get the same results out of a jpeg.
      • rickdeckard 2y ago
        None of this is disputed (or relevant) in this conversation
        • mort96 2y ago
          I disagree. Bufferoverflow frames raw formats as something that's really only there for R&D purposes, and it's more or less just an afterthought that it's available to photographers. In reality, Narretz points out, getting access to the raw sensor data is a key feature to many photographers; it's an essential aspect of the product from a user perspective.
          • rickdeckard 2y ago
            Since you disagree: where in this thread did anyone state the opposite of what you just wrote, who said that RAW is NOT a key feature to many photographers?
  • weinzierl 2y ago
    I always thought camera RAW formats were optimize continuous shooting rates. About being able to linearly write an image as fast as possible. I don't know the details of DNG but even the slightest complication could be a no-go for some manufacturers.
    • gbin 2y ago
      This was my guess too, get the raw bayer data from the sensor in one go + some metadata. Then as the sensors and cameras evolve they are just accumulating different formats?
    • m000 2y ago
      I believe this might have been the case in the past, where (a) sensor resolutions were lower - so the raw images less bulky, (b) camera CPUs were slower - so you would like to take them out of the equation. These days, the bottleneck for achieving continuous shooting rate is probably writting to the sd card (which is the standard for the consumer/pro-sumer models).
    • octacat 2y ago
      It is always written into a memory buffer first, which could be like 256 megabytes... it tooks time to fill it up, once it is filled, memory card speed becomes a bottleneck. So, actually, writing only jpegs would trigger the slowdown later, so you could take more frames before the buffer fills up
    • tmoravec 2y ago
      The bottleneck is usually in SD card write speeds, however. Sport photographers often skip raw and only use JPG because the files are smaller and as a result, one can take more photos in one burst.
      • tomatocracy 2y ago
        For raw at high frame rates, high end cameras don't use SD cards but things like CFexpress which absolutely can keep up (and there are also various compressed RAW formats these days which apply a degree of lossy compression to reduce file size). As I understand it, the reason some professional sports photographers don't shoot RAW (or it's less important) is more because they are in an environment where publishing quickly is important, so upload speeds matter and there isn't really much time to postprocess.
  • auxym 2y ago
    It took a long time for Canon CR3 raw format to be supported by darktable because, although the format itself had been reverse engineered, there was a fear from the developers that it was covered by a patent and that they risked a lawsuit by integrating it in DT. IIRC, they had attempted to contact Cabon legal to obtain some sort of waiver, without success. I'm fact I'm not sure how that saga ended and CR3 support was finally added a few years after the release of the Canon mirrorless cameras that output CR3.
  • rdtsc 2y ago
    > Cameras absolutely could emit DNG instead, but that would require more development friction: coordination (with Adobe), potentially a language barrier, and potentially making it harder to do experimental features. I am a weirdo and always liked and used Pentax (now Ricoh) they do support the DNG format.
    • remlov 2y ago
      Pentax/Ricoh really is a hidden gem. I love my "dinosaur" K1 Mark II and my GR IIIx goes EVERYWHERE I go.
  • bufferoverflow 2y ago
    Why would you need coordination with Adobe? Their software already reads DNG files just fine.
  • dllu 2y ago
    Fujifilm lossy compressed raw still isn't supported after many years [1]. [1] https://github.com/darktable-org/rawspeed/issues/366 https://github.com/darktable-org/rawspeed/issues/366 And in my experience there has been lots of bugs with Fujifilm raws in darktable: [2] https://github.com/darktable-org/rawspeed/issues/354 https://github.com/darktable-org/rawspeed/issues/354 [3] https://github.com/darktable-org/darktable/issues/18073 https://github.com/darktable-org/darktable/issues/18073 However, Fujifilm lossless compressed raw actually does a decent job keeping the file sizes down (about 50% to 60% the file size of uncompressed) while maintaining decent write speed during burst shooting.
    • zahlman 2y ago
      It's really strange to me that a lossy compressed format could be called "raw". Does that just mean that it hasn't been e.g. gamma-corrected before the compression was applied? (Is it even a good idea to do lossy compression before such correction?)
      • 4ad 2y ago
        All raw means is scene-referred data. The idea that somehow raw means "raw" data from the sensor is an often repeated idea, but unfortunately is completely nonsense. Modern sensors do on-chip noise reduction, they can be programmed to give data in all kind of formats and with different processing done to it. The same sensor used in different cameras can have different ISO. The same sensor used in different cameras can produce different RAW files even at the same ISO. Not just in the sense of a different file format, in the sense of different data in the file, from the exact same sensor, but programmed differently.
        • jiggunjer 2y ago
          Source? There are lossless and lossy transformations. In many scientific contexts raw implies no lossy transformations in terms of information.
          • danudey 2y ago
  • sandofsky 2y ago
    > Cameras absolutely could emit DNG instead, but that would require more development friction: coordination (with Adobe), potentially a language barrier, and potentially making it harder to do experimental features. I think this is being too generous. DNG is just an offshoot of TIFF. Having written a basic DNG parser having never read up on TIFFs before, it really isn’t that hard. As far as experimental features, there’s room in the spec for injecting your own stuff, similar to MakerNote in EXIF if I recall. If you are planning to do experimental stuff, I’d say what Apple pulled off with ProRAW is the most innovative thing that a camera manufacturer has done in forever. They worked with Adobe to get it into the spec. All of these camera manufacturers have similar working relationships with Adobe, so there’s really no excuse. And if you can’t wait that long, again, MakerNote it. In my opinion, custom RAW formats are a case study in “Not Invented Here” syndrome.
  • Syzygies 2y ago
    When my father bought his first digital camera, he insisted on raw format access. Inexplicably I didn't understand at the time why he (Bryce Bayer) wanted this. He was modest about his work. I do now!
    • dx4100 2y ago
      What an awesome dad! Lived during the golden age of photography and retired before its demise.
  • danudey 2y ago
    > Cameras absolutely could emit DNG instead, but that would require more development friction: coordination (with Adobe), potentially a language barrier, and potentially making it harder to do experimental features I've worked with medical imaging systems from the largest imaging companies in the world -- GE, Siemens, etc. -- all of which use a standardized image format/protocol/etc. called DICOM. DICOM has standardized fields for the vast majority of information you would need to record for medical imaging - patient ID, study ID, image # if it's an image sequence, etc. - as well as metadata about where it came from, like the vendor ID of the machine that did the scan (the CT scanner, MRI, X-ray, etc). There are also arbitrary fields for vendor-specific information that doesn't have a defined field in the specification. All of these fields have clear purposes and definitions and all are available to every DICOM reader/writer, and yet the company I worked for had a huge table of re-mappings because some scanners, for some reason, would put the patient ID in the vendor field, or the vendor ID in the scanner name field, and so on. There's no reason for this, there's no complication that might cause this; it's all standard fields that everything supports. These are manufacturers who, while using the standard that everyone else uses, deliberately screw things up in ways that their own hardware and software can silently compensate for but which other vendors then have to work around in order to inter-operate. In other words cameras absolutely could emit DNG instead, but aside from the arguments that you've made, I have every confidence that manufacturers would fuck it up on purpose just to make it harder for other vendors' software to inter-operate, which would mean that instead of software having to explicitly support e.g. Canon's RAW format, and being able to say "we don't yet support this new format", software would "support" DNG but it would be completely broken for some random cameras because the software developer hasn't had the chance to implement idiotic workarounds for these specific broken images yet.
• stefantalpalaru 2y ago
  [dead]
• yjftsjthsd-h 2y ago
  This isn't really my area, so I'm probably wrong... I'd always assumed that RAW files were, well, raw data straight off the sensor (or as close as possible)? In which case, you could standardize the container format, but I wouldn't think it was possible to have a standard format for the actual image data. Would appreciate if anyone could correct me (a quick skim of wikipedia didn't clear it up)
  • buildbot 2y ago
    Typically they are not to my knowledge! Though I am also not an expert. Most camera makers apply a fixed sensor profile, and possibly a dark frame to remove noise before writing out the values to whatever file. Some of them may apply lens optimizations to correct distortion or vignetting as well.
    • mubou 2y ago
      On top of that, I hear the RAW format on some smartphones is saved after the phone does its fake-HDR and computational photography bullshit, so it's even further from "raw" with those cameras.
      • kalleboo 2y ago
        This is true for Apple's "ProRAW" that you get if you choose RAW in the iOS camera app. Third party camera apps like Photon can shoot actual RAW raw though, the hardware and OS APIs do support it.
    • tobyhinloopen 2y ago
      The corrections are just metadata, the RAW data is still there. This is true for both DNG and ARW (Sony). Dont know the other brands. The corrections can even look different based on what program you use to interpret them.
      • buildbot 2y ago
        I don’t think that’s true in general. As a sibling comments points out, this is not true for some DNGs - for example, the output of an iPhone is in DNG, but with many, many transforms already baked in. A DNG might even be debayered already. GFX 100s II’s apply a transform to RAW data at iso 80, see: https://blog.kasson.com/gfx-100-ii/the-reason-for-the-gfz-100-ii-iso-80-unusually-good-read-noise/ https://blog.kasson.com/gfx-100-ii/the-reason-for-the-gfz-10... I don’t know much about ARW, but I do know that they offer a lossy compressed format - so it’s not just straight off the sensor integer values in that case either.
• lotyrin 2y ago
  While I generally prefer compatibility and shared standards, camera RAW formats seem to be a reasonable place to lean into implementation details if needed in order to gain performance and ease/quality of implementation over interoperability.
• schobi 2y ago
  When building a camera, you decide once and then most parameters stay fixed. It would be trivial to just append 1000 bytes for a mostly fixed DNG header to each image. But how do you test this? While the DNG specification is open source, the implementation was/is(?) not. Do I really need a copy of Photoshop to test if my files are good? How would I find good headers to put into my files? what values are even used in processing? Maybe the situation has changed, but in the old days when I was building cameras there was only a closed-source Adobe library for working with DNGs. That scared me off.
  • Cadwhisker 2y ago
    For testing, there's the Adobe DNG SDK: https://helpx.adobe.com/camera-raw/digital-negative.html https://helpx.adobe.com/camera-raw/digital-negative.html You'll find the whole spec there, too. I think the source is also available somewhere.
  • goeiedaggoeie 2y ago
    Things like camera intrinsics and extrinsics are not fixed. 1000 bytes seems small to me given the amount of processing in modern cameras to create a raw image. I could easily imagine storing more information like focus point, other potential focus points with weights as part of the image for easier user on device editing.
  • seba_dos1 2y ago
    The camera app on my GNU/Linux phone stores DNGs with no troubles using FLOSS only.
• v1ne 2y ago
  I think the article misses the point: It's not about how complex the data structures are, it's about the result, in all its details. Comparing different RAW converters (Lightroom, DXO), their image rendering is slightly different. If you compare the colors with the JPEG image, even more so. If the goal is to faithfully reproduce the colors as they were shown in the camera, you depend on the manufacturer's knowledge. To me, it makes not sense to have some "open" DNG format in the middle, when it's flanked by proprietary processing. It's not about the format, it's about knowing the details, including parameters, of the image processing pipeline to get a certain look.
• kazinator 2y ago
  > Photo editing software needs to specifically support not just each manufacturer’s file type but also make changes for each new camera that shoots it You mean, your proprietary, closed-source Photo-editing sofware? Why can't the vendors of that shit just make a library that they all share ...
  • _ph_ 2y ago
    The ironic part is, that basically all the closed-source photo-editing software (and of course all open-source) are just using the open source LibRaw. Any special features as color profiles comes on top of that. So yes, the camera manufacturers could just donate to LibRaw or just use DNG instead.
• t0bia_s 2y ago
  Compressed RAF (Fujifilm, 24MP) ~20 MB DNG (24MP) ~90 MB It cost about 4 times more to store RAW files in DNG format.
  • 4ad 2y ago
    No it doesn't. A DNG is just a container that can hold anything, including compressed data, just like the RAF. Coincidentally, most proprietary RAW formats are just bastardized TIFFs, and DNG is also a TIFF derivative... There is zero technical reason not to use DNG. Leica and Pentax use it just fine.
    • t0bia_s 2y ago
      Please explain how can I convert compressed RAF to DNG with same size.
      • immibis 2y ago
        It's a very common mistake in free software to not design a system end-to-end. Free software people will design some kind of generic container format and go "look! you can put anything in the container!", declare the job done, and then not write tools which actually do put anything in the container, or tools which can make use of anything in the container other than one specific thing. (See: ActivityPub)
        • 4ad 2y ago
          DNG has nothing to do with free software, it's an Adobe format similar to PDF. It is an open standard, but it is covered by patents and it comes with a no-cost patent license (except that it's less open than PDF because it doesn't define the interpretation for Adobe-specific tags (which ACR/Lightroom uses), not that that matters for raw file in any way). Of course that even Adobe DNG converter can do what the GP asked for (I just tried it[1]), not that I would recommend it for Fuji files. And not that it matters anyway, since the whole point is producing DNG files directly, not converting them. Edit: on my Fuji X-T5 files, using mosaiced data with lossless JPEG-XL compression (supported by MacOS 14+, iOS 17+, and the latest Lightroom/ACR): edengate:1$ ls -l total 163664 -rwx------@ 1 aram staff 40894672 Apr 4 15:25 DSCF1483.RAF -rw-r--r--@ 1 aram staff 42894224 Apr 7 16:51 DSCF1483.dng [1] https://llum.chat/?sl=3MCDl4 https://llum.chat/?sl=3MCDl4
• aleph_minus_one 2y ago
  > Cameras absolutely could emit DNG instead, but that would require more development friction: coordination (with Adobe), potentially a language barrier, and potentially making it harder to do experimental features. Why no simply provide documentation of the camera-specific format that is used? EDIT: This should have been an answer to https://news.ycombinator.com/item?id=43584261 https://news.ycombinator.com/item?id=43584261
• DeathArrow 2y ago
  Because most users of the raw format are photographers and they don't care about it?
  • jedisct1 2y ago
    Depends. When a new camera is out, it often takes time before Lightroom, Capture One, etc. support it. While I don't care about how the raw format works, being able to keep using my usual software even with a brand new camera is something I care a lot about.
• kookamamie 2y ago
  DSLRs have just dropped off the wagon a long time ago, when it comes to software and especially meaningful UX innovation. As an anecdote, I have a Sony a7r and operating it via its mobile app is one of the worst user experiences I have had in a while. Same goes to the surrounding ecosystem of software. E.g. Adobe's Lightroom is full of obsolete paradigms and weird usability choises.
  • Mashimo 2y ago
    Yes, same for my Sony A6000 and A6400. I just wanted to take some selfies and it's exhausting to use the remote app.
    • kookamamie 2y ago
      Yes, it's mind-bogglingly difficult and complicated.
      • to11mtm 2y ago
        At the time of release of the A6000, having a mobile app to take a picture (and not having to buy an adapter like some other brands) was cool enough that you could deal with the (at the time, relatively minor) jank. Biggest thing is they never really improved the mobile apps... and in some cases IMO they got worse.
  • wongarsu 2y ago
    Most hardware companies are just terrible at software in general. Camera makers are pretty average in that regard. Usability of the camera hardware and software ecosystem is another matter. I think the common wisdom is that most paying users don't want beginner-friendly, they want powerful and familiar. So everything emulates the paradigms of what came before. DSLRs try to provide an interface that would be familiar to someone used to a 50 year old SLR camera, and Lightroom tries to emulate a physical darkroom. Being somewhat hostile to the uninitiated might even be seen as a feature.
    • 2y ago
• Cadwhisker 2y ago
  '.dng's share the same file format structure as '.tiff's, but they have some extra fields. For stills photography, Adobe's '.dng' format does fairly well, from 8-bit to 16-bit. It copes with any of the 4 possible standard RGGB Bayer phases and and has some level of colour look-up table in the metadata. Sometime this is not enough for a camera's special features and The Verge's article covers those reasons quite well. For video, things get much more complicated. You start to hit bandwidth limits of the storage media (SD cards at the low end). '.dng' files were not meant to be compressed but Blackmagic Design figured out how to do it (lightly) and still remain compatible with standard '.dng' decoding software. Other, better compressed formats were also needed to get around the limits of '.dng' compression. Red cameras used a version of JPEG 2000 on each Bayer phase individually (4 of them), but they wrapped it in patents and litigated hard against anyone who dared to make a RAW format for any video recording over 4k. Beautifully torn apart in this video: https://www.youtube.com/watch?v=IJ_uo-x7Dc0 https://www.youtube.com/watch?v=IJ_uo-x7Dc0 So, for quite a few years, video camera companies tip-toed around this aggressive patent with their own proprietary formats, and this is another reason why there's so many (not mentioned by The Verge). There's also the headache of copying a folder of 1,000+ '.dng' stills that make up a movie clip; it takes forever, compared to a single concatenated file. So, there's another group of RAW video file formats that solve this by recording into a single file which is a huge improvement.
• ChrisMarshallNY 2y ago
  Raw decoding is not as simple as you might think. It’s the best place to add “signature steps.” Things like noise reduction, chromatic aberration correction, and one-step HDR processing. I used to work for a camera manufacturer, and our Raw decoder was an extremely intense pipeline step. It was treated as one of the biggest secrets in the company. Third-party deinterlacers could not exactly match ours, although they could get very good results.
  • koiueo 2y ago
    Can you share what company have you worked for?
    • ChrisMarshallNY 2y ago
      Not publicly. It’s not difficult to figure out, but I make it a point, not to post stuff that would show up in their search algorithms. But it was a pretty major one, and I ran their host image pipeline software team. [Edited to Add] It was one of the “no comment” companies. They won’t discuss their Raw format in detail, and neither will I, even though it has been many years, since I left that company, and it’s likely that my knowledge is dated.
      • Zak 2y ago
        > They won’t discuss their Raw format in detail Can you share the reason for that? It seems to me that long ago, camera companies thought they would charge money for their proprietary conversion software. It has been obvious for nearly as long that nobody is going to pay for it, and delayed compatibility with the software people actually want to use will only slow down sales of new models. With that reasoning long-dead, is there some other competitive advantage they perceive to keeping details of the raw format secret?
        • ChrisMarshallNY 2y ago
          The main reason is that image Quality is the main coefficient of their corporation. They felt that it was a competitive advantage, and sort of a "secret ingredient," like you will hear from master chefs. They feel that their images have a "corporate fingerprint," and are always concerned that images not get out, that don't demonstrate that. This often resulted in difficulty, getting sample images. Also, for things like chromatic aberration correction, you could add metadata that describes the lens that took the picture, and use that to inform the correction algorithm. In many cases, a lens that displays chromatic aberration is an embarrassment. It's one of those "dirty little secrets," that camera manufacturers don't want to admit exists. As they started producing cheaper lenses, with less glass, they would get more ChrAb, and they didn't want people to see that. Raw files are where you can compensate for that, with the least impact on image quality. You can have ChrAb correction, applied after the demosaic, but it will be "lossy." If you can apply it before, you can minimize data loss. Same with noise reduction. Many folks here, would absolutely freak, if they saw the complexity of our deBayer filter. It was a pretty massive bit of code.
• octacat 2y ago
  "Sony’s software for processing ARW RAW files is called Imaging Edge. Like most first-party software from camera manufacturers, it’s terrible and unintuitive to use — and should be saved for situations like a high-resolution multishot mode where it’s the only method to use a camera’s proprietary feature." I think the primarily reason is that they have great hardware developers and terrible software developers. So, having ARWs it is maximum they could provide to photographer, so they could take the files and run away from sony as soon as possible (i.e. do the rest in the better software). Pentax could save DNGs, there are zero reasons for other companies not to do the same.
• octacat 2y ago
  By the way (hello, adobe). DNG compression compresses worse than 7zip, haha. Too much for an open standard, too hard to add LZ4 as an option in the times of having 24 core setups :D
• _aavaa_ 2y ago
  DNG is easy to adopt so long as your images look like a Bayer image, and you don’t need any special treatment of the raw data. Sigma’s camera’s are notorious for their lack of support in most editors because their Foveon files require extra steps and adjustments that don’t fit the paradigm assumed by DNGs (and they claim it would release proprietary information if they used dngs). The bigger issue is that at the end of the day the dng format is very broad (but not broad enough) and you rely on the editor to implement it correctly (and completely). DNGs that you can open in one of the major editors will simply not open in another.
  • uamgeoalsk 2y ago
    Sigma currently doesn't produce any Foveon cameras - the fp, fp L and BF all use DNG.
    • _aavaa_ 2y ago
      Sure, and? My statement stands. And more to the point, for their foveon cameras that produced with both x3f and dng files, the image quality from their dng files are objectively and substantially worse than the x3f files.
• gwbas1c 2y ago
  > If a manufacturer comes up with additional data that isn’t included in the DNG standard, the format is extensible enough that a camera manufacturer can throw it in there, anyway. It sounds like DNG has so much variation that applications would still need to support different features from different manufacturers. I'm not sure it (DNG) will really solve interoperability problems. This issue smells like someone is accidentally playing politics without realizing it. Kind of reminds me of the interoperability fallacy with XML. Just because my application and your application use XML, it doesn't mean that our applications are interoperable. I suspect that a better approach would be a "RAW lite" format that supports a very narrow set of very common features; but otherwise let camera manufacturers keep their RAW files as they see fit.
  • AntonyGarand 2y ago
    Seems like DNG does behave like the RAW lite format you've just described: Everything common would be stored within the base DNG file, while everything "advanced" / more specific to a camera would be stored in additional metadata properties, which do not need to be parsed to still be able to process the base image. You can add support for these metadata on a case-by-case basis without breaking the original format, so you're not stuck re-implementing your whole raw parsing when a new camera is released as the base subset of DNG would still work.
  • sandofsky 2y ago
    Hey. I’m the guy quoted. RAW is ultimately about sensor readings. As a developer, you just want to get things from there into a linear, known color space (XYZ in the DNG spec). So from that perspective, interoperability isn’t the issue. How you process that data is another matter. Handling a traditional bayer pattern vs a quad-bayer vs Fujifilm’s x-trans pattern obviously requires different algorithms, but that’s all moot given DNG is just a container.
• lizknope 2y ago
  Every camera has a unique RAW format even cameras from the same company. The article briefly mentions this but doesn't go into that much detail. I've got at least 10 Nikon cameras going back to 2005 and every "NEF" Nikon RAW file is different so if you buy your camera on the first day it is released you have to wait for your software vendor to add support or shoot in JPEG format. There have been a few times when the RAW files are so similar that you can use a hex or EXIF editor and change the camera model EXIF field to an older supported camera and load the file. But in theory the RAW converter has been profiled for each specific camera using ICC color targets and stuff like that.
  • Zak 2y ago
    > But in theory the RAW converter has been profiled for each specific camera using ICC color targets and stuff like that. In practice too, if consistent results are desired. The format being identical doesn't mean the values the sensor captures under the same conditions will be identical, so a color-calibrated workflow could produce wrong results. It would be nice to have a setting for "treat camera Y like camera X (here there be dragons)" though. I've had to do something similar with the Lensfun database to get lens corrections working on Mk. II of a lens where Mk. I was supported, but a GUI would be nice. A prompt to guess the substitution automatically would be even nicer.
• frognumber 2y ago
  This is on a long list of why camera companies are dying. There is a long list of issues like this which have prevented ecosystems from forming around cameras, in the way they have around Android or iOS. It's like the proprietary phones predating the iPhone. The irony is that phones are gradually passing dedicated cameras in increasing numbers of respects as cameras are now in a death spiral. Low volumes means less R&D. Less R&D and no ecosystem means low volumes. It also all translates into high prices. The time to do this was about a decade ago. Apps, open formats, open USB protocols, open wifi / bluetooth protocols, and semi-open firmware (with a few proprietary blobs for color processing, likely) would have led things down a very different trajectory. Sony is still selling cameras from 2018: https://electronics.sony.com/imaging/interchangeable-lens-cameras/full-frame/p/ilce7m3-b https://electronics.sony.com/imaging/interchangeable-lens-ca... The price new fell by just 10% over the 7 years ($2000 -> $1800). And in a lot of conditions, my Android phone takes better photos, by virtue of more advanced technology. I have tens of thousands of dollars of camera equipment -- mostly more than a decade old -- and there just haven't been advancements warranting an upgrade. A modern camera will be maybe 30% better than a 2012-era one in terms of image quality, and otherwise, will have slightly more megapixels, somewhat better autofocus, and obviously be much smaller by the loss of a mirror. Video improved too. The quote of the day is: "I wish it weren’t like this, but ultimately, it’s mostly fine. At least, for now. As long as the camera brands continue to work closely with companies like Adobe, we can likely trudge along just fine with this status quo." No. We can't. The market has imploded. The roof is literally falling in and everyone says things are "fine." Does any know how much volume there would be if cameras could be used in manufacturing processes for machine vision, on robots / drones, in self-driving cars, on building for security, as webcams for video conferencing, for remote education, and everywhere else imaging is exploding? No. No one does, because they were never given the chance.
  • tristor 2y ago
    > And in a lot of conditions, my Android phone takes better photos, by virtue of more advanced technology. > I have tens of thousands of dollars of camera equipment -- mostly more than a decade old -- and there just haven't been advancements warranting an upgrade. A modern camera will be maybe 30% better than a 2012-era one in terms of image quality, and otherwise, will have slightly more megapixels, somewhat better autofocus, and obviously be much smaller by the loss of a mirror. Video improved too. I thought the same thing, and then I went and rented a Nikon Z8 to try out over a weekend and I was blown away by the "somewhat better autofocus". As someone who used to travel with a Pelican case full of camera gear, to just carrying an iPhone, I'm back to packing camera gear because I'm able to do things like capture tack-sharp birds in flight like I'm taking snapshots from the hip thanks to the massive increase in compute power and autofocus algorithms. "Subject Eye Detection AF" is a game-changer, and while phones do it, they don't have enough optical performance in their tiny sensors/lenses to do it at the necessary precision and speed to resolve things on fast-moving subjects. In terms of IQ, weight, and all that, it's definitely not a huge difference. I would say it's better, but not so much that I particularly cared coming from a 12-year old DSLR. But the new AF absolutely shocked me with how good it is. It completely changed my outlook. I say this, not to take away from your overall point, however, which is that a phone is good enough for almost everyone about 90% of the time. It's good enough that even though I upgraded my gear, I only bought one body when I traded in two, because my phone can handle short-focal length / landscape just fine, I don't need my Z8 for that. But a phone doesn't get anywhere close to what I can do with a 300mm or longer focal length lens on the Z8 with fast moving subjects.
• PaulHoule 2y ago
  Don't know what's confusing about it... I mean, I shoot ARW with my Sony, these work fine with Lightroom and work fine with DxO PhotoLab [1] at least as long as my ARWs are not compressed (it's not that the compression is proprietary, it's that the compression is lossy and breaks denoising) [1] Shoot ISO 12,800, process with DxO, people will think you shot at ISO 200; makes shooting sports indoor look easy, see https://bsky.app/profile/up-8.bsky.social/post/3lkc45d3xcs2x https://bsky.app/profile/up-8.bsky.social/post/3lkc45d3xcs2x so I got zero nostalgia for film.
  • AntonyGarand 2y ago
    Proprietary formats require 3rd party developers to adapt their tools: While most mainstream software will be updated to support most/all cameras, this makes it harder for smaller projects to do. If they used an open standard, the advanced features could still require additional work to be compatible (ex: If they store custom metadata), but you could normalize everything that's shared, ensuring the core capabilities will never break for a new camera with its updated proprietary RAW like it currently does.
• FollowingTheDao 2y ago
  What? They did not say it was Capitalism and greed? I am shocked! They are just trying to lock people in to their format and make them dependent on the company instead of an open source and universal format.
  • sbuk 2y ago
    How? Support for RAW formats is reasonably complete. I can hop between different editors without much, if any, hassle at all. Since getting my first DSLR in the early 00's, I have used (in no order) Photos, Bibble, Lightroom, Aperture, Capture One, Photoshop, Pixelmator, Photomator, Darkroom, On1, Raw Power, Nitro Photo, Luminar, Darktable and RawTherapee, all without fuss. Where is the lock in?
  • gnarlynarwhal42 2y ago
    The company literally makes the hardware... You choose which company you prefer and generally stick within that ecosystem. Almost everybody uses third party software to process the images anyway, so we aren't really "locked in" If anything maybe you'd have a point if you said they should open up the specs of the mount and lens/body communication, but the RAW format really just has near-zero impact in the real world
• WalterBright 2y ago
  Reminds me of all the weird and confusing object file formats. The specs for them are inevitably confusing, wrong and incomplete.
• thatcks 2y ago
  One problem is that you cannot have a universal format that is both truly raw and doesn't embed camera specific information. Camera sensors from different companies (and different generations) don't have the same color (or if you prefer, spectral) responses with both their Bayer filter layer and the underlying physical sensor. If you have truly raw numbers, you need the specific spectral response information to interpret them; if you don't need spectral response information, you don't actually have truly raw numbers. People very much want raw numbers for various reasons, and also camera companies are not really enthused about disclosing the spectral response characteristics of their sensors (although people obviously reverse engineer them anyway).
  • fracus 2y ago
    What does RAW really mean then? Couldn't they simply redefine what RAW means to create a standard that can include proprietary technology? Like why not define it as including a spectral response?
    • thatcks 2y ago
      There is no 'RAW' format as such. In practice, 'RAW' is a jargon term for "camera specific format with basically raw sensor readings and various additional information". Typically the various RAW formats don't embed the spectral information, just a camera model identifier, because why waste space on stuff the camera makers already know and will put in their (usually maker specific) processing software. (Eg Nikon's format is 'NEF', Canon's is 'CR3', and so on, named after the file extensions.) I don't know if DNG can contain (optional) spectral response information, but camera makers were traditionally not enthused about sharing such information, or for that matter other information they put in their various raw formats. Nikon famously 'encrypted' some NEF information at one point (which was promptly broken by third party tools).
  • sandofsky 2y ago
    > Camera sensors from different companies (and different generations) don't have the same color (or if you prefer, spectral) responses with both their Bayer filter layer and the underlying physical sensor This is all accommodated for in the DNG spec. The camera manufacturers specify the necessary matrix transforms to get into the XYZ colorspace, along with a linearization table. If they really think the spectral sensitivity is some valuable IP, they are delusional. It should take one Macbeth chart, a spreadsheet, and one afternoon to reverse engineer this stuff. Given that third party libraries have figured this stuff out, seems they have failed while only making things more difficult for users.
• mannyv 2y ago
  This article is just as useless as asking "why are all your file formats different and confusing?" I mean, it's just binary data, right? Why can't they just write all their ones and zeros the same way?
  • omoikane 2y ago
    I thought the main value of this article was that they went out and asked various vendors that question ("why proprietary formats?") and put all their answers in one place. Too bad Nikon and Fujifilm didn't respond, but I can imagine their motivations being similar to other vendors.
• momojo 2y ago
  This reminds me of a current problem facing the bio-imaging community; many microscopes, many formats. My company specifically deals with one of the post-processing steps and we've had to build our own 'universal adapter'. Its frustrating because it feels like microscope companies are just re-inventing the wheel instead of using some common standard. There has been an effort to standardize how these TB size datasets are distributed[1]. A different but still interesting issue. [1] https://ngff.openmicroscopy.org/ https://ngff.openmicroscopy.org/
  • DrFalkyn 2y ago
    There’s DICOM but last I worked with it it was a disaster. , because everyone implemted it differently. Seemed like a solution looking for a problem. Also didn’t help the standard is like 200 pages and it was a design by committee, none of whom appeared actually interested in coming up with a reference implementation. I never understood why everyone didn’t just use TIFF
• csense 2y ago
  This is confusing. A 1920x1080 24-bit RAW image is a file of exactly 6,220,800 bytes. There are only a few possible permutations of parameters: Which of the 4 corners comes first, whether the row-major or column-major order, what order the 3 colors are in (RGB or BGR), and whether the colors are stored as planes or not. (Without planes, a pixel's R, G and B bytes are adjacent. With planes, you essentially have three parallel monochrome images, i.e. cat r.raw g.raw b.raw > rgb.raw) [1] What the article is describing sounds like something that's not a raw file, but a full image format with a header. [1] One may ask, how does the receiving software know the file is 1920 x 1080 and not, say, 3840 x 540? Or for that matter, a grayscale image of size 5760 x 1080? The answer is that, with no header, you have to supply that information yourself when importing the image. (E.g. you have to manually type it into a text entry field in your image editor's file import UI.)
  • porphyra 2y ago
    > what order the 3 colors are in (RGB or BGR) Camera raw files typically come in a raw bayer mosaic so each pixel has only one colour.
  • MagerValp 2y ago
    > This is confusing. We’ll, yes. You’re thinking of the classic RAW format that was just a plain array of RGB pixels without a header. When talking about digital cameras RAW refers to a collection of vendor specific file formats for capturing raw sensor data, together with a bunch of metadata.
• seanp2k2 2y ago
  I can't wait for DJI to launch their mirrorless video-focused dSLR and smash everyone out there. I just hope that the tariffs are fixed before then so that it doesn't end up costing double MSRP here, which would absolutely kill the appeal of it.
• butlike 2y ago
  So if they use the open format, they aren't able to offer 'quality' upgrades only available if they hack their own hardware? So if they are forced to use the open DNG format, the cameras are in parity now?
• insane_dreamer 2y ago
  tl;dr: vendor lock-in ("best" results for Sony camera RAW achieved with Sony preprocessing sw, etc.)
• leecarraher 2y ago
  Is raw a defined format? Or is it just whatever is most performant for the camera to write to storage, while also allowing for conversation to standard formats. Most are likely similar, running same or similar architectures, but in and of itself there is no raw format spec that must be adhered to. So if one way is more amenable to the system architecture, then that wins.
  • deathanatos 2y ago
    > Is raw a defined format? It's not /dev/urandom written to disk, no. Yes, a raw format has a structure. There's not one "RAW" format though (and TFA notes this): e.g., my Canon's RAW format specifically referred to is called "CR3". And its predecessor was "CR2", so even within a manufacturer there are multiple such formats. All undocumented. But a Pentax won't write out CR3s, it'll write out some other, yet bespoke format. I've vaguely reverse engineered some of CR3: it is a container that contains multiple copies of the photo taken; IIRC it contains a thumbnail JPEG, the JPEG, and the raw data itself. I doubt it's the most performant to write to storage: the format is vaguely TLV (its fairly similar to RIFF, if you understand the RIFF format), so it can't really be streamed to storage due to needing to know the lengths of the containing chunks (all the way out to the outermost chunk).
    • leecarraher 2y ago
      By defined format I of course don't mean random, or that it doesn't have structure, you have to be able to read it back otherwise what would be the point. I mean when you write out jpeg it has to adhere to a standard otherwise it isnt jpeg. Granted the jpeg format has evolved and has been augmented over the years. But it still is standard across iterations and devices.
• ziofill 2y ago
  Ha. My first digital camera was a Pentax kx from 2010. I didn’t know about RAW standards and I started using DNG by pure luck. BTW I loved that camera: it had the image stabilizer built into the sensor, which meant I could use amazing k-mount lenses from the 80s and still use image stabilization (unlike my roommate who had a Canon).
• hexo 2y ago
  Simple, as it is from Adobe, I wouldn't touch it as dev and rather make my own.
• vanderZwan 2y ago
  So as someone who once converted an entire archive of RAW files to DNG: the resulting DNGs were always larger, and much, much slower. Perhaps because they're TIFFs under the hood, which in my experience are also bloated and slow to load. I always wondered if the problem was a lack of interest in writing fast decoders for DNGs, or if this was inherent to the format.
• ksec 2y ago
  Interesting not a single mention yet. Some companies are considering using JPEGXL Lossless as RAW.