18 ms·
The growing image-processor unpleasantness
- deleted 4y ago[deleted]
- sneak 4y ago> no SoC vendor is willing today to open their imaging algorithms. I know at least in rPI land this is not a matter of unwillingness but inability: the GPU boot blobs are protected under copyright law by upstream licenses that do not permit distribution of the source (which is not f/oss anyway). It’s like three vendors removed (GPU firmware author, SoC maker, board maker). I imagine the ISP situation is somewhat similar; Intel probably has licensed some of this code versus making all of it from scratch. The firmware industry is like an alternate reality where f/oss never happened, it seems. People guard their codez as jealously as the underground blackhat scene. Code libraries for implementing stuff in hardware (on FPGAs) is literally referred to as “IP”. The mind reels. Totally different culture over there.
- rasz 4y agoin the case of image processing it was "we paid a lot of money for it so you cant have it without drm" https://forums.raspberrypi.com/viewtopic.php?p=16989#p16951 https://forums.raspberrypi.com/viewtopic.php?p=16989#p16951 >The GPU contains a very sophisticated ISP - the pipeline of image processing to get some raw camera data to pretty JPG's. This pipeline also needs to be 'tuned' to the particular camera and this can take some times (months for a specialised team to do a really good job) >All the work above needs to be done for each new camera module - and will be done for any camera sold for the Raspi. Raspi cameras ship with drm https://gist.github.com/marcan/6dde73a9a0c917cd4fc9784a0a73efe3 https://gist.github.com/marcan/6dde73a9a0c917cd4fc9784a0a73e...
- cleverca22 4y agolibcamera also supports the rpi and moves all of that control loop stuff into linux userland, so you would think there are no secrets left but the actual register writes to drive the ISP are still in the blob, and libcamera is just telling the blob how to tweak every knob in the hardware it makes no sense why they have opened it up so much, yet still insist on keeping that last bit a secret
- techdragon 4y agoDon’t discount the nature of the business, a lot of the low level code may never get updated, or even be capable of updates, it’s burned into ROMs and the only thing that might keep their business afloat is the fact they squeezed a bit better performance out of otherwise common parts that one big factory is using to make webcams or other widgets for dozens of indivisible companies selling their own products. It’s unfortunate that they fear competition so greatly they refuse to even document their external interfaces because that might make it slightly easier for someone to work out how their system works, but I don’t fault their decision making.
- HidyBush 4y agoIt seems to me that what the FOSS world needs in order to succeed is an extreme push towards forming a class of programmers specialized in reverse engineering. We already have plenty of capable coders who have written so much useful software, but does is matter if we can't use all the hardware at our disposal? Our devices are filled to the brim with accelerators, sensors and assorted specialized hardware that we could be taking advantage of, but they are all buried behind proprietary drivers or just plain inaccessible firmware.
- keepquestioning 4y agoJudging by Asahi, FOSS will be fine. The real question is where the hell is the open source hardware? Engineers working for NVIDIA, or camera IP core specialists are not gods. Let's just make our own god damn camera and GPU.
- iforgotpassword 4y agoOn that note, I recently read that Asahi now has pretty much a finished driver for the M1 GPU. How did they pull that off that quickly? Was there extensive documentation available? Thanks to nouveau being mediocre at best and lagging 2 or 3 generations behind until it has usable support for a GPU, I always assumed writing a decent GPU driver through reverse engineering must be next to impossible. So what's up with Asahi? Were there just some gifted Wunderkind working on it? Is the architecture so much simpler? Is there more information available? Iirc it's based on powerVR, so not completely unknown, but then again the nouveau team got some hints from Nvidia in the past, and also they have been working on it for a really long time in comparison. Just really curious.
- mgdlbp 4y agohttps://news.ycombinator.com/item?id=25873887 https://news.ycombinator.com/item?id=25873887 (“Dissecting the Apple M1 GPU, Part II”)
- pabs3 4y agonouveau has the problem that they rely on nvidia signed firmware blobs for some functionality (like reclocking), and if nvidia don't want to give them those blobs, then they can't use the features the blobs enable. With nvidia's Linux kernel driver going open (but firmware and userspace staying proprietary), the firmware will be released separately and be redistributable, so nouveau will be able to use it.
- black_puppydog 4y agoOwning an XPS 13 2in1 from 2021, I get the impression that this issue started way before IPU6. At least my webcam never showed any signs of life, and the only thing I found about this is an ancient git repo [1] where other frustrated device owners shout their anger into the wind. I might be misinterpreting this, I'm not at all a uardware/driver person, if that's even a thing. Can anyone confirm this? [1]: https://github.com/intel/intel-camera-drivers/tree/master/drivers/media https://github.com/intel/intel-camera-drivers/tree/master/dr...
- AshamedCaptain 4y agoYes, the webcam situation in Linux has been degenerating for years by now, as many laptop models and specially everything that is an ultraportable or convertible has switched to IPU-based webcams. I have multiple laptops from 5 years ago where the webcams don't work and will likely never be able to work under Linux. Basically the main difference is that with IPU-based stuff the kernel must support not just a "webcam"-like device, but rather must contain support for all the individual components making up the webcam. Think not only actual raw sensor hardware, but even stuff such as controlling the shutter and voice coil motors directly. The situation is similar if not identical to a smartphone where the OS interfaces directly with the raw camera hardware instead of like in traditional computers where it was all abstracted in the form of a USB UVC webcam. In fact, some of the newer sensor drivers are adapted/stolen from Android kernels. Many, many laptop models come with IPU3 hardware, which is around 7 years old hardware at this point, and it is still mostly unsupported in Linux. In fact, practically _all_ the support in current Linux comes from the efforts of one guy who has been trying to make the Surface Pro line of devices work under Linux ( https://github.com/linux-surface/linux-surface/wiki/Camera-Support https://github.com/linux-surface/linux-surface/wiki/Camera-S... ). I have had some success with these patches (which are now mostly upstream), by manually writing/converting from Android kernels/HALs the required sensor drivers in my laptop, but the image quality was just garbage. I don't even know how much image processing is done software-side on Windows. Intel has published code, but not bothered to upstream it; Google, who is using this hardware in some ChromeOS devices, only cares about ChromeOS support which does things just too differently to Windows devices (e.g. basic things like enumerating which camera hardware is connected). None of the big OEMs is behaving nicely here. IPU4 appeared more recently and is basically entirely unsupported. There is zero upstream support and don't bother with the out-of-tree driver since it will be a waste of time (you also need the drivers for the specific hardware sensors used in your laptop + the "pipeline configuration"). IPU6 is probably just a newer iteration of this. I don't know if it will have a bit more developer/manufacturer oomph and thus more Linux support than previous iterations, but even if it does, it's highly likely that all existing laptops with IPU hardware will never have a working webcam in Linux.
- anewhnaccount2 4y agoReading between the lines here, the situation seems to be a little like the softmodem situation, which was also a major pain for Linux support. The abstraction previously given in hardware has been split into software/hardware to save on hardware costs, but now they refuse to open source the software. As for the provided user-space software, it's not clear why this is happening. As long as kernel modules keep to the allowed interfaces, (and not the GPL-only internal interfaces) they can be closed source. Is it a case of too much caution from the vendors (after all -- NVidia have closed source drivers)? Is this supposed to be more forward compatible? Is it possible to have a higher performance (zero copy?) user-space exposed abstraction for this kind of video processing?
- rwmj 4y agoIt exactly reminded me of softmodems, except just maybe here there's room for innovation. Apparently getting the raw sensor data allows you to do clever AI-based enhancements to the image. A bit different from softmodems because there you were always limited by what the other end of the phone line (ie. your ISP) would accept so you couldn't invent a better encoding system.
- mistrial9 4y agoquick add of onboard Unmanned Aerial Vehicle cameras and sensors, in the other direction. Being without precedent, there is no pattern of evolution in the consumer space as with soft-modems, but.. next-gen UAVs can benefit from onboard filters (and more?) however, the temptation to make proprietary, definitely black-box, processing chains is huge. Precisely for the purpose of manipulating the output or censoring the output, it turns out. Hard to deal with in the markets since first-mover and contract-wielding advantages, are not fake. unlike the photos and video being generated?
- giantrobot 4y agoThese are the inverse of soft modems though. Those were just a PHY layer with the software driver doing all the DSP work. Softmodems removed DSP hardware. Modern image sensors have more smarts than just a CMOS sensor.
- taffronaut 4y ago> Senozhatsky posted a pointer to a repository containing a CAM implementation, but that code is only enlightening in the vaguest terms. There is no documentation and no drivers actually implementing this API. It is highly abstracted; as Sakari Ailus put it: "I wouldn't have guessed this is an API for cameras if I hadn't been told so". He's not kidding. It's hilarious. It could be a driver for anything. https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/3668500/ https://chromium-review.googlesource.com/c/chromiumos/third_...
- phh 4y agoI thought it was a bit hyperbolic, but yeah, looking at headers (both userspace and kernel), I can't see anything remotely related to camera. I would have at least expected a "color"space somewhere (colorspace which could have included jpeg/heif/... as format)
- gambiting 4y ago"Just this week I have a friend who bought a brand new Alder Lake laptop who is desperately trying to install the latest available kernel and firmware seeing the HDMI port does not work, nor does the laptop go to sleep properly." Good to see that the good old "laptop doesn't go to sleep on linux" problem has never truly gone away.
- jeroenhd 4y agoTo be fair to Linux, sleep doesn't work reliably on Windows either. The new Intel sleep states just don't seem to be reliable enough with modern software, to the point of at least one laptop manufacturer recommending customers to not carry a laptop in sleep mode in a backpack. The HDMI issue is ever present. Intel has lately been dropping the ball significantly with their drivers, but at least it's not Nvidia this time, so that's a change!
- gambiting 4y agoI mean, yes, of course - I'm just making (a little bit of) fun out of the fact that "laptop sleeping reliably" has traditionally been a benchmark for how complete Linux support is for that specific laptop. But yes, I've had 4 Dell XPSes, all running windows, and all of them have had at least one episode of the laptop cooking itself inside a bag - I pull it out and it's literally burning to the touch, because some stupid event decided to wake it up, despite the lid being shut.
- maven29 4y agoI've had to switch to linux full-time on my latitude for precisely this reason. It sleeps perfectly and the battery life is a lot better on linux than on windows. The good thing about Dell is that they ship with linux on certain configurations, so linux installs are very robust with the caveat that you will not see the agressive fan curves or higher tdp limits that you can get on windows.
- moffkalast 4y agoYeah my 6 year old i7 laptop just shuts down completely when told to sleep, both in Windows and Ubuntu (always has since I bought it). It's definitely some kind of firmware thing that Intel is royally messing up on a grand scale.
- pabs3 4y agoDoes anyone know why the image processing code needs to remain proprietary? Is it violating patents or something?
- berkut 4y agoIt probably doesn't need to be in all cases, and for simple stuff like generic image processing (resizing/rotation), there's probably no "secret sauce" to worry about. You could maybe technically argue things like denoising algorithms / foreground/background detection might be more secret as there might be competitive advantages there somewhere...
- fffobar 4y agoThere has been a lot of progress in cameras for smartphones, for very good reasons. And I mean a lot, see the physical design of a modern smartphone lens for example. I would guess that laptop cameras got hit with extra protection because they're "too close" to what's considered core intellectual property for a camera-maker. But it's just a guess really.
- kkielhofner 4y agoI worked on a embedded hardware application using Linux and "cameras". We (like many people) were using sensors from OmniVision[0] and there was a dedicated contractor who's entire area of focus (and life's work) was getting photons from the sensor of the camera into something usable by software. In our case that was an unholy combination of vendor software, custom code derived from very obscure NDA-only datasheets, and a bunch of custom plugins for gstreamer that could actually make use of the sensor, interfaces, and hardware encoding silicon. It was one of the more eye-opening "WOW, so that's how the sausage is made" experiences of my life. [0] - https://www.ovt.com/ https://www.ovt.com/
- colonwqbang 4y agoPatents is often a concern. Maybe you think you're not violating any patents. But somebody with access to your code could find something that looks like a violation. Real or not, it can spell trouble for you. Your code delivery could also contain parts that are licensed from other companies, limiting your options with regards to licensing. One way to limit the problem is to put most of your "secret sauce" code onto one or more sub-processors embedded in the imaging hardware. The register bank of your hardware would only be accessible to these sub-processors, meaning that you don't have to publish your register definitions. Then you can deliver the sub-processor microcode as a binary blob (independent of the host OS) and let the kernel module itself be open-source. This approach is taken by at least some vendors.
- londons_explore 4y agoI see image processors as a stopgap measure that won't exist for long. Just like GPU's were first dedicated hardware that could only render triangles, but are now fully fledged parallel compute machines which, if asked to, can also render triangles. Image processors will be the same - and eventually the parallel compute abilities will be merged into a GPU. They'll be replaced with GPU's that can directly input the gigabits of data that comes from a CCD sensor straight into the GPU RAM, ready to have noise reduction, demosaicing, de-shake, stacking, de-warping, focus, white balance, etc. algorithms applied to it.
- Kon-Peki 4y agoUnfortunately, I think we are heading in the opposite direction. The Moore’s Law/Denard Scaling disconnect has created a growing need for dark silicon [1] on chips. Thus we are likely to see even more specialized components rather than fewer. The Apple M1/M2 is a perfect example. [1] https://en.wikipedia.org/wiki/Dark_silicon https://en.wikipedia.org/wiki/Dark_silicon
- marcosdumay 4y agoYes, but it makes much more sense to specialize the components by form than by function. And GPUs and image processors look a lot alike. The largest difference today is on the IO links, but it makes too much sense to link the camera directly to the DRM.
- Kon-Peki 4y agoThe theory is that for every node shrink, a greater percentage of your transistors have to be dark. If you use a component too frequently, it isn't dark enough. I'll trust your statement that GPUs and image processors look a lot alike (I am no expert by any means), but as long as there is a difference an image processor is going to be a candidate for inclusion on the chip. The designers have a transistor budget, thermal constraints, and a prioritized list of components they can put on the chip given available transistors. I'm sure that if they had better alternatives, they'd skip the image processor. But they didn't, which gives you some information on how Intel thinks about image processors.
- londons_explore 4y agoFYI, these image processors often also do autofocus. Sometimes, they don't even expose an API to the operating system to do focussing, so you can't even manually do focus in software if you want to - the best you can do is say which area of the image should be focussed, and the algorithm in the image processor will try to achieve it. Thats suboptimal, because there are lots of bits of research into better ways to do focus. Todays image processors typically just adjust back and forth till they find a frame with more high frequency components. It kinda works, but requires them to overshoot a bit and go back, and doesn't work well for things like mirrors or windows where they may focus on the window rather than whats behind it or vice versa. They also have to adjust the focus slowly because they need to get a new frame and process it to see that the focus needs adjusting further - despite the fact the coils in the camera can move the lenses in milliseconds if commanded to. Future image focus techniques do things like run machine learning over the image, and say "thats a car, and it looks about 25 yards away, so set focus straight to 25 yards".
- mort96 4y agoI'm sure these image processor manufacturers wouldn't mind if they could take their old hardware, implement a better autofocus algorithm, and sell you the same hardware but with the new algorithm as a new product which you have to buy.
- Georgelemental 4y agoWho buys a new internal webcam for their laptop, though?
- codewiz 4y agoThis url is a "free link" to subscriber-only LWN content, and shouldn't have been published on HN. Quoting from the MakeLink page: The "subscriber link" mechanism allows an LWN.net subscriber to generate a special URL for a subscription-only article. That URL can then be given to others, who will be able to access the article regardless of whether they are subscribed. This feature is made available as a service to LWN subscribers, and in the hope that they will use it to spread the word about their favorite LWN articles. If this feature is abused, it will hurt LWN's subscription revenues and defeat the whole point. Subscriber links may go away if that comes about.
- detaro 4y agohttps://news.ycombinator.com/item?id=1966033 https://news.ycombinator.com/item?id=1966033
- mardifoufs 4y agoHonestly in this case, it makes it more likely for me to subscribe. I wasn't aware of the quality of lwn articles before I started seeing them regularly on the HN frontpage, and I wouldn't have even thought about subscribing to a news source centered mostly around Linux before.
- tgflynn 4y agoIs there public documentation on these "image processors" that is sufficiently detailed to implement drivers/firmware for them ? It seems to be that making such documentation available should be the core demand on hardware manufacturers, not that they necessarily open source their own software/firmware.
- deleted 4y ago[deleted]
- sillystuff 4y ago> ...making such documentation available should be the core demand... not that they necessarily open source their own software... That would work only if companies could be compelled to release complete documentation. E.g., Microsoft even corrupted[1][2] the ISO standardization process to force through standardization of their OOXML (to take the wind out of worldwide efforts around standardization on the existing ISO standard, ODF) even though the OOXML Microsoft shipped _after_ standardization could not be implemented from the specification in the standard [2]. Forcing companies to provide source for a fully functional driver under a suitable license would prevent this sort of abuse from bad actors. [1] http://www.groklaw.net/article.php?story=20070312083134403 http://www.groklaw.net/article.php?story=20070312083134403 [2] https://www.garshol.priv.no/blog/154.html https://www.garshol.priv.no/blog/154.html
- tgflynn 4y agoI think it's fair to expect hardware companies to publish sufficient documentation so that independent software developers can write code to make use of their HW. Preferably that would be done through market pressure, but may at times need government action along antitrust lines. However if the nature of a device is such that a driver for it requires the implementation of non-trivial algorithms then I wouldn't support government action to force companies to open source such code. I think standardization processes are a rather different topic than hardware, but I'm wondering, did the Microsoft shenanigans you mentioned actually prevent people from developing compatible software ? My impression is that the compatibility issues that people had with MS file formats in the 90's have largely been resolved in practice. Is that not the case ?