10 ms·
Pixel Visual Core: Google’s first custom-designed co-processor
- sandGorgon 9y agoThis is pretty much Tensorflow on metal. Considering that tensorflow works pretty well on most mobile phone GPU, im betting this feature can be turned on for most phones if Google wants. I would love to see the apples to apples comparison of the Pixel tensorflow computation running on a vanilla snapdragon GPU vs this custom core. Also will be interesting to know why halide and not opencv.
- blt 9y agoHalide and OpenCV are not comparable. Halide is mainly about decoupling the description of a low-level image processing operation from implementation details regarding tiling, storage of intermediate results, etc. OpenCV is a library implementing high level algorithms like multi-view 3D reconstruction that Halide does not even attempt to address.
- sandGorgon 9y agoMy mistake, I phrased my question incorrectly. Why not opencv with a halide backend. The tensorflow whitepaper compares the TF ir to the halide ir. In fact, the work in tensorflow core is around XLA https://www.tensorflow.org/performance/xla/ https://www.tensorflow.org/performance/xla/ (which brings compatibility across various architecture) It seems that this is more an attempt at locking the tensors to target custom metal, so that it can't be copied and made to work on other phones. For examples, the Google Camera NX version is a port of the HDR+ feature on compatible phones.. Even the cheap Xiaomi. They actually give results comparable to Nexus. However, by locking the model to a very custom image processor (and custom IR) , this can't be ported. This looks to me as a version of walled garden more than a performance win.
- trishume 9y agoHalide is much more flexible. OpenCV is a fixed high level toolbox, combining things requires lots of intermediate buffers. Halide allows compiling arbitrary kernels in a fused way with extremely good performance. Sure you could implement OpenCV functions using Halide, but better to expose the lower level more flexible API. Source: I’ve rewritten parts of an OpenCV app in Halide for performance. Also your argument that it’s not about perf makes no sense to me. Sure other phones have HDR, I bet its way slower and more energy intensive than the Pixel 2. Except maybe the iPhone, which may also have custom silicon. If they had custom HDR tech in software they wanted to keep they could just keep the source closed, or patent it.
- sandGorgon 9y agoThat bet is the problem. Tensorflow on mobile already targets GPU and is going to be even more so with XLA. This debate is going to be hearsay unless we have an apples to apples comparison of the base tensorflow code running on the halide ir targeted to the tpu vs a vanilla ARM GPU... Even using opencv. I find it telling that they compared cpu vs tpu... Not GPU vs tpu.
- joshuamorton 9y agoI'm really confused why you keep bringing up opencv. What does opencv have to do with the (presumably) TF ML model running on the IPU that you want to benchmark against the TF ML model running on a GPU?
- sandGorgon 9y agoBecause halide is positioned as a high performance improvement over opencv. While I believe it is just an architectural decision to enable a non-portable ML model.
- joshuamorton 9y agoWhere did I mention halide? My question is why do you care about opencv when benchmarking neural network performance on various hardware?
- j_s 9y agoCurious if anyone here knows anything about this alternative for existing phones: Ask HN: What happened to tensorflow lite? | https://news.ycombinator.com/item?id=15494368 https://news.ycombinator.com/item?id=15494368 (Oct 2017, no comments) >zitterbewegung: They announced it at Google I/O in May See : https://techcrunch.com/2017/05/17/googles-tensorflow-lite-brings-machine-learning-to-android-devices/ https://techcrunch.com/2017/05/17/googles-tensorflow-lite-br... I can't find another announcement is this vaporware?
- londons_explore 9y agoIt became tensorflow slim: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim https://github.com/tensorflow/tensorflow/tree/master/tensorf...
- mholt 9y agoCool, but am I the only one who thinks the photos on the right (toward the bottom) are a little overexposed / bland / washed-out?
- Synaesthesia 9y agoThe point is they recovered the shadow detail of people’s faces which would otherwise be lost.
- justhackedme 9y agoWhat's the point in recovering something just to waste it again?
- jwarren 9y agoAs landscapes, they may well be less interesting. As communicative portraits, they're (IMO) more successful. The first one is probably the best example. Having a clearer and brighter picture of (for example) your partner and daughter would likely be preferable. If it was a candid and unposed shot, it might look weird and unnatural though. This is why the fourth example is probably the least successful, that hazy summer glow in the "before" picture is somewhat desirable. To my eyes, the shadow boost is a little too much. I'd ideally like a slider to customise exactly how much brightness you're adding. Personally, I'd probably turn most of these down a touch. Google clearly has fantastic technology driving this image processing, but I think they need a little more work on their taste.
- rndmio 9y agoIMO from a photography point of view the first is the worst displaying very clear haloing around the subjects, one of the clear signs of "not good" HDR. I guess it depends what exactly you're optimising for but the results aren't pleasing to my eye.
- dharma1 9y agoI would also like more controls. You can have that shooting RAW and editing with Snapseed, but it's a bit more involved and the usable dynamic range of a single mobile raw image is below exposure fusion from HDR+.
- ramshanker 9y agoIt looks like they are releasing these bits and pieces before the Finale. Custom ARM core, SOC in entirety.
- shrewduser 9y agoI'm not so sure... Have they bought or hired any taken like apple did?
- jsolson 9y agoYou're begging the question here. Apple's acquisition of PA Semi filled a void. You assume a similar void.
- jjrrmm 9y agoAbsence is the default for any ontology.
- Symmetry 9y agoThe sophistication of the team you need to plug together components into a workable SoC or design something like the TPU processor core is much less than what you'd need to design a top of the line OoO application core. They clearly do have a team that can put together SoCs like the original TPU and if they keep working at it I'm sure they'll be able to train up to doing that eventually but I'd expect a decade long learning process unless they bring on people who've done that sort of thing before.
- Aaargh20318 9y agoI'm not so sure about that. I don't doubt they have the skill to do it, but I doubt they can make the numbers work. Apple can make custom CPU cores because they sell a shitload of phones. Google does not.
- gehsty 9y agoMaybe they would license it to manufacturers - use our core design AND our operating system gratis, as long as you send users to our services? Googles ad revenue is so important that they should do absolutely anything and everything to protect it, spending billions on R&D to keep android at the forefront is essential, and at the moment they are only really competitive on the software side (A11 smokes latest snapdragon chips, as does the A10....), offering a custom chip would give android the vertical integration that Apple leverage to be so dominant.
- ttoinou 9y agoThe left/right comparison for HDR are pointless : they should show the same image with same contrast/exposure with the two techniques at 100% pixel. The difference between without HDR and with HDR should be less noise (otherwise if it's just about having more exposure in the darkest parts of the image, then a little bit of image processing can do the trick)
- justhackedme 9y agoYep. Here is a real image on the left, here's one that's been edited to an inch of it's life and looks nothing like the real capture. HDR is awesome, but so easily used for the wrong things. Makes a good shot turn into some weird half-baked fantasy view.
- alkonaut 9y agoHopefully the idea is that it looks at the scene, decides which exposures are needed to capture the DR (for example 1/100s ISO800 for the foreground and 1/200s at ISO100 for the background) and then quickly captures exactly those two exposures in rapid succession. Then the software would try to compensate movement between the frames and stack these exposures into one. So long as it involves multiple pictures and the inputs (exposure params)are computed are taken done when the photo is captured, then it does things you can't do with image processing in any single image.
- ttoinou 9y agoYes you can, there are multiple inputs BUT the effect is about extending dynamic range only (not create an artistic effect between the inputs). You can always extend numerically the DR of any set of values but you will stumble upon some noise eventually.. This HDR+ module is supposed to have less noise in the shadows OR have better sun effects / beautiful highlights (candles for example) / good looking clouds. A good test would be shooting in strong direct sunlight and look at the shadows.
- alkonaut 9y agoRight, the key "artistic" thing is identifying foreground/background, faces, interesting elements etc., and simply try to balance the exposure to create a pleasing end result. I have no idea what this thing does, or even if it uses multiple exposures (or even multiple cameras), but I was simply thinking that facial recognition/focus and automatic toning has been around for ages, it has to do something that requires fast decision (at trigger time) about exposure params. I couldn't see whether that was done though, it could be done with most of the same effect by simply exposing e.g. +2EV and -2EV and simply deciding afterwards how to blend those images into a HDR given the image content (faces, ...)
- dharma1 9y agoIn addition to still photos, would love to see exposure fusion (or HDR+ as Google calls it) in mobile video too. RED Epic and Scarlet cameras do this with 2 exposures per frame and call it HDRx, I've found it adds about 4 stops of dynamic range in grading. RED doesn't do any fancy anti-ghosting stuff though like HDR+. Magic Lantern firmware for Canon dSLR's also does something similar with dual ISO frames. This together with 10bit video profile for h265 (think phone hardware is capable now but haven't seen any phones with 10bit video yet) would make mobile video a whole lot better, right now it frankly sucks for anything with high contrast or dynamic range.
- agumonkey 9y agoAnyone feels it's tech for the sake of tech ? I don't even appreciate the HDR processed shades. There's a lot more detail that's true, but 1) it's fake IMO 2) pics don't have to be perfect, it's just a moment of life, the details are precise enough.
- sfifs 9y agoCan't see any pics on Firefox Focus browser on mobile. Are they using some weird Desktop only CSS?
- bearjaws 9y agoHDR+ is one of my favorite features from my 6p. I am routinely blown away by how well it performs at all lighting. Everyone i've shown pictures from low light scenarios is always impressed. HDR+ is a big reason I bought the Pixel 2, I know its going to keep getting better.
- amelius 9y ago> The camera on the new Pixel 2 is packed full of great hardware, software and machine learning (ML) Is that hardware only accessible by the camera? If so, isn't that wasteful?
- deleted 9y ago[deleted]
- lorenzhs 9y agoI think the following bit at the very bottom of the article suggests that it isn't: "HDR+ will be the first application to run on Pixel Visual Core. Notably, because Pixel Visual Core is programmable, we’re already preparing the next set of applications. The great thing is that as we port more machine learning and imaging applications to use Pixel Visual Core, Pixel 2 will continuously improve. So keep an eye out!"
- MBCook 9y agoIt seems odd to me that they’ve made this special part of the chip but it won’t be enabled until a future software update that will come out 1+ months after release. The basic idea of using special hardware makes sense. Apple has done the same thing, and as you get more and more data off the sensor this seems like the only way to do analysis/adjustment in real time without wasting a ton of power.
- jhallenworld 9y agoLooks like Movidius Myriad... I'm surprised Google would make their own chip for this.
- GuB-42 9y agoSo what is that IPU exactly? Lots of buzzwords but few actual details about the architecture. Is it like a integer GPU? The massive amount of ALUs suggests something like that. But later, I see "Notably, because Pixel Visual Core is programmable, we’re already preparing the next set of applications.". So, is it FPGA-like?
- lorrysomm 9y agoIt's a CPU designed to collect private user data and sell ads (and occasionally work with the NSA in Prysm-like programs).
- arca_vorago 9y agoYou got down voted but the way Google has been acting I wouldn't be surprised.
- sp332 9y agoIt's an Image Processing Unit. Architecturally I'd expect it to look like a GPU, maybe with something like the new matrix-multiply unit from their TPU. https://www.slideshare.net/AntoniosKatsarakis/tensor-processing-unit-tpu https://www.slideshare.net/AntoniosKatsarakis/tensor-process... Edit: wrong name
- godelski 9y agoSo I'm seeing this and another comment downvoted. Is that because you're wrong? If so, can one of the people downvoting (or someone else) expand and actually answer the original question?
- londons_explore 9y agoIt does not have a TPU matrix multiply unit. That unit is massive, and wouldn't fit in a small die like this. It seems likely it's optimized for the same kind of operations though, even if it doesn't have a matrix multiply unit. I would guess the main intended use for this silicon is running neural networks, even though the initial use case is for photos. Nearly all AI things on phones (voice recognition, google assistant's local features, keyboard predictive language model, offline translation, etc.) are severely compute limited, and could perform much better with this silicon. Some features are obvious candidates to put on-device, like realtime recognition of the contents of a photo, realtime wavenet voice synthesis, yet compute limitations preclude it.
- ggambetta 9y ago> HDR+ addresses this problem by taking a burst of shots with short exposure times, aligning them algorithmically, and replacing each pixel with the average color at that position across all the shots. This is almost exactly how you'd make super-resolution pictures (adding a nearest-point scaling up step at some point! Yet their super-resolution app (Photoscan) has some sort of artificial upper resolution limit. If they do plan to expose this capability to 3rd parties, I hope someone makes a proper scan/super-resolution app! Also tried Office Lens, also left disappointed :(
- TheAdamist 9y agoSuper resolution would want to utilize the OIS to artificially shift the lens around instead of cancelling movements. I dont think they are exposing anything like that.
- londons_explore 9y agoTurns out one doesn't have to. The users hand movements are at least one pixel as long as the phone is handheld. Just use natural movement to do superresolution. Everything you need is available in the camera2 RAW API. Using the users hand movements help cancel out various types of per-physical-pixel noise too.
- bitL 9y agoI wish my Nikon pro camera had a similar chip inside. Now we have two worlds - awesome pro cameras with incapable post-processing chips (I mean beyond RAW and initial processing) and horrible mobile cameras with outstanding post-processing chips.
- ggus 9y agoAlso the double lens, one for wide angle, one for tele, snapping at the same time and combining the images into one. It will be a looooong time until we see that on a pro camera with interchangeable lenses, if ever.
- ksk 9y agoWe already have zoom lenses that can do wide angle and take portrait shots. Having multiple full frame sensors with multiple lenses doesn't make much sense to me TBH. What is the problem you're trying to solve here?
- bradleyjg 9y agoIf you have burst mode and can get the raw frames, then shouldn't you be able to recreate the effect in post-processing? With the severe power constraints of a mobile phone a co-processor makes sense, but I'd think a general purpose desktop processor could easily handle this algorithm.
- londons_explore 9y agoCan you burst 80 frames @ 60fps of 12 bit 12 megapixel RAW and save all that output, together with the output of gyroscopes and accelerometers timed to the start and end of exposure of every row of the image? If you can, you might be able to postprocess it... But as far as I know, no DSLR cameras even have gyros, nor sufficiently high burst rates, so that kind of photography is out of the question. It comes out to over a gigabyte of data per image... and you can't compress it before processing...
- 9y ago
- simonh 9y agoIt seems odd to me that Samsung isn't doing things like this. After all they have both the chip design talent in house, they design their own Eynos chips, and even have the fabs in house as well. They're not as well regarded for software, but they do at least have some in house dev capability. A Samsung Galaxy with that sort of differentiation would, like this phone, make even a die hard iPhone user like me take serious notice. Or is Samsung actually right not to go down that path and the economics of Android handsets just don't support the significant extra spend and commensurate price bumps that sort of investment would require? The Pixels look like great devices, but without dramatically upping their distribution and marketing game is this really a vanity project for Google? I hope not, I'm crossing my fingers that with the talent they acquired from HTC, Google is really intent on making a splash in the market with the Pixel and not just in the tech press.
- ktta 9y agoI'm not sure if they have the software dev capability. Sure, they make nice UIs/apps on top of Android, but what about core computer science and domain specific skills? The new OS they tried to make was apparently full of kernel level bugs. Google has ML experts, compiler experts, etc. who probably contributed a lot to this. I guess it is a perk of keeping domain experts in the company who are there to advise when time comes.
- deleted 9y ago[deleted]
- wmf 9y agoSince half of Samsung phones use Snapdragon and half use Exynos, I've heard that Exynos is gimped to not exceed the features/performance of Snapdragon too much.
- jrs95 9y agoMaybe Google should start designing it's own CPUs/SoCs as well, since Qualcomm doesn't really seem capable of keeping up with what Apple is doing. Snapdragon 835 came out 6 months after the A10, and scored about 2/3 as well at single core, and less than 10% on Geekbench 4. A11 scored twice what the 835 did on single core, and over 50% more on multi core, and the 835 is going to be the best chip available in the U.S. for Android until sometime next year. Plus, you've got Samsung getting exclusive launch deals for the newest Snapdragon chips now, so other manufacturers are just now launching their flagship devices with hardware that is increasingly far behind what Apple is putting out. This gap could end up being as bad as Intel and AMD pre-Ryzen if something isn't done about it. Imagine what the high end laptop market would have looked like if only Macs had Intel chips!
- MBCook 9y agoI don’t think google has enough volume to make that worthwhile. Wouldn’t you need to sell tens of millions of chips for that to pay off?
- jrs95 9y agoIf they only used it in their own devices, it would probably be too expensive. If they sold those devices to other manufacturers, they could probably get enough volume to at least avoid taking big losses. It's less about making money from selling the chips, and more about keeping the hardware behind the Android ecosystem competitive.
- MBCook 9y agoI wonder if the other OEMs would buy it. I could see Samsung wanting to make their own or at least not strengthen their dependence on Google.
- simonh 9y agoThe problem is there is no magic wand the Android SOC manufacturers can wave to match Apple on chip performance. It's a multi-front war. Firstly yes Apple is well ahead in CPU core design and has been since 2013 when they brought out the first 64-bit ARM chip. And yes I know 64 bit by itself doesn't account for the performance increase, but the 64-bit ARM core architecture is much more streamlined and efficient than 32-bit which had accumulated all sorts of cruft over the years. The A7 wasn't just 64-bit, it was step change up in performance creating a gap that the other ARM vendors have never caught up with. That's not the whole story though, core design alone isn't the only reason Apple chips are faster, another is they have dramatically more cache memory on chip that other designs. The A10 had a total of 7MB of L2 and L3 cache, more than double the maximum configuration for the 835. Apple SOCs are massive, which makes them expensive. Finally, they can build in customised capabilities in hardware to support specific high level features, such as the secure enclave and the neural engine in A11 Bionic to support image post-processing effects and face recognition. This is clearly what Google is trying to do, but in an add-on chip because they're not (yet?) up to designing their own SOC. So it's not just the the Apple SOCs are faster, the reasons why they are faster are on several different axes. Some SOC vendors apparently say their profit margin lies in shaving off just one or two square millimetres of die size. Market forces like that would never produce a chip like the A11. So really this approach by Google of designing their own custom co-processor, and maybe eventually their own SOCs, could be the only way to close this gap.
- phn 9y agoEven with all the fancy ML stuff, the HDR "glow" is still clearly visible on most of these photos. On the plus side, the co-processor seems programmable, which means quality can improve without the need to upgrade the hardware.
- kbumsik 9y agoWow. It feels like the new era of custom-designed SoCs. I should've studied HDL like Verilog while I was in EE major in the school.
- without_words 9y agoIs the HDR+ stuff running on the PVC already, or is it still running on the application processor until they enable it? They say it uses Tensorflow, but is it a limited subset of Tensorflow designed only for image processing, like running a pretrained model on a TPU?
- tfolbrecht 9y agoConsider how inexpensive and high quality modern image sensors are. With this SoC, you can use so many amazing image processing algorithms. ML based object masking, deconvolution, focus stacking, exposure stacking, sensor fusion like FLIR, frame stitching multiple cameras, color compensating in camera flash. For serious heavy lifting you can send these bundles of images to the cloud for processing. All done without any more user input than tapping the take photo button. It's going to seem like magic to the user.
- tfolbrecht 9y agoA tag-a-long comment, this SoC + imaging system is going to make it much easier to write the software because you know all of the particulars of the optical system, sensor, position data, etc etc. It's not like implementing this stuff on the application layer which is nightmare mode difficult. Cameras are so fast, their controls are so fast, this is like some super human time dilation photography and photogrammetry. I've held out for so long, it's finally time to replace my HTC One m7.