22 ms·

PixelNN – Example-Based Image Synthesis

497 points by pentestercrab 9y ago

Annella 9y ago
Thanks for sharing! Very interesting!
otp124 9y ago
I used to roll my eyes at crime television shows, whenever they said "Enhance" for a low quality image. Now it seems the possibility of that becoming realistic are increasing with a steady clip, based on this paper and other enhancement techniques I've seen posted here.
- ACow_Adonis 9y ago
  Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". You fundamentally can't get back information that has been destroyed/or never captured in the first place. What you can do is fill in the gaps/information with plausible values. I don't know whether this sounds like I'm splitting hairs, but it's really important that the general public not think we're extracting information in these procedures, we're interpolating or projecting information that is not there. Very useful for artificially generating skins for each shoe on a shoe rack in a computer game or simulation, potentially disastrous if the general public starts to think it's applicable to security camera footage or admissible as evidence...
  - jopsen 9y ago
    Sometimes US justice system seems very "approximate". So why not convict people based on interpolated evidence? - I'm joking of course :) hehe
    - WillReplyfFood 9y ago
      You seem like a funny guy- the batmaNN interpolates that you are very likely a joker.
    - dahart 9y ago
      Sadly, it actually happens sometimes. https://www.wired.com/2017/04/courts-using-ai-sentence-criminals-must-stop-now/ https://www.wired.com/2017/04/courts-using-ai-sentence-crimi... This thread a year ago worried about it too, but the paper itself seems implausible and problematic. https://news.ycombinator.com/item?id=12983827 https://news.ycombinator.com/item?id=12983827
  - ZeroGravitas 9y ago
    To give specific examples from their test data, it added stubble to people who didn't have stubble, gave them a different shape of glasses, changed the color of cats, changed the color and brand of sport shoe. And even then, I'm a little suspicious of how close some of the images got to original without being given color information. It appears that info was either hidden in the original in a way not apparent to humans or was implicit in their data set in some way that would make it fail on photos of people with different skin tones.
    - omtinez 9y ago
      I haven't read the paper in full detail, but reading between the lines I'm guessing that there's a significant portion of manual processing and hand waving involved. From the abstract, emphasis mine: > the second stage uses a pixel-wise nearest neighbor method to map the smoothed output to multiple high-quality, high-frequency outputs in a controllable manner. My interpretation is that they select training data by hand and generate a bunch of outputs. Repeating the process until they like the final result. From the paper: > we allow a user to have an arbitrarily-fine level of control through on-the-fly editing of the exemplar set (E.g., “resynthesize an image using the eye from this image and the nose from that one”).
      - IshKebab 9y ago
        Could be pretty great for police sketch artists. (Although pretty misleading for juries too.)
        adrianN 9y ago
        Just train the model with the suspect's Facebook photostream and presto you have convincing evidence.
      - WhitneyLand 9y ago
        There's nothing weak or negative about that, it's exactly what'd you expect. Obviously for a given input there will be multiple plausible outputs. With any such system it would make sense to allow some control in choosing among the outputs.
  - gambiting 9y ago
    No, but think of these blurred images as a "hash" - in an ideal situation, you only have one value that encodes to a certain hash value, right? So If you are given a hash X you technically can work out that it was derived from value Y - you're not getting back information that was lost - in a way it was merely encoded into the blurred image, and it should be possible to produce a real image which, when blurred, will match what you have. Don't get me wrong, I think we're still far far far off situation where we can get those reliably, but I can see how you could get the actual face out of a blurred image.
    - ComputerGuru 9y ago
      > you only have one value that encodes to a certain hash value, right? Errr wrong. A perfect hash, yes. But they're never perfect. You have a collision domain and you hope that you don't have enough inputs to trigger a birthday paradox. Look at the pictures on the article. It's an outline of the shoe. That's your hash. ANY shoe with that general outline resolves to that same hash. If your input is objects found in the Oxford English Dictionary, you'll have low collisions. An elephant doesn't hash to that outline. But if your inputs is the Kohl's catalog, you'll have an unacceptable collision rate. Hashes are attempts at creating a _truncated_ "unique" representation of an input. They throw away data they hope isn't necessary to uniquely identify between possible inputs (bits). A perfect hash for all possible 32 bit values is 32 bits. You can't even have a collision free 31 bit hash. So back to the blurry security camera footage of a license plate or a face. Sure, that "hash" can reliably tell you that it wasn't a sasquatch that committed the robbery, but it literally doesn't contain the data necessary to _ever_ prove it was the suspect in question, even if the techs _can_ prove that the suspect hashes to the image in the footage.
      - consp 9y ago
        Fortunately there is a limit: the universe (in a practical sense). You cannot encode all states it has in a hash as it would require more states than you want to encode as you already mentioned (pigeon hole). But representing macroscopic data like text (or basically anything bigger than atomic scale) uniquely can be done with 128+ bits. Double that and you are likely safe for collisions, assuming the method you use is uniform and not biased to some input. If you want ease collision examples you can take a look at people using CRC32 as hashes/digests. It is notoriously prone to collisions (since only 32 bits).
  - usrusr 9y ago
    Yeah, replace the training set with cartoon characters and the crime show dialog goes like this: "Zoom! Enhance! Zoom! Enhance! Enhance! Oh my god it's full of Smurfs..."
    - deleted 9y ago
      [deleted]
  - scarface74 9y ago
    What you can do though, in limited circumstances, is create a still picture with more detail from a lower quality video. https://photo.stackexchange.com/questions/17098/csi-image-resolution-enhance-how-real-is-it https://photo.stackexchange.com/questions/17098/csi-image-re...
    - murkle 9y ago
      It's a well-known technique in astronomy, eg https://www.aanda.org/articles/aa/ps/2005/22/aa2320-04.ps.gz https://www.aanda.org/articles/aa/ps/2005/22/aa2320-04.ps.gz
      - mcbits 9y ago
        For anyone put off by the .ps.gz, it's actually just a normal web page that links to the full article in HTML and PDF. Not sure what they were thinking with that URL. I almost didn't bother to look. (Maybe that's what they were thinking?)
    - dahart 9y ago
      I seem to remember from my computer vision class way back when that there's a fundamental theoretical limit to the amount of detail you can get out of a moving sequence. Recovering frequencies a little higher than the pixel sampling is definitely possible, but I feel like it was maybe something like 10x theoretical maximum. I also get the feeling, from looking around at available software, that in practice achieving 2-3x is the most you can get in ideal conditions, and most video is far from ideal.
  - throwaway613834 9y ago
    > I don't know whether this sounds like I'm splitting hairs Somewhat no, but somewhat yes. Thing is, while there can be lots of input images that generate the same output, it could be that only one (or a handful) of them would occur in reality. If this happens to sometimes be the case, and if you could somehow guarantee this was the case in some particular scenario, it could very well make sense to admit it as evidence. Of course, the issue is that figuring this out may not be possible...
  - sweezyjeezy 9y ago
    > Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". I would argue that this is a form of enhancement though, and in some cases will be enough to completely reconstruct the original image. For example, if I give you a scanned PDF, and you know for a fact that it was size 12 black Ariel text on a white background, this can feasibly let you reconstruct the original image perfectly. The 'prior' that has been encoded by the model from the large amount of other images increases the mutual information between grainy image and high-res. The catch is that uncertainty cannot be removed entirely, and you need to know that the target image comes from roughly the same distribution as the training set. But knowing this gives you information that is not encoded in the pixels themselves, so you can't necessarily argue that some enhancement is impossible. For example with celebrity images, if the model is able to figure out who is in the picture, this massively decreases the set of plausible outputs.
    - trevyn 9y ago
      > The catch is that you need to know that the target image comes from roughly the same distribution as the training set. When humans think about "enhance", they imagine extracting subtle details that were not obvious from the original, which implies that they know very little about what distribution the original image comes from. If they did, they wouldn't have a need for "enhance" 99% of the time -- the remaining 1% is for artistic purposes, which this is indeed suited for. It'll be interesting to see how society copes with the removal of the "photographs = evidence" prior. > when enhancing celebrity images, if the model is able to figure out who is in the picture this massively decreases the set of plausible outputs. This is an excellent insight.
      - ZenPsycho 9y ago
        Do you think knowing which state the license plate is from is enough prior knowledge?
    - yorwba 9y ago
      See also https://dheera.net/projects/blur https://dheera.net/projects/blur
  - 7952 9y ago
    The benefit depends on how predictable the phenomenon is that your are interpolating from. Sometimes it will be quantitatively better than a low resolution version, sometimes not. A good example is with compression algorithms for media. They work because the sound or image is predictable. And they are ineffective when the input becomes more unpredictable. But if the output is all you have then running the decompression will probably be better than just reading the raw compressed data. But you have to be aware of the limitations.
  - jonathanstrange 9y ago
    The white shoe output vs black shoe output illustrates this fairly well.
  - matt4077 9y ago
    > You fundamentally can't get back information that has been destroyed/or never captured in the first place. I love this cliché. I've seen it thousands of times, and probably written it myself a few times. We all repeat stuff like that ad nauseam, without ever thinking. Because it's fundamentally flawed, especially in the context that it has usually been applied to, namely criticising the CSI:XYZ trope of "enhancing images". The truth is that there is a lot more information in a low-res image than meets the eye. Even if you can't read the letters on a license plate, it can be recovered by an algorithm. If the Empire State Building is in the background, it's likely to be a US license plate. Maybe only some letters would result in the photo's low-res pattern. If you only see part of a letter, knowing the font may allow you to rule out many letters or numbers etc... It's similar to that guy who used Photoshop's swirl effect to hide his face, not knowing that the effect is deterministic, and can easily be undone. The error mostly appears to be in assuming that the information has been destroyed, when in reality it's often just obscured. And Neural Nets are excellent in squeezing all the information out noisy data.
    - rootw0rm 9y ago
      It's not cliché, it's true. You fundamentally can't get back information that has been destroyed/or never captured in the first place. Yes, a low-res image has lots of information. You can process that information in many ways. Missing data can't just be magically blinked into existence though. Copy/pasting bits of guessed data is NOT getting back information that has been destroyed or never captured. Obscured data is very different from non-existent data. Could the software recreate a destroyed painting of mine based on a simple sketch? Of course not, because it would have to invent details it knows nothing about. I think it's almost dangerous to call this line of thinking cliché. It should be celebrated, not ridiculed.
    - amelius 9y ago
      > It's similar to that guy who used Photoshop's swirl effect to hide his face, not knowing that the effect is deterministic, and can easily be undone. The effect does not only need to be deterministic, but also invertible. A low-res image has multiple "inverses" (yikes), supposedly each with an associated probability (if you would model it that way). So it would be more honest if the algorithm shows them all.
  - k__ 9y ago
    On the other hand, this is what the brain does all the time.
    - eternalban 9y ago
      Wouldn't it be ironic if a mystified and superstitious GAI emerges out of all these efforts.
  - gus_massa 9y ago
    In comparison of output vs original it is clear that the skin color is not accurate.
  - WhitneyLand 9y ago
    >we're interpolating or projecting information that is not there But that's not fully accurate either. Sometimes the information in total will really be a more accurate representation of reality than the blurred image. Maybe it could be described as an educated guess, sometimes wrong, sometimes invaluable. It would be interesting to see the results starting with higher quality images. With the camera quality increasing, many times there should be more data to start with. A
    - phkahler 9y ago
      >> Maybe it could be described as an educated guess, sometimes wrong, sometimes invaluable. When is a guess invaluable?
      - WhitneyLand 9y ago
        When it identifies an established terrorist and prevents a mass casualty event.
  - kevin_thibedeau 9y ago
    "Ladies and gentlemen of the jury, we will definitively prove that the black smudge captured on camera was in fact a gun" Already being done today with DNA.
  - teekert 9y ago
    Exactly, this may be possible: [0] but only of the NN has seen such images before, the output will match the training data but says nothing about reality. [0] https://i.pinimg.com/originals/b5/29/1b/b5291bba7250abd12010644ca848dd75.jpg https://i.pinimg.com/originals/b5/29/1b/b5291bba7250abd12010...
- thaw13579 9y ago
  Approaches like these are hallucinating the high resolution images though--not something that we'd ever want being used for police work. That said, I wonder if it would perform better than eyewitness testimony...
  - xyzzy_plugh 9y ago
    You could e.g. ostensibly produce valid license plates, which could be further reduced by matching the car color and model, to produce a small set of calid records.
    - gambiting 9y ago
      Sure, but if we go by how the police works now, they will take a plate produced by the computer as 100% given and arrest/shoot the owner of that plate because "computer said so".
    - IncRnd 9y ago
      Such an algorithm would likely get the state wrong. This is error prone and fraught with real world difficulties that could get people shot.
    - asfdsfggtfd 9y ago
      You could also just pick a random license plate. It would be just as accurate.
  - netsharc 9y ago
    It would be useful to reduce the number of suspects... calculate possible combinations, match them against the mugshots database and investigate/interrogate those people. Or if you're the NSA/KGB, you can match against the social media pictures database, and then ask the social media company to tell you where these users were at the time of the crime (since the social media app on the phone track their users' location...)
  - smallnamespace 9y ago
    > hallucinating the high resolution images though To play devil's advocate though, modern neuroscience and neuropsychology basically tells us that that our brains reconstruct and recreate our memories every time we try to remember them. Our memories are highly malleable and prone to false implantation... and yet witness testimony is still the gold standard in courts.
- c12 9y ago
  The low resolution to high resolution image synthesis reminds me of the unblur tool that Adobe demoed during Adobe MAX in 2011. Here is the relevant clip if you're interested https://www.youtube.com/watch?v=xxjiQoTp864 https://www.youtube.com/watch?v=xxjiQoTp864
  - ajnin 9y ago
    That demo was quite impressive, but the technique is completely different. Adobe uses deconvolution to recover information and details that are actually in the picture, but not visible (unintuitively blurring is a mathematically reversible transformation. If you know the characteristics of the blur, then you can reverse it. In fact most of Adobe demo's magic comes from knowing the blur kernel and path in advance, not sure how it works in practice for real photos). But the Neural net demoed in this post just "makes up" the missing info using examples from photos it learned from, there is no information recovery.
- jlebrech 9y ago
  It can give possible matches, i don't think it would be admissible in court. they could still trick a confession out of someone using that image.
  - KGIII 9y ago
    It could also narrow down the list of suspects. From there, additional investigation can find more evidence. Having access to big data can help this.
    - jlebrech 9y ago
      true, it cannot be used to "nail" a perp tho, just to help gain extra evidence.
      - KGIII 9y ago
        Yup. In a court of law, the value as evidence is going to be weighted fairly low, even with expert testimony. It may be enough to get a warrant, or a piece in the process of deduction during the investigation phase.
  - ZeroGravitas 9y ago
    You don't specify, but presumably you mean a true confession. It could also be used to generate a false confession. If the prosecutor says "We have proof you were there at the scene" and shows you some generated image, then you as an innocent person have to weigh the chances of the jury being fooled by the image (and even if it's not admissable in court, it may be enough to convince the investiging team that you are responsible and stop looking for the real perpetrator) and the expected sentences if you maintain your innocence vs "admitting" your guilt.
- mathw 9y ago
  Yes! Although what we don't have is any certainty that the enhanced face actually looks like the killer.
- dispo001 9y ago
  Out of sheer curiosity I had a go at manually enhancing the Roundhay Garden Scene by dramatically enlarging the frames, stacking them, aligning them, erasing the most blurred ones and the obvious artifacts. It went from this: https://media.giphy.com/media/pUf3YfamV7BV6/giphy.gif https://media.giphy.com/media/pUf3YfamV7BV6/giphy.gif To this: http://img.go-here.nl/Roundhay_Garden_Scene.gif http://img.go-here.nl/Roundhay_Garden_Scene.gif The funniest part was that the resolution really goes up if you make 1 px into 40 and align the frames accurately (then adjust opacity to the level of blur) The crime television thing would be possible if you have enough frames of the gangster.
- oever 9y ago
  This image from the article shows that the original image and the fantasy image are not alike at all. The faces look to have different ages. The computer even fantasized a beauty mark. http://www.cs.cmu.edu/~aayushb/pixelNN/freq_analysis.png http://www.cs.cmu.edu/~aayushb/pixelNN/freq_analysis.png The computer is fantasizing.
  - O1111OOO 9y ago
    > This image from the article shows that the original image and the fantasy image are not alike at all. This is another avenue that could be further explored, which I quite like. That is, a non-artist can doodle images and create a completely new photo-realistic image based on the line drawings. I was modifying a few images (from link on another comment here: https://affinelayer.com/pixsrv/ https://affinelayer.com/pixsrv/ ) and the end results were interesting.
- seanmcdirmid 9y ago
  You'll get something that looks plausible for sure, maybe not what was originally there though. In the future, someone will be falsely convicted of a crime because a DNN enhance decided to put their picture in some fuzzy context.
- api 9y ago
  It's still impossible. These algorithms find in gaps with their biases, not reality. If information is not there it is not there.
XorNot 9y ago
So is there an analagous process that would apply to audio I wonder?
- jerrre 9y ago
  What would the lo-res starting point be? Low sample-rate, bit depth, ...?
  - eru 9y ago
    Look up compressed sensing for audio. (Eg first result: http://sunbeam.ece.wisc.edu/csaudio/ http://sunbeam.ece.wisc.edu/csaudio/)
  - XorNot 9y ago
    I was thinking like a recording of extremely muffled voices.
- magnat 9y ago
  There kind of already is audio equivalent: MIDI. It supplies low resolution timing and pitch information and it's up to synthesizer to produce audio output matching those data.
  - jeeceebees 9y ago
    I think the interesting part would be example based audio synthesis. Could you replace a synthesizer with a neural network which, when fed examples, would allow you to generate sounds / explore some latent space between the examples. For example an approach similar to https://gauthamzz.github.io/2017/09/23/AudioStyleTransfer/ https://gauthamzz.github.io/2017/09/23/AudioStyleTransfer/ but then using the methods described in the PixelNN paper.
    - radarsat1 9y ago
      I'll just plug my recent work on my sound synthesis "copier": http://gitlab.com/sinclairs/sounderfeit http://gitlab.com/sinclairs/sounderfeit It more or less attempts to be what you describe. Not very polished yet, but I had some basic success in modeling the parameter space of a synth, and adding new latent spaces with regularization.
Wildgoose 9y ago
Very clever. I wonder if something like this could be used for other forms of sensor data as well?
- dispo001 9y ago
  Ah like, what do I look like I want to eat?
tinyrick2 9y ago
This is amazing. I especially like how the result can somewhat be interpreted by showing from what image the part of the generated image is copied (see Figure 5).
maho 9y ago
I hope some day this will generalize to video. I don't care about the exact shape of background trees in an action movie - with this approach, they could be compressed to just a few bytes, regardless of resolution.
- IncRnd 9y ago
  That's what video compression does now.
  - roel_v 9y ago
    No, today's compression is about compressing what's already in the one movie. But imagine that you run your training set over 100's or 1000's of films, and extract just enough to represent say different types of trees in a few bytes. You could 'compress' a film by replacing data with markers that essentially describe some properties of the tree, and those properties + the training set are then used during 'decompression' to recreate (an approximation of) the tree. This would of course not give you any space savings when you want to distribute 1 movie. There would be some minimum number of movies where the training set + actual movies would be smaller than the sum of the sizes of the individual movies compressed. I'm not saying this would be a net space saver, or necessarily a good technique at all, but the concept is intriguing.
    - makapuf 9y ago
      And some action movies could then be compressed to mere bytes if you basically have a virtual movie studio in your PC.
      - eesmith 9y ago
        Might make it easy to watch the Sweded version.
- adrianN 9y ago
  Plug in the script and some artist's impressions of the sets and generate the whole movie on the fly.
imron 9y ago
Seems to have a thing for beards.
verytrivial 9y ago
A pair of the inputs in the edge-to-edge faces are swapped. I have nagged an author.
- verytrivial 9y ago
  ... and I followed up with an annotated screenshot. I tried, I really did!
avian 9y ago
I found the title somewhat misleading. I was expecting some clever application of the nearest-neighbor interpolation. But this seems to involve neural nets and appears far from "simple" to me (I'm not in the image processing field though).
- SeanDav 9y ago
  Agree. This appears to be more a clever implementation of an algorithm generating "artistic" impressions. In some cases, creating artifacts which simply were not part of the original picture.
  - doomlaser 9y ago
    The term in neural net research is 'face hallucination': https://people.csail.mit.edu/celiu/FaceHallucination/fh.html https://people.csail.mit.edu/celiu/FaceHallucination/fh.html Take a low resolution input image, and hallucinate a higher resolution version by statistically assembling bits from similar images in a large data set of training images.
    - phkahler 9y ago
      If anyone ever tries to use this in court I hope they call it "Face Hallucination" and not "Image Reconstruction". On the research side, I wonder what the point of this is. I find it interesting but of little practical value.
      - aristus 9y ago
        It's a way to refine their models. A systematic model-based representation of data is basically also a generator of that data. Why is that? Blame Kolmogorov. There are deep connections between compression, serialization, and computation. An optimal compression scheme is a serialization and the Turing-complete program to decode it. For example: you can compress pi into a few lines of algorithm plus a starting constant like 4.
- jampekka 9y ago
nl 9y ago
To paraphrase Google Brain's Vincent Vanhoucke, this appears to be another example where using context prediction from neighboring values outperforms an autoencoder approach. If 2017 was the year of GANs, 2018 will be the year context prediction.
the8472 9y ago
All those examples are fairly low-resolution. Does this approach scale or can it be applied in some tiled fashion? Or would the artifacts get worse for larger images?
throwaway00100 9y ago
No code available.
- jszymborski 9y ago
  Which is sadly par for the course in this field, or at least my experience. You can always email the group...
  - sosuke 9y ago
    I spent too long trying to get RAISR to work when that paper came out. You can try it out from some Github repos but no one has been able to recreate the results Google presented. I would be hard pressed to say my hires photos looked any better than the originals when scaled up on my iPhone screen. I do wish they would release the code AND any related training images they used to get those results.
debuggerpk 9y ago
Hollywood got it right!!!
imaginenore 9y ago
It almost looks like they mixed training and testing data in some of the examples. The bottom-left sample in the normals-to-faces is extremely suspicions.
- jj12345 9y ago
  I was looking at this as well, but I'm willing to suspend my disbelief because the normal vaguely looks like it has a good deal of information (in a basic fidelity sense).
  - jameshart 9y ago
    seems astonishing that the normal information includes enough detail to tell you which direction the eyes are pointing, though?
nathan_f77 9y ago
This is cool, but in the comparison with Pix-to-Pix, it seems like Pix-to-Pix is the clear winner.
laythea 9y ago
I wonder if this could be applied to "incomplete" 3D models and the work shifted to the GPU!?
kensai 9y ago
OMG, now the "enhance" they say in investigative TV series and movies will actually be reality! :p
joosters 9y ago
I don't understand how the edges-to-faces can possibly work. The inputs seem to be black & white, and yet the output pictures have light skin tones. How can their algorithm work out the skin tone from a colourless image. Perhaps their training data only had white people in it?
- jtanderson 9y ago
  I had the same thought. Maybe it's not that there were only white people in the dataset, but it's actually taking the shape of the face into account, and it most closely matches those with white skin tones. I suggest this by looking at the cat one: it has the stripes coming off the eyes, so suggests one of the grey striped breeds rather than, e.g. all black or calico. It's probably more than pixel-by-pixel NN interpolation, but also taking into account some of the actual structure of the edges.
- cbr 9y ago
  Color comes from the initial neural network step. Since skin color is relatively predictable from facial features (ex: nose width), it should be able to do reasonably well.
  - joosters 9y ago
    Really? With what accuracy? This is the kind of assumption that will get research groups into very deep water... Just imagine the kind of CCTV usage being discussed elsewhere in this thread. But the neural network happens to have a wrong bias towards skin colour...
    - dahart 9y ago
      Using image synthesis at all can't be used for up-rezing CCTV imagery, the output is a fabrication and the researchers have all said so. People imagining bad use cases shouldn't be relied on. ;) If an investigator used this to track down criminals, they are the ones getting into deep water and making assumptions.
    - radarsat1 9y ago
jokoon 9y ago
I have a large collection of images, many being accessible through google image search. I wonder if there could be a way to "index" those images so I can find them back without storing the whole image, using some type of clever image histogram or hashing-kind function. I wonder if that thing already exist, since there are many images, and since most images have a lot of difference in their data, could it be possible to create some kind of function that describe an image in a way that entering such histogram redirects to (or the closest) the image it indexed? I guess I'm lacking the math, but it sounds like some "averaging" hashing function.
- danielmorozoff 9y ago
  This is the current approach for large sale image retrieval. By using some model to extract features and then performing distance calculations. This is usually done with hashing once speed and the size of the dataset become large.
- dannyw 9y ago
  That's perceptual hashing. Check out https://www.phash.org/ https://www.phash.org/
  - mlevental 9y ago
    so will this do something like image recognition? ie does it work as well as surf/sift?
    - aub3bhat 9y ago
      Perceptual hashing is useful for copy detection. Its not robust to changes/transformations nor do the hashes encode any semantic information.
  - jokoon 9y ago
    Is there simpler way to implement it? This is a library, but aren't more common ways to do this?
- 9y ago
sgtAtom 9y ago
Enhance.
tke248 9y ago
Does anything like this exist on the web would like to send a blurry license plate picture through this and see what it comes up with..
deevolution 9y ago
Apparently you grow a beard after using their nn model?
- XnoiVeX 9y ago
  I noticed that too. I hope it is just a documentation error.
smrtinsert 9y ago
"Enhance" is real. When will this stuff trickle into lower level law enforcement?
- nashashmi 9y ago
  Should it? The accuracy of the pictures were dismal.
- TFortunato 9y ago
  Hopefully never, but I'm sure someone will see this and try! (Because these kind of techniques aren't really enhancing the images in a way that gives you new and useful information: they are taking the low-res images as input, and giving you a plausible high-res image as output, based on it's training data. It is NOT however trying to say "this is the ACTUAL high res image that generated this low-res image"
- asfdsfggtfd 9y ago
  Hopefully never. This does not enhance the image - it makes up a plausible imaginary image. EDIT: Furthermore the range of plausible imaginary images that match a given input is high (infinite?).
  - pc86 9y ago
    Just need to look at the picture of Fred Armisen to see that this technique can generate a picture of a plausibly real human who bears no/very little resemblance to the original image.
  - smrtinsert 9y ago
    Why not? A recreation that leads to an identification should be enough for a warrant that could be used for a continued investigation.
    - asfdsfggtfd 9y ago
      We could also just pick a random person off the street and punish them - it would be similarly accurate and fair (actually probably fairer - if this is trained on pictures with a certain bias it will return pictures with that bias). This paper does not demonstrate an enhancement technique but a phenomena which those using inverse methods called "overfitting".
stevespang 9y ago
So, where's the app ?
mlwelles 9y ago
I noticed that all of the human examples are caucasian. I'd be very interested to see how accurate it is with more representative range of human faces than how it handles animals or handbags. Had personal experience on a project where the facial scanning engine failed spectacularly when anyone except white men like me tried to use it. An experience that's pretty common, too: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwi9mvHUzcDWAhUD4WMKHfNxARAQtwIIJjAA&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dt4DT3tQqgRM&usg=AFQjCNFvkH7_egyu4lzpmInmlC5-na5TOA https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&c... http://www.telegraph.co.uk/technology/2017/08/09/faceapp-sparks-racist-backlash-black-white-asian-filters/ http://www.telegraph.co.uk/technology/2017/08/09/faceapp-spa... https://www.theatlantic.com/technology/archive/2016/04/the-underlying-bias-of-facial-recognition-systems/476991/ https://www.theatlantic.com/technology/archive/2016/04/the-u... https://www.theguardian.com/technology/2017/may/28/joy-buolamwini-when-algorithms-are-racist-facial-recognition-bias https://www.theguardian.com/technology/2017/may/28/joy-buola...
yazanator 9y ago
Is there a GitHub repository link?
ScoutOrgo 9y ago
Can we use this to identify the leprechaun and find where da gold at?
ChuckMcM 9y ago
Is anyone in the FX business playing with this stuff? I'm thinking generational backdrops with groups of people/stuff/animals in them without a lot of modelling input.

497 points by pentestercrab 9y ago

Thanks for sharing! Very interesting!

I used to roll my eyes at crime television shows, whenever they said "Enhance" for a low quality image. Now it seems the possibility of that becoming realistic are increasing with a steady clip, based on this paper and other enhancement techniques I've seen posted here.

ACow_Adonis 9y ago
Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". You fundamentally can't get back information that has been destroyed/or never captured in the first place. What you can do is fill in the gaps/information with plausible values. I don't know whether this sounds like I'm splitting hairs, but it's really important that the general public not think we're extracting information in these procedures, we're interpolating or projecting information that is not there. Very useful for artificially generating skins for each shoe on a shoe rack in a computer game or simulation, potentially disastrous if the general public starts to think it's applicable to security camera footage or admissible as evidence...
- jopsen 9y ago
  Sometimes US justice system seems very "approximate". So why not convict people based on interpolated evidence? - I'm joking of course :) hehe
  - WillReplyfFood 9y ago
    You seem like a funny guy- the batmaNN interpolates that you are very likely a joker.
  - dahart 9y ago
    Sadly, it actually happens sometimes. https://www.wired.com/2017/04/courts-using-ai-sentence-criminals-must-stop-now/ https://www.wired.com/2017/04/courts-using-ai-sentence-crimi... This thread a year ago worried about it too, but the paper itself seems implausible and problematic. https://news.ycombinator.com/item?id=12983827 https://news.ycombinator.com/item?id=12983827
- ZeroGravitas 9y ago
  To give specific examples from their test data, it added stubble to people who didn't have stubble, gave them a different shape of glasses, changed the color of cats, changed the color and brand of sport shoe. And even then, I'm a little suspicious of how close some of the images got to original without being given color information. It appears that info was either hidden in the original in a way not apparent to humans or was implicit in their data set in some way that would make it fail on photos of people with different skin tones.
  - omtinez 9y ago
    I haven't read the paper in full detail, but reading between the lines I'm guessing that there's a significant portion of manual processing and hand waving involved. From the abstract, emphasis mine: > the second stage uses a pixel-wise nearest neighbor method to map the smoothed output to multiple high-quality, high-frequency outputs in a controllable manner. My interpretation is that they select training data by hand and generate a bunch of outputs. Repeating the process until they like the final result. From the paper: > we allow a user to have an arbitrarily-fine level of control through on-the-fly editing of the exemplar set (E.g., “resynthesize an image using the eye from this image and the nose from that one”).
    - IshKebab 9y ago
      Could be pretty great for police sketch artists. (Although pretty misleading for juries too.)
      - adrianN 9y ago
        Just train the model with the suspect's Facebook photostream and presto you have convincing evidence.
    - WhitneyLand 9y ago
      There's nothing weak or negative about that, it's exactly what'd you expect. Obviously for a given input there will be multiple plausible outputs. With any such system it would make sense to allow some control in choosing among the outputs.
- gambiting 9y ago
  No, but think of these blurred images as a "hash" - in an ideal situation, you only have one value that encodes to a certain hash value, right? So If you are given a hash X you technically can work out that it was derived from value Y - you're not getting back information that was lost - in a way it was merely encoded into the blurred image, and it should be possible to produce a real image which, when blurred, will match what you have. Don't get me wrong, I think we're still far far far off situation where we can get those reliably, but I can see how you could get the actual face out of a blurred image.
  - ComputerGuru 9y ago
    > you only have one value that encodes to a certain hash value, right? Errr wrong. A perfect hash, yes. But they're never perfect. You have a collision domain and you hope that you don't have enough inputs to trigger a birthday paradox. Look at the pictures on the article. It's an outline of the shoe. That's your hash. ANY shoe with that general outline resolves to that same hash. If your input is objects found in the Oxford English Dictionary, you'll have low collisions. An elephant doesn't hash to that outline. But if your inputs is the Kohl's catalog, you'll have an unacceptable collision rate. Hashes are attempts at creating a _truncated_ "unique" representation of an input. They throw away data they hope isn't necessary to uniquely identify between possible inputs (bits). A perfect hash for all possible 32 bit values is 32 bits. You can't even have a collision free 31 bit hash. So back to the blurry security camera footage of a license plate or a face. Sure, that "hash" can reliably tell you that it wasn't a sasquatch that committed the robbery, but it literally doesn't contain the data necessary to _ever_ prove it was the suspect in question, even if the techs _can_ prove that the suspect hashes to the image in the footage.
    - consp 9y ago
      Fortunately there is a limit: the universe (in a practical sense). You cannot encode all states it has in a hash as it would require more states than you want to encode as you already mentioned (pigeon hole). But representing macroscopic data like text (or basically anything bigger than atomic scale) uniquely can be done with 128+ bits. Double that and you are likely safe for collisions, assuming the method you use is uniform and not biased to some input. If you want ease collision examples you can take a look at people using CRC32 as hashes/digests. It is notoriously prone to collisions (since only 32 bits).
- usrusr 9y ago
  Yeah, replace the training set with cartoon characters and the crime show dialog goes like this: "Zoom! Enhance! Zoom! Enhance! Enhance! Oh my god it's full of Smurfs..."
  - deleted 9y ago
    [deleted]
- scarface74 9y ago
  What you can do though, in limited circumstances, is create a still picture with more detail from a lower quality video. https://photo.stackexchange.com/questions/17098/csi-image-resolution-enhance-how-real-is-it https://photo.stackexchange.com/questions/17098/csi-image-re...
  - murkle 9y ago
    It's a well-known technique in astronomy, eg https://www.aanda.org/articles/aa/ps/2005/22/aa2320-04.ps.gz https://www.aanda.org/articles/aa/ps/2005/22/aa2320-04.ps.gz
    - mcbits 9y ago
      For anyone put off by the .ps.gz, it's actually just a normal web page that links to the full article in HTML and PDF. Not sure what they were thinking with that URL. I almost didn't bother to look. (Maybe that's what they were thinking?)
  - dahart 9y ago
    I seem to remember from my computer vision class way back when that there's a fundamental theoretical limit to the amount of detail you can get out of a moving sequence. Recovering frequencies a little higher than the pixel sampling is definitely possible, but I feel like it was maybe something like 10x theoretical maximum. I also get the feeling, from looking around at available software, that in practice achieving 2-3x is the most you can get in ideal conditions, and most video is far from ideal.
- throwaway613834 9y ago
  > I don't know whether this sounds like I'm splitting hairs Somewhat no, but somewhat yes. Thing is, while there can be lots of input images that generate the same output, it could be that only one (or a handful) of them would occur in reality. If this happens to sometimes be the case, and if you could somehow guarantee this was the case in some particular scenario, it could very well make sense to admit it as evidence. Of course, the issue is that figuring this out may not be possible...
- sweezyjeezy 9y ago
  > Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". I would argue that this is a form of enhancement though, and in some cases will be enough to completely reconstruct the original image. For example, if I give you a scanned PDF, and you know for a fact that it was size 12 black Ariel text on a white background, this can feasibly let you reconstruct the original image perfectly. The 'prior' that has been encoded by the model from the large amount of other images increases the mutual information between grainy image and high-res. The catch is that uncertainty cannot be removed entirely, and you need to know that the target image comes from roughly the same distribution as the training set. But knowing this gives you information that is not encoded in the pixels themselves, so you can't necessarily argue that some enhancement is impossible. For example with celebrity images, if the model is able to figure out who is in the picture, this massively decreases the set of plausible outputs.
  - trevyn 9y ago
    > The catch is that you need to know that the target image comes from roughly the same distribution as the training set. When humans think about "enhance", they imagine extracting subtle details that were not obvious from the original, which implies that they know very little about what distribution the original image comes from. If they did, they wouldn't have a need for "enhance" 99% of the time -- the remaining 1% is for artistic purposes, which this is indeed suited for. It'll be interesting to see how society copes with the removal of the "photographs = evidence" prior. > when enhancing celebrity images, if the model is able to figure out who is in the picture this massively decreases the set of plausible outputs. This is an excellent insight.
    - ZenPsycho 9y ago
      Do you think knowing which state the license plate is from is enough prior knowledge?
  - yorwba 9y ago
    See also https://dheera.net/projects/blur https://dheera.net/projects/blur
- 7952 9y ago
  The benefit depends on how predictable the phenomenon is that your are interpolating from. Sometimes it will be quantitatively better than a low resolution version, sometimes not. A good example is with compression algorithms for media. They work because the sound or image is predictable. And they are ineffective when the input becomes more unpredictable. But if the output is all you have then running the decompression will probably be better than just reading the raw compressed data. But you have to be aware of the limitations.
- jonathanstrange 9y ago
  The white shoe output vs black shoe output illustrates this fairly well.
- matt4077 9y ago
  > You fundamentally can't get back information that has been destroyed/or never captured in the first place. I love this cliché. I've seen it thousands of times, and probably written it myself a few times. We all repeat stuff like that ad nauseam, without ever thinking. Because it's fundamentally flawed, especially in the context that it has usually been applied to, namely criticising the CSI:XYZ trope of "enhancing images". The truth is that there is a lot more information in a low-res image than meets the eye. Even if you can't read the letters on a license plate, it can be recovered by an algorithm. If the Empire State Building is in the background, it's likely to be a US license plate. Maybe only some letters would result in the photo's low-res pattern. If you only see part of a letter, knowing the font may allow you to rule out many letters or numbers etc... It's similar to that guy who used Photoshop's swirl effect to hide his face, not knowing that the effect is deterministic, and can easily be undone. The error mostly appears to be in assuming that the information has been destroyed, when in reality it's often just obscured. And Neural Nets are excellent in squeezing all the information out noisy data.
  - rootw0rm 9y ago
    It's not cliché, it's true. You fundamentally can't get back information that has been destroyed/or never captured in the first place. Yes, a low-res image has lots of information. You can process that information in many ways. Missing data can't just be magically blinked into existence though. Copy/pasting bits of guessed data is NOT getting back information that has been destroyed or never captured. Obscured data is very different from non-existent data. Could the software recreate a destroyed painting of mine based on a simple sketch? Of course not, because it would have to invent details it knows nothing about. I think it's almost dangerous to call this line of thinking cliché. It should be celebrated, not ridiculed.
  - amelius 9y ago
    > It's similar to that guy who used Photoshop's swirl effect to hide his face, not knowing that the effect is deterministic, and can easily be undone. The effect does not only need to be deterministic, but also invertible. A low-res image has multiple "inverses" (yikes), supposedly each with an associated probability (if you would model it that way). So it would be more honest if the algorithm shows them all.
- k__ 9y ago
  On the other hand, this is what the brain does all the time.
  - eternalban 9y ago
    Wouldn't it be ironic if a mystified and superstitious GAI emerges out of all these efforts.
- gus_massa 9y ago
  In comparison of output vs original it is clear that the skin color is not accurate.
- WhitneyLand 9y ago
  >we're interpolating or projecting information that is not there But that's not fully accurate either. Sometimes the information in total will really be a more accurate representation of reality than the blurred image. Maybe it could be described as an educated guess, sometimes wrong, sometimes invaluable. It would be interesting to see the results starting with higher quality images. With the camera quality increasing, many times there should be more data to start with. A
  - phkahler 9y ago
    >> Maybe it could be described as an educated guess, sometimes wrong, sometimes invaluable. When is a guess invaluable?
    - WhitneyLand 9y ago
      When it identifies an established terrorist and prevents a mass casualty event.
- kevin_thibedeau 9y ago
  "Ladies and gentlemen of the jury, we will definitively prove that the black smudge captured on camera was in fact a gun" Already being done today with DNA.
- teekert 9y ago
  Exactly, this may be possible: [0] but only of the NN has seen such images before, the output will match the training data but says nothing about reality. [0] https://i.pinimg.com/originals/b5/29/1b/b5291bba7250abd12010644ca848dd75.jpg https://i.pinimg.com/originals/b5/29/1b/b5291bba7250abd12010...
thaw13579 9y ago
Approaches like these are hallucinating the high resolution images though--not something that we'd ever want being used for police work. That said, I wonder if it would perform better than eyewitness testimony...
- xyzzy_plugh 9y ago
  You could e.g. ostensibly produce valid license plates, which could be further reduced by matching the car color and model, to produce a small set of calid records.
  - gambiting 9y ago
    Sure, but if we go by how the police works now, they will take a plate produced by the computer as 100% given and arrest/shoot the owner of that plate because "computer said so".
  - IncRnd 9y ago
    Such an algorithm would likely get the state wrong. This is error prone and fraught with real world difficulties that could get people shot.
  - asfdsfggtfd 9y ago
    You could also just pick a random license plate. It would be just as accurate.
- netsharc 9y ago
  It would be useful to reduce the number of suspects... calculate possible combinations, match them against the mugshots database and investigate/interrogate those people. Or if you're the NSA/KGB, you can match against the social media pictures database, and then ask the social media company to tell you where these users were at the time of the crime (since the social media app on the phone track their users' location...)
- smallnamespace 9y ago
  > hallucinating the high resolution images though To play devil's advocate though, modern neuroscience and neuropsychology basically tells us that that our brains reconstruct and recreate our memories every time we try to remember them. Our memories are highly malleable and prone to false implantation... and yet witness testimony is still the gold standard in courts.
c12 9y ago
The low resolution to high resolution image synthesis reminds me of the unblur tool that Adobe demoed during Adobe MAX in 2011. Here is the relevant clip if you're interested https://www.youtube.com/watch?v=xxjiQoTp864 https://www.youtube.com/watch?v=xxjiQoTp864
- ajnin 9y ago
  That demo was quite impressive, but the technique is completely different. Adobe uses deconvolution to recover information and details that are actually in the picture, but not visible (unintuitively blurring is a mathematically reversible transformation. If you know the characteristics of the blur, then you can reverse it. In fact most of Adobe demo's magic comes from knowing the blur kernel and path in advance, not sure how it works in practice for real photos). But the Neural net demoed in this post just "makes up" the missing info using examples from photos it learned from, there is no information recovery.
jlebrech 9y ago
It can give possible matches, i don't think it would be admissible in court. they could still trick a confession out of someone using that image.
- KGIII 9y ago
  It could also narrow down the list of suspects. From there, additional investigation can find more evidence. Having access to big data can help this.
  - jlebrech 9y ago
    true, it cannot be used to "nail" a perp tho, just to help gain extra evidence.
    - KGIII 9y ago
      Yup. In a court of law, the value as evidence is going to be weighted fairly low, even with expert testimony. It may be enough to get a warrant, or a piece in the process of deduction during the investigation phase.
- ZeroGravitas 9y ago
  You don't specify, but presumably you mean a true confession. It could also be used to generate a false confession. If the prosecutor says "We have proof you were there at the scene" and shows you some generated image, then you as an innocent person have to weigh the chances of the jury being fooled by the image (and even if it's not admissable in court, it may be enough to convince the investiging team that you are responsible and stop looking for the real perpetrator) and the expected sentences if you maintain your innocence vs "admitting" your guilt.
mathw 9y ago
Yes! Although what we don't have is any certainty that the enhanced face actually looks like the killer.
dispo001 9y ago
Out of sheer curiosity I had a go at manually enhancing the Roundhay Garden Scene by dramatically enlarging the frames, stacking them, aligning them, erasing the most blurred ones and the obvious artifacts. It went from this: https://media.giphy.com/media/pUf3YfamV7BV6/giphy.gif https://media.giphy.com/media/pUf3YfamV7BV6/giphy.gif To this: http://img.go-here.nl/Roundhay_Garden_Scene.gif http://img.go-here.nl/Roundhay_Garden_Scene.gif The funniest part was that the resolution really goes up if you make 1 px into 40 and align the frames accurately (then adjust opacity to the level of blur) The crime television thing would be possible if you have enough frames of the gangster.
oever 9y ago
This image from the article shows that the original image and the fantasy image are not alike at all. The faces look to have different ages. The computer even fantasized a beauty mark. http://www.cs.cmu.edu/~aayushb/pixelNN/freq_analysis.png http://www.cs.cmu.edu/~aayushb/pixelNN/freq_analysis.png The computer is fantasizing.
- O1111OOO 9y ago
  > This image from the article shows that the original image and the fantasy image are not alike at all. This is another avenue that could be further explored, which I quite like. That is, a non-artist can doodle images and create a completely new photo-realistic image based on the line drawings. I was modifying a few images (from link on another comment here: https://affinelayer.com/pixsrv/ https://affinelayer.com/pixsrv/ ) and the end results were interesting.
seanmcdirmid 9y ago
You'll get something that looks plausible for sure, maybe not what was originally there though. In the future, someone will be falsely convicted of a crime because a DNN enhance decided to put their picture in some fuzzy context.
api 9y ago
It's still impossible. These algorithms find in gaps with their biases, not reality. If information is not there it is not there.

XorNot 9y ago

So is there an analagous process that would apply to audio I wonder?

jerrre 9y ago
What would the lo-res starting point be? Low sample-rate, bit depth, ...?
- eru 9y ago
  Look up compressed sensing for audio. (Eg first result: http://sunbeam.ece.wisc.edu/csaudio/ http://sunbeam.ece.wisc.edu/csaudio/)
- XorNot 9y ago
  I was thinking like a recording of extremely muffled voices.
magnat 9y ago
There kind of already is audio equivalent: MIDI. It supplies low resolution timing and pitch information and it's up to synthesizer to produce audio output matching those data.
- jeeceebees 9y ago
  I think the interesting part would be example based audio synthesis. Could you replace a synthesizer with a neural network which, when fed examples, would allow you to generate sounds / explore some latent space between the examples. For example an approach similar to https://gauthamzz.github.io/2017/09/23/AudioStyleTransfer/ https://gauthamzz.github.io/2017/09/23/AudioStyleTransfer/ but then using the methods described in the PixelNN paper.
  - radarsat1 9y ago
    I'll just plug my recent work on my sound synthesis "copier": http://gitlab.com/sinclairs/sounderfeit http://gitlab.com/sinclairs/sounderfeit It more or less attempts to be what you describe. Not very polished yet, but I had some basic success in modeling the parameter space of a synth, and adding new latent spaces with regularization.

Wildgoose 9y ago

Very clever. I wonder if something like this could be used for other forms of sensor data as well?

dispo001 9y ago
Ah like, what do I look like I want to eat?

tinyrick2 9y ago

This is amazing. I especially like how the result can somewhat be interpreted by showing from what image the part of the generated image is copied (see Figure 5).

maho 9y ago

I hope some day this will generalize to video. I don't care about the exact shape of background trees in an action movie - with this approach, they could be compressed to just a few bytes, regardless of resolution.

IncRnd 9y ago
That's what video compression does now.
- roel_v 9y ago
  No, today's compression is about compressing what's already in the one movie. But imagine that you run your training set over 100's or 1000's of films, and extract just enough to represent say different types of trees in a few bytes. You could 'compress' a film by replacing data with markers that essentially describe some properties of the tree, and those properties + the training set are then used during 'decompression' to recreate (an approximation of) the tree. This would of course not give you any space savings when you want to distribute 1 movie. There would be some minimum number of movies where the training set + actual movies would be smaller than the sum of the sizes of the individual movies compressed. I'm not saying this would be a net space saver, or necessarily a good technique at all, but the concept is intriguing.
  - makapuf 9y ago
    And some action movies could then be compressed to mere bytes if you basically have a virtual movie studio in your PC.
    - eesmith 9y ago
      Might make it easy to watch the Sweded version.
adrianN 9y ago
Plug in the script and some artist's impressions of the sets and generate the whole movie on the fly.

imron 9y ago

Seems to have a thing for beards.

verytrivial 9y ago

A pair of the inputs in the edge-to-edge faces are swapped. I have nagged an author.

verytrivial 9y ago
... and I followed up with an annotated screenshot. I tried, I really did!

avian 9y ago

I found the title somewhat misleading. I was expecting some clever application of the nearest-neighbor interpolation. But this seems to involve neural nets and appears far from "simple" to me (I'm not in the image processing field though).

SeanDav 9y ago
Agree. This appears to be more a clever implementation of an algorithm generating "artistic" impressions. In some cases, creating artifacts which simply were not part of the original picture.
- doomlaser 9y ago
  The term in neural net research is 'face hallucination': https://people.csail.mit.edu/celiu/FaceHallucination/fh.html https://people.csail.mit.edu/celiu/FaceHallucination/fh.html Take a low resolution input image, and hallucinate a higher resolution version by statistically assembling bits from similar images in a large data set of training images.
  - phkahler 9y ago
    If anyone ever tries to use this in court I hope they call it "Face Hallucination" and not "Image Reconstruction". On the research side, I wonder what the point of this is. I find it interesting but of little practical value.
    - aristus 9y ago
      It's a way to refine their models. A systematic model-based representation of data is basically also a generator of that data. Why is that? Blame Kolmogorov. There are deep connections between compression, serialization, and computation. An optimal compression scheme is a serialization and the Turing-complete program to decode it. For example: you can compress pi into a few lines of algorithm plus a starting constant like 4.
jampekka 9y ago

nl 9y ago

To paraphrase Google Brain's Vincent Vanhoucke, this appears to be another example where using context prediction from neighboring values outperforms an autoencoder approach. If 2017 was the year of GANs, 2018 will be the year context prediction.

the8472 9y ago

All those examples are fairly low-resolution. Does this approach scale or can it be applied in some tiled fashion? Or would the artifacts get worse for larger images?

throwaway00100 9y ago

No code available.

jszymborski 9y ago
Which is sadly par for the course in this field, or at least my experience. You can always email the group...
- sosuke 9y ago
  I spent too long trying to get RAISR to work when that paper came out. You can try it out from some Github repos but no one has been able to recreate the results Google presented. I would be hard pressed to say my hires photos looked any better than the originals when scaled up on my iPhone screen. I do wish they would release the code AND any related training images they used to get those results.

debuggerpk 9y ago

Hollywood got it right!!!

imaginenore 9y ago

It almost looks like they mixed training and testing data in some of the examples. The bottom-left sample in the normals-to-faces is extremely suspicions.

jj12345 9y ago
I was looking at this as well, but I'm willing to suspend my disbelief because the normal vaguely looks like it has a good deal of information (in a basic fidelity sense).
- jameshart 9y ago
  seems astonishing that the normal information includes enough detail to tell you which direction the eyes are pointing, though?

nathan_f77 9y ago

This is cool, but in the comparison with Pix-to-Pix, it seems like Pix-to-Pix is the clear winner.

laythea 9y ago

I wonder if this could be applied to "incomplete" 3D models and the work shifted to the GPU!?

kensai 9y ago

OMG, now the "enhance" they say in investigative TV series and movies will actually be reality! :p

joosters 9y ago

I don't understand how the edges-to-faces can possibly work. The inputs seem to be black & white, and yet the output pictures have light skin tones. How can their algorithm work out the skin tone from a colourless image. Perhaps their training data only had white people in it?

jtanderson 9y ago
I had the same thought. Maybe it's not that there were only white people in the dataset, but it's actually taking the shape of the face into account, and it most closely matches those with white skin tones. I suggest this by looking at the cat one: it has the stripes coming off the eyes, so suggests one of the grey striped breeds rather than, e.g. all black or calico. It's probably more than pixel-by-pixel NN interpolation, but also taking into account some of the actual structure of the edges.
cbr 9y ago
Color comes from the initial neural network step. Since skin color is relatively predictable from facial features (ex: nose width), it should be able to do reasonably well.
- joosters 9y ago
  Really? With what accuracy? This is the kind of assumption that will get research groups into very deep water... Just imagine the kind of CCTV usage being discussed elsewhere in this thread. But the neural network happens to have a wrong bias towards skin colour...
  - dahart 9y ago
    Using image synthesis at all can't be used for up-rezing CCTV imagery, the output is a fabrication and the researchers have all said so. People imagining bad use cases shouldn't be relied on. ;) If an investigator used this to track down criminals, they are the ones getting into deep water and making assumptions.
  - radarsat1 9y ago

jokoon 9y ago

I have a large collection of images, many being accessible through google image search. I wonder if there could be a way to "index" those images so I can find them back without storing the whole image, using some type of clever image histogram or hashing-kind function. I wonder if that thing already exist, since there are many images, and since most images have a lot of difference in their data, could it be possible to create some kind of function that describe an image in a way that entering such histogram redirects to (or the closest) the image it indexed? I guess I'm lacking the math, but it sounds like some "averaging" hashing function.

danielmorozoff 9y ago
This is the current approach for large sale image retrieval. By using some model to extract features and then performing distance calculations. This is usually done with hashing once speed and the size of the dataset become large.
dannyw 9y ago
That's perceptual hashing. Check out https://www.phash.org/ https://www.phash.org/
- mlevental 9y ago
  so will this do something like image recognition? ie does it work as well as surf/sift?
  - aub3bhat 9y ago
    Perceptual hashing is useful for copy detection. Its not robust to changes/transformations nor do the hashes encode any semantic information.
- jokoon 9y ago
  Is there simpler way to implement it? This is a library, but aren't more common ways to do this?
9y ago

sgtAtom 9y ago

Enhance.

tke248 9y ago

Does anything like this exist on the web would like to send a blurry license plate picture through this and see what it comes up with..

deevolution 9y ago

Apparently you grow a beard after using their nn model?

XnoiVeX 9y ago
I noticed that too. I hope it is just a documentation error.

smrtinsert 9y ago

"Enhance" is real. When will this stuff trickle into lower level law enforcement?

nashashmi 9y ago
Should it? The accuracy of the pictures were dismal.
TFortunato 9y ago
Hopefully never, but I'm sure someone will see this and try! (Because these kind of techniques aren't really enhancing the images in a way that gives you new and useful information: they are taking the low-res images as input, and giving you a plausible high-res image as output, based on it's training data. It is NOT however trying to say "this is the ACTUAL high res image that generated this low-res image"
asfdsfggtfd 9y ago
Hopefully never. This does not enhance the image - it makes up a plausible imaginary image. EDIT: Furthermore the range of plausible imaginary images that match a given input is high (infinite?).
- pc86 9y ago
  Just need to look at the picture of Fred Armisen to see that this technique can generate a picture of a plausibly real human who bears no/very little resemblance to the original image.
- smrtinsert 9y ago
  Why not? A recreation that leads to an identification should be enough for a warrant that could be used for a continued investigation.
  - asfdsfggtfd 9y ago
    We could also just pick a random person off the street and punish them - it would be similarly accurate and fair (actually probably fairer - if this is trained on pictures with a certain bias it will return pictures with that bias). This paper does not demonstrate an enhancement technique but a phenomena which those using inverse methods called "overfitting".

stevespang 9y ago

So, where's the app ?

mlwelles 9y ago

I noticed that all of the human examples are caucasian. I'd be very interested to see how accurate it is with more representative range of human faces than how it handles animals or handbags. Had personal experience on a project where the facial scanning engine failed spectacularly when anyone except white men like me tried to use it. An experience that's pretty common, too: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwi9mvHUzcDWAhUD4WMKHfNxARAQtwIIJjAA&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dt4DT3tQqgRM&usg=AFQjCNFvkH7_egyu4lzpmInmlC5-na5TOA https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&c... http://www.telegraph.co.uk/technology/2017/08/09/faceapp-sparks-racist-backlash-black-white-asian-filters/ http://www.telegraph.co.uk/technology/2017/08/09/faceapp-spa... https://www.theatlantic.com/technology/archive/2016/04/the-underlying-bias-of-facial-recognition-systems/476991/ https://www.theatlantic.com/technology/archive/2016/04/the-u... https://www.theguardian.com/technology/2017/may/28/joy-buolamwini-when-algorithms-are-racist-facial-recognition-bias https://www.theguardian.com/technology/2017/may/28/joy-buola...

yazanator 9y ago

Is there a GitHub repository link?

ScoutOrgo 9y ago

Can we use this to identify the leprechaun and find where da gold at?

ChuckMcM 9y ago

Is anyone in the FX business playing with this stuff? I'm thinking generational backdrops with groups of people/stuff/animals in them without a lot of modelling input.