14 ms·

Using neural networks to upscale a famous 1896 video to 4k quality

349 points by eyegor 7y ago

eyegor 7y ago
Tangentially related, a real-time video upscaler: https://github.com/bloc97/Anime4K https://github.com/bloc97/Anime4K
- marzell 7y ago
  IMO in the v1.0 comparisons, v0.9 is sharper in every case.
  - jcims 7y ago
    Yeah .9 has more aliasing artifacts but has much better contrast and is easier on my eyes. Edit: Smooth gradients are much better in v1.0 though, look at the background differences here: https://raw.githubusercontent.com/bloc97/Anime4K/master/results/Comparisons/0.9-1.0/0_RC.png https://raw.githubusercontent.com/bloc97/Anime4K/master/resu...
miles 7y ago
Here is a colorized version: https://www.youtube.com/watch?v=EqbOhqXHL7E https://www.youtube.com/watch?v=EqbOhqXHL7E
- dredmorbius 7y ago
  Upscaled further: https://invidio.us/watch?v=9-hcR2rjg3o https://invidio.us/watch?v=9-hcR2rjg3o
  - jarrad2000 7y ago
    enhance!
- ekianjo 7y ago
  Not as convincing when clothes change colors between frames
  - sk0g 7y ago
    Makes it feel like a psychedelic trip without having to take anything though, so that's a positive. Maybe.
intellijdd 7y ago
At 35 seconds, the woman running with the child in the direction of the camera, her face is almost completely blurred out. If you look really hard you can see an outline of her nose and what not but overall it is interesting to see how the neural net works in this region of the video. Not surprising, but still drives the point home that good ML/AI algorithms have a bottleneck at the input data.
- mwcremer 7y ago
  I think she is wearing a kind of veil, as are several other women.
  - seanmcdirmid 7y ago
    I thought it was a veil also. This was France, so it could have been someone from one of their colonies at the time?
    - andrepd 7y ago
      Old French (and Italian and Portuguese and etc) women commonly wear veils/babushkas, even today.
      - Udik 7y ago
        They're wearing a veil that covers the entire face though. Eyes included- I assume a very thin see-through veil.
        azepoi 7y ago
        yes it's a see-through veil to protect their faces from the sun. Like this https://www.dhgate.com/product/wholesale-fashion-foldable-summer-veil-sun/451965835.html https://www.dhgate.com/product/wholesale-fashion-foldable-su... or http://www.coastalchrome.ca/womens-sun-hats-wkmovizpxj-6/abusun-sunscreen-hat-veil-sun-hat-summer-hat-outdoor-sunscreen-hat-for-women-b071gnb7hg-gmsmpjzkyy-1164.html http://www.coastalchrome.ca/womens-sun-hats-wkmovizpxj-6/abu...
Izkata 7y ago
Can someone point at specific things to see the upscaling at work? The two videos look the same to me, both when taken as a whole and when zoomed in to compare the same points in the videos...
- kick 7y ago
  Look at the ground in the thumbnail, for example, or the hill, for that matter, as well as the building in the background. And the fence, too. The details are much more distinct in the upscale. It actually surprised me, I was just expecting it to effectively be a de-speckle with edge sharpening, but it's exceeded my expectations.
  - grogenaut 7y ago
    It's sharper, but it's the type of sharper that happens in a dream, look at the text on the left building (quonset hut with a hole in the side). The text flutters as the net fever dreams context from it. Like a dream it has more detail the more you look but it also isn't real. It's like how I can read shit even why my eyes are very dry or it's beyond the extent of my vision, it's tossing it's guesses into a viewable object, but it's just extrapolations, like how I can tell the score in a nfl game across the bar, I know the game isn't 88 to 9, it's likely 28 to 7. This doesn't make it any less cool, just putting it in context. Zoom and enhance away.
    - dredmorbius 7y ago
      The inference resolves to a high-detail and plausible result, but not necessarily one that has any fidelity with the ground truth. Which is somewhat like a dream, in ways. Additional information -- say, a close-up image of the quonset structure's signage, or even knowing what the text was, might result in additional detail. A method that was specifically designed for extracting textual information out of blurred images (several exist, and they're frighteningly accurate -- any image-hiding based on only pixelation or blurring without additional noise introduction is not particularly secure) might also be able to identify a text from the original film source.
    - 7y ago
krackers 7y ago
Are there techniques to ensure temporal consistency when performing this kind of upscaling? Currently it seems like each frame was upscaled independently leading to weird artifacts.
- eyegor 7y ago
  > it seems like each frame was upscaled independently leading to weird artifacts Yep, they used an off the shelf nn (gigapixel) to upscale each individual frame. You could try to achieve your proposal using a guassian convolution windowed in 3d across frames. I suspect it may make a mess of the video's framerate, but the motion appears smooth enough that it might work.
  - ma2rten 7y ago
    You could also train a model that conditions on the previous frame in addition on the surrounding pixels.
  - egypturnash 7y ago
    It’s not as if the source material’s got a steady frame rate, there’s a lot of places where it jumps about 2-4 frames. Which is quite possibly due to lapses in the hand-cranking of the film through the camera back in 1896, or maybe due to frames getting lost when the printed reel of film got damaged and spliced in the hundred and twenty-odd years between then and now. Possibly both, one could probably work out some of the cranking speed variations by looking at how the train’s frame-to-frame motions change, at least until it’s come to a stop.
  - dTal 7y ago
    I like that the two sibling comments suggest "train a model" and "model a train", respectively.
  - BubRoss 7y ago
    That would just create a slightly better onion skinning of overlayed frames, it wouldn't align them.
- 7y ago
jcims 7y ago
There's a sort of swimming/breathing effect in the video that's nauseating to me in the same way that TV motion smoothing is. It looks like artifacts from trying to perfectly crop the video when the source film likely has become distorted over time, but it's hard to watch.
- Bnshsysjab 7y ago
  Could be the frame rate, I feel a similar way when I first started watching 60fps. Now I feel like it’s a more substantial jump than 4K->8k
  - bitL 7y ago
    Rolling shutter is one possible explanation of "wavy" effects if they digitized cinematic footage improperly. However, in this case I think we see aliasing effects of static Gigapixel AI.
  - gwbas1c 7y ago
    You can see the problems in the side-by-side clip at the end. A guy walks through the divide and the upscaled side of his body does not move with the un-enhanced side of his body.
- quickthrower2 7y ago
  Yes it did seem weird.
  - mxfh 7y ago
    For this very movie it would be presumeable quite trivial to correct the varying frame rate caused by the hand cranked camera with the help of the quite stable deceleration of the train and it's position to determine a corrected timestamp for each frame and then interpolate from there. It's fair to assumem that at the time, a skilled presenter knowing the material was able to compensate for that too to some extent, since the Cinematograph was camera and projector in one, both manually operated. Stable frame rates were simply not a thing in earliest film. https://en.wikipedia.org/wiki/Cinematograph https://en.wikipedia.org/wiki/Cinematograph
bitL 7y ago
Using Deep Learning models trained on static images is usually not a good idea. But one can simply extend the model to use 3D convolutions instead of 2D, use time as 3rd axis and then feed a sequence of images for training, getting much better results out of it without the wavy effects.
- londons_explore 7y ago
  And have the amount of training data and time scale from n^2 to n^3... That's why nobody has achieved good results with this yet...
  - bitL 7y ago
    Oh yes, but that might be comparable to attention-based models for NLP, complexity-wise. Using C3D is quite common since 2015 actually: http://vlg.cs.dartmouth.edu/c3d/ http://vlg.cs.dartmouth.edu/c3d/ https://arxiv.org/pdf/1412.0767.pdf https://arxiv.org/pdf/1412.0767.pdf Training a great model is expensive these days, FB routinely pays 6 figures per training run.
takk309 7y ago
A thought I had while reading about this was what is the limit to upscaling via this method? By that I mean if one where to just keep upscaling and had a monitor capable of displaying the image, how long until the image became unrecognizable? Surely upscaling an already upscaled image would result in some strange artifacts.
- macrolime 7y ago
  There's no limit when you upscale with neural nets. Make one good enough and it can fill in the blanks properly.
  - tlobes 7y ago
    Enhance
  - rnestler 7y ago
    > Make one good enough and it can fill in the blanks properly. I would say it can fill in the blanks likely or plausibly. But one can't guarantee that it is the proper or real content, since the neural net is imagining / inventing the upscaled version.
    - macrolime 7y ago
      Sure. I'm not saying it will represent reality 100%, but it could be good enough that you probably couldn't tell the difference. Current neural nets might perhaps not be able to upscale with no limit and still make it look good, but it's not like it can't be done. A good human artist could take a small grainy image and given enough effort draw an upscaled version that would be a very reasonable representation of what a larger image would have looked like, there's no reason to believe that artificial neural networks can't do the same.
- tedunangst 7y ago
  You'll end up seeing more and more of the training set. You might zoom in on a newspaper on a desk through a window across the street reflected off the windshield of a car and get perfectly legible text, but it's just going to have whatever lorem ipsum headlines were used to train the net.
agentbellnorm 7y ago
I think the framereate is what makes it feel more vivid. I didn’t experience that big of a difference with regards to resolution.
- l33tman 7y ago
  Yeah, it seems the NN in this particular implementation just does edge sharpening (almost like a "dumb" algorithm), the interior of textures and faces etc. still seem to lack detail.
- chooseaname 7y ago
  Right. And it makes some things worse. In the original the lady who runs across at about :27, you can make our her nose and mouth (barely). Her again in the upscaled version at :38 (it's hard to stop it at the exact same spot), she is virtually featureless.
dusted 7y ago
That's a lot worse than the original.
- bscphil 7y ago
  I agree. The upscaled version is both more smoothed and more mottled. This matches the "look" of a lot of low bitrate HD video, but I would argue that information has actually been lost in the upscaling process, not gained. It would be much easier to tell with before / after screenshots, I think, but at any rate I don't see a lot of reason to hope neural net upscaling will be useful for many "real world" purposes. The HFR was pretty convincing to me, however.
  - shirman 7y ago
    You can see comparison at the end of the video
    - bscphil 7y ago
      You're right. Unfortuate that the comparison clip is so short, and there's really no replacement for being able to flip between individual upscaled frames. From that very limited comparison, I think I still prefer the original.
  - pastrami_panda 7y ago
    I agree with your points, but I'd also argue that Nvidias DLSS implementation is a real world purpose, no?
    - bscphil 7y ago
      That's a great point. Using approaches like this as a less computationally complex method of resampling for the purpose of anti-aliasing (and not upscaling) seems worthwhile, definitely. I was thinking more of attempts to create neural nets that map from e.g. the set of 480p images to the set of 1080p images. The "best case" results seem to be trained on low bitrate HD video, which gives results that "look good" to many people (especially those who grew up watching Youtube) but in terms of real detail are worse than a simple upscale (with e.g. Lanczos). I haven't yet seen results where content aware upscaling provides a real improvement over "dumb" algorithms for this purpose.
Joe-Z-L 7y ago
We can finally csi pictures then!? Yeah
mrb 7y ago
Honestly, I watched the original and the 4k and don't really notice any big difference. The wider aspect ratio of the 4k is more pleasant, but this has nothing to do with the enhanced resolution.
- Thorrez 7y ago
  The aspect ratio looks like it was just accomplished by cropping.
- shirman 7y ago
  Have not you notice FPS boost?
galacticaactual 7y ago
Strange to think none of these people knew what was coming their way 18 years hence...
- excalibur 7y ago
  This was filmed at La Ciotat, on the southern coast of France. This region actually survived WWI relatively unscathed. It did see a bit more action in 1944. But the war would have been a major event in these people's lives, even if the location wasn't directly in the middle of it.
  - mytailorisrich 7y ago
    The town did not see any action but, as all towns in France, all fit men were sent to the front lines, and many never came back, as illustrated by the monuments listing the deads of WWI and WWII in all French towns and villages.
- ThinkingGuy 7y ago
  See also: "A Trip Down Market Street," filmed just 4 days before the 1906 San Francisco earthquake. https://en.wikipedia.org/wiki/A_Trip_Down_Market_Street https://en.wikipedia.org/wiki/A_Trip_Down_Market_Street
valiant-comma 7y ago
There are a number of speed-corrected vintage film clips on Youtube. Here’s an example from late-1890s France: https://youtu.be/NjDclfAFRB4 https://youtu.be/NjDclfAFRB4 While not upscaled, they are still pretty impressive, and interesting to watch.
- ekianjo 7y ago
  Peter Jackson made a whole ww1 movie using these techniques (documentary).
- bangboombang 7y ago
  Somewhat OT: Can someone enlighten me why exactly we need to "speed correct" them in the first place? It always struck me as odd that those old b/w clips were often shown playing too fast on TV. Isn't it more like they were simply recorded at lower fps rates? Then how can you call this speed correcting, when you merely just played them at the wrong frame rate before, just because TV is 25/30/50/60/59.4i² frames/fields per second and you didn't want to convert it in any way. And then suddenly a few years ago these speed corrected versions started to pop up on the internet, like "hey we just fixed the past, people didn't actually move twice as fast back then!" I guess I just don't like the term, but maybe there is more to it.
  - Tehdasi 7y ago
    Early film cameras were hand cranked, so the actual speed of the film was all over the place. I assume it was just too much of a hassle to manually correct the speed of these films, so they didn't bother.
    - felixyz 7y ago
      It wasn't all over the place (in professional productions). Rather, it was an important part of the craft of cinematography to adjust the speed of cranking to fit the mood of a scene. Experienced cinamatographers would vary the speed even during a take, envisioning how the film would look when projected at standard speed - slower crank, faster movement, and vice versa.
kstenerud 7y ago
One thing I've never understood is why people keep playing back old film at too high a frame rate. It seems that the film cameras of the day recorded at a lower-than-expected framerate, and the playback machines played at a higher frame rate, resulting in the telltale weird effect of everyone moving too fast. If they'd recorded audio along with it, everyone would be talking like chipmunks.
- anigbrowl 7y ago
  Correct. Hand-cranked cameras are obviously extra-problematic (though fun to work with), but the first automatically cranked cameras worked at 18fps. Most broadcast systems work at ~30, 25, or 24 fps (and there's a system for doing 24fps playback on 30fps TV). Retiming video used to be ridiculously tedious/labor-intensive, and it's only in the last 5 years or so that it's become sufficiently easy to pull it off and have it look good.
- Animats 7y ago
  Well, before everybody had frame interpolation, early 16FPS film tended to be copied to new film run at 24FPS. There were horrible hacks like "3-2 pulldown", to convert 24FPS film to 30 FPS TV. (30 FPS TV is 60 interlaced half-frames, so one 24FPS frame was shown for 3 half frames, then the next for 2 half-frames. This was done mechanically in the projector. Then that got copied to VHS...
- ghaff 7y ago
  Most of the footage we're talking about is before synchronized sound, so speech doesn't really come into the equation. But, as others have noted, adjusting film speed today in digital may just be a matter of hitting a few keys. However, with physical film stock, it's a lot harder. Something you might do for the restoration of a famous film, but not something you're going to do routinely.
01100011 7y ago
Totally unrelated but I'm curious if anyone else experienced this phenomenon. When I was a kid, people in old pictures and movies(say, pre 1920s) looked genuinely strange. The structure of their faces, the shape of their bodies... it just looked really weird to me. Now that I'm older, I find people in old media very familiar, like I know people who look enough like them and I can suddenly relate to them on a new level. I'm guessing my younger brain just hadn't seen enough faces yet. In any case it always struck me as odd.
- drcode 7y ago
  I think part of it is that people back then we're far more likely to not have diverse ancestors- the people in the OP video probably all had parents from the same small town, very different from our modern era. This probably enhanced a lot of distinct body and facial features that have mostly washed away in modern times. However, as an adult you've met enough people to still be able to recognize some of them.
- kingkawn 7y ago
  Culture transforms us
- adrianmonk 7y ago
  "Middle age is when you've met so many people that every new person you meet reminds you of someone else." - Ogden Nash
  - MaxBarraclough 7y ago
    How does he define old age?
    - deleted 7y ago
      [deleted]
    - nkrisc 7y ago
      When everyone you meet reminds you of someone dead?
krtong 7y ago
Oh this makes me happy. Im stoked with the 4k results. The gravel under the tracks looks so good. I feel like it's a definite improvement and I know it's only going to get better.
WalterBright 7y ago
This is pretty awesome! But those dresses the women wore look awfully hot an uncomfortable.
- mytailorisrich 7y ago
  Not least if you consider than it's very sunny and 30+ degrees (C) during summer in that area.
skywal_l 7y ago
Seeing this little girl, long born and dead, jumping and smiling got me quite emotional...
- feintruled 7y ago
  Watching old movies is always a very melancholy experience for me for this reason. To think that everyone in the film you are watching is dead, when they seem so alive before your eyes. (Interesting thing I read once about the Hanna-Barbera cartoons with the canned laugh track - it was using some old recording that had been knocking around since the 20s. So if you watch them now, the whole action is ringing with the laughter of the long dead. Now how's that for a macabre thought!)
shirman 7y ago
I am an author of this video, you can feel free to ask anything – I covered algos that I have used in video description, you can find them there
- janekm 7y ago
  I was wondering if you tried to run something like warp stabiliser on it first to avoid the wavy effect?
  - shirman 7y ago
    I wanted to see what could happen with a video without any manual editing, wavy effect could be removed in after effect, it is not a hard task, but just a question of a manual work :)
- jacobwilliamroy 7y ago
  Why did you crop it? 25% of the picture is gone.
  - dylan604 7y ago
    This is an age old question when converting 4:3 footage to 16:9. Do you keep the original aspect ratio (OAR) in the converted footage to keep the purists happy (small percentage of audience), or do you fill the frame with as much picture as possible to avoid mattes to keep the average content consumer happy (much larger % of audience)? At the end of the day, the producers will make the decision that they feel will please the largest group of viewers. I have personally been involved with these types of projects. The purists are the smaller section, but they are much more vocal on the internet. While reading forums and tweets and other soapboxes, these purists will eviscerate you publicly. However, using other metrics (sales figures, independent surveys, etc), the numbers show that the overall product is well received. There are so many people online that believe they know everything, but know nothing; yet these people are the most vocal. I was once accused of uprezing existing video tapes rather than re-scanning the film like the company claimed. Never mind that I was personally carrying the film to the transfer house. We took pictures of the employees with the film, took shots of the film on the shelves in the vaults, and wound up making "making of" type documentaries showing the process of the film on the telecine, interviewing the colorist transferring the film, showing the artists doing the restoration, and the sound department working with the audio from the original optical track, etc. Still, the know-it-alls kept on yelling.
p1necone 7y ago
The upscaled version looks worse to me. Seems like it's lost information, not gained.
isaacrolandson 7y ago
Call me crazy but I prefer watching the original
anjc 7y ago
Can people in this thread genuinely not see a difference between the before and after or are you just being contrary? The change in resolution should be clear enough, but the FPS change is amazing and gives a great sense of presence to me when watching. There are some jarring artifacts, such as the ladies' hats glitching at 10sec, but these artifacts are in the original and simply become clear in the upscaled version. It's very impressive. Side note, They Shall Not Grow Old is just stunning, with its restoration pipeline. I can't wait until all media can be updated in such a fashion, and I assume it's just a matter of time before e.g. YouTube and Spotify have filters to offer restored VHS footage, old phonograph recordings, etc
- aedron 7y ago
  I honestly couldn't see much difference. I did not study it in detail to find the differences, just a casual view of both clips. I did look at the comparison clips at the end of the 4K version but still didn't see much of a difference. A problem may be that the source material is simply too high quality to begin with? I watched the original first and thought "wow, 1896, this enhancement is great", only turns out that was the original clip. They should have chosen something that has more room for improvement (which would also give them a lot more interesting material to choose from).
  - anjc 7y ago
    > A problem may be that the source material is simply too high quality to begin with? I watched the original first and thought "wow, 1896, this enhancement is great", only turns out that was the original clip. Funny, I did the same
pcf 7y ago
Could this be used to generate "missing" frames in stop-motion animation as well? For example for this sequence from "The Terminator" (1984): https://www.youtube.com/watch?v=wSXl_XKXZAI https://www.youtube.com/watch?v=wSXl_XKXZAI
- Erlich_Bachman 7y ago
  There is a user-friendly mature software for this already (which technically did not even use ML in the recent sense because it is much older): SVP project. You can connect it to many players and play any video of your choice in realtime. Then there are high quality tools in video editing software like Premiere or Davinci Resolve which do this also (pixel flow framerate adjustment)... Then there is hardware accelerated version of this in NVIDIA graphics cards which can be used in some video players too... This is mostly a solved problem, it has existed for many years, no real need for ML here. (Possibly for some quality improvements?)
melling 7y ago
I’ve seen lots of old sports and news footage from the 1970’s that looks bad. The color looks particularly bad, for example. It would be nice to automatically fix that.
brownbat 7y ago
If we're killing garbage in garbage out, that's going to take us some weird places.
onhn 7y ago
It may appear higher resolution but I don’t enjoy looking at the extra details knowing that they are probably fictitious.
Renaud 7y ago
I can't help to think that we will -one day, probably soon- have enough 3D face models to just replace those in these movies with some that will fit the original version. The resulting faces won't be an exact match due to the low quality of the original, but unless you knew these people, no-one will be able to tell. Same with pretty much every other aspect of the scene: have good models that fit the original scene and re-create a full 3D version of the whole movie, at any level of definition. Once this process can be automated, we'll be able to recreate all the classics, shot for shot.
- jacobwilliamroy 7y ago
  For what purpose?
  - hammerbrostime 7y ago
    VR and re-editing come to mind.
- dylan604 7y ago
  It has become pretty standard in Hollywood where at the beginning of a production, the actors have headshots taken from various angles specifically for use in post. Enough actors have died during production to make this necessary to complete the project. I wouldn't be surprised if it has become a mandatory practice insurance companies require (this is pure speculation on my part) to minimize payouts.
chriskanan 7y ago
The same approach is being used to enhance the textures of older games, ranging from Doom, Witcher 2, Final fantasy 7, and others. People have turned them into mods [0,1]. I actually showed a few examples on day one of the deep learning course I'm teaching this semester as an emergent application of the technology. [0] https://www.pcgamer.com/heretic-looks-stunning-with-neural-network-texture-improvements/ https://www.pcgamer.com/heretic-looks-stunning-with-neural-n... [1] https://www.pcgamer.com/AI-upscale-mod-list/ https://www.pcgamer.com/AI-upscale-mod-list/
- stared 7y ago
  See "AI Neural Networks being used to generate HQ textures for older games (You can do it yourself!)" https://www.resetera.com/threads/ai-neural-networks-being-used-to-generate-hq-textures-for-older-games-you-can-do-it-yourself.88272/ https://www.resetera.com/threads/ai-neural-networks-being-us... (Dec 2018) - people create it for old games, it is easy to set in on one's own computer, or on Colab. Also, super-resolution is being used not only to statically enhance textures, but also - work on the flight. See Atomic Heart RTX Tech Demo, with ray-tracing and deep learning super-sampling (DLSS) https://mundfish.com/en/rtxtechdemo/ https://mundfish.com/en/rtxtechdemo/.
- metadaemon 7y ago
  I'm pumped about the Dark Souls Remastered up-scaling! Thanks for the links.
72deluxe 7y ago
It states that humanity's standards for video have risen in 125 years. Viewing the film, it seems that the standard for clothing to wear whilst going outside has done the opposite, quite drastically.
- cm2187 7y ago
  And if there was sound, you could argue the same with the quality of the language. Idiocracy!
  - raverbashing 7y ago
    From a linguistic perspective this couldn't be further from the truth. We only perceive older language as more polite, but it doesn't mean it was. What matters is the level of rudeness perceived by the receiver at the time. If someone was called a 'Scoundrel' they might have been shook while today it's a laughable insult.
    - 72deluxe 7y ago
      You scoundrel!
    - cm2187 7y ago
      I completely disagree. If you look at speeches or interviews from the 30s or the 40s, in French or English, the quality of the spoken language is significantly better than today. Sentences are better constructed, richer vocabulary. And if you look at XVIII french literature, it is night and day with anything written today.
      - raverbashing 7y ago
        > If you look at speeches or interviews from the 30s or the 40s, in French or English, the quality of the spoken language is significantly better than today... And if you look at XVIII french literature There's probably some selection bias there. I'm not disagreeing there might have been a reduction in vocabulary, etc, but we tend to see older (usage) words as more erudite as well. Compare the vocabulary/grammar on social media with current newspapers or a book and there's a big difference as well.
amelius 7y ago
Has anyone else experienced this: when watching old movies (or just black/white movies), it feels uncomfortable for only the first few minutes, and then you completely forget about the bad image quality. It's like the brain performs the upscaling/coloring for you.
- mlvljr 7y ago
  Via a neural network, no less!
- reportgunner 7y ago
  What makes the difference for me is the sound quality. If the sound track is synced to the video well, I don't care too much about the video quality/depth/colors/contrast.
- LeonM 7y ago
  I remember experiencing this with the stop-motion movie 'chicken run' from 2000 [0]. The movie has a rather low frame-rate (due to the stop-motion production) and when I started watching my immediate thought was: I can't watch this, this is terribly stuttery. But after about 10 minutes or so my brain adjusted and I no longer noticed the stuttering. [0] https://en.wikipedia.org/wiki/Chicken_Run https://en.wikipedia.org/wiki/Chicken_Run
- GordonS 7y ago
  I'm colour blind, and strangely when I start watching something in black and white, I don't know it's in black and white! - I only realise when someone else mentions it, and even then it takes a while for me to see it. It's weird, like my brain is automatically filling in colours that aren't there.
anonymous_fun 7y ago
Math upscale'd it, and my internet connection downscale'd it. I suppose it was math on either end.
ubercow13 7y ago
It is common in video player software these days to use neural network based filters to upscale and downscale, eg see https://artoriuz.github.io/mpv_upscaling.html https://artoriuz.github.io/mpv_upscaling.html
squarefoot 7y ago
I wonder if a similar technique could be used to turn old 4/3 videos to widescreen just by learning the surroundings of a scene through other shots, then mapping them back and forth to build a complete widescreen scene on all frames.
- aphextim 7y ago
  Off the top of my head a reverse content aware type system maybe could work. https://www.theverge.com/2019/4/3/18293526/adobe-after-effects-content-aware-fill-remove-unwanted-objects-videos https://www.theverge.com/2019/4/3/18293526/adobe-after-effec...
- KMnO4 7y ago
  I've watched a few movies in what's called ScreenX[0]. It's a cinema experience where they project the movie on the main (front) screen, and project the surroundings on the two side walls. The result is an immersive viewing experience. Obviously the film is not recorded in the ridiculously wide aspect ratio, so usually the side images are computer generated. For landscape imagery, this is fairly seamless. You can't really tell that the images aren't real. However, during a scene with people, the side walls projected static images that drifted around slightly (as if they were cardboard cutouts). I guess the idea is that most things happening in your peripheral vision are lost anyway, so no need to make them detailed. [0]: https://www.cineplex.com/Theatres/ScreenX https://www.cineplex.com/Theatres/ScreenX
- dylan604 7y ago
  Depending on the specific shot, you can borrow data from frames before/after the current frame rather than just flat out recreating from nothing. For instance, a shot where the camera pans left to right you can borrow frames from later in the shot to fill in the content on the right half of the image. Or a shot that zooms in/out will have actual footage from the scene that can be used as well. This kind of technique is used frequently for other things as well like when removing objects from a scene or even fixing/restoring film with scratches, holes, dust, etc.
monktastic1 7y ago
Anyone know of similar projects for audio? Music within a genre tends to be similar enough that you ought to be able to upsample. I tried doing this once myself but got nowhere.
the_dripper 7y ago
AMAZING!
cal97g 7y ago
Okay, this application of ML is actually cool.
jtbayly 7y ago
Am I the only one that finds the "improved" version much worse? There are digital artifacts all over the place, like the steam at the beginning, the people's faces, any place there is movement. I see that some things don't move as "jerky" but I think the original is far superior to watch. Edited to add: Here's a link to a quick comparison image. These aren't exactly the same frame or size, but close enough to see what I'm talking about. https://imgur.com/a/A3roH54 https://imgur.com/a/A3roH54
- godson_drafty 7y ago
  I agree, the improved version is worse. At first glance, it looks better than the original, but then numerous artifacts become apparent. For example, check out the little girl's face. It's jumbled in the "improved" version, but very clear in the original. It looks like most of the perceived improvement comes simply from the upgrade to 60fps, which is achieved trivially through interpolation. I wonder if the video would have gotten as much attention had AI not been mentioned.
- tachyonbeam 7y ago
  Someone posted a colorized[0] version and it had somewhat "jumpy" colors that change over time. I think one problem is you need temporal consistency for this to really work properly. If you pick a color for an object, or you upscale an object a certain way, it has to stay that way over a potentially unlimited number of frames. That's already hard to achieve because neural networks are typically bad at long-term dependency, but it's even more difficult if the object moves, because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with. At the moment, neural networks don't have sophisticated models of the world. [0]: https://www.youtube.com/watch?v=EqbOhqXHL7E https://www.youtube.com/watch?v=EqbOhqXHL7E
  - sangnoir 7y ago
    >...because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with. I agree with most of what you said, but disagree with the need to know underlying fundamentals: most 5 year olds can throw and catch a ball without needing to understand calculus or Newton's laws of gravity.
chriscatoya 7y ago
I think it's great we're developing this capability, but I prefer the original-- we're seeing the film as it was originally seen then. The captured scene isn't made better by higher fidelity, it's the opposite for me. As with most advancements in technology and art, we'll see a proliferation of people using the new technique with mixed results at first. It's the shiny new toy. Then over time, people will figure out how to use it and apply it with intent. The upscaling projects we've seen in gaming have generally produced spectacular results, probably because the intent in gaming graphics has always been for the sort of pixel density we're finally able to start achieving. Remember the huge disparity between the box art and the game? lol. Gaming aside, I don't think we've left the toy phase just yet.

349 points by eyegor 7y ago

eyegor 7y ago

Tangentially related, a real-time video upscaler: https://github.com/bloc97/Anime4K https://github.com/bloc97/Anime4K

marzell 7y ago
IMO in the v1.0 comparisons, v0.9 is sharper in every case.
- jcims 7y ago
  Yeah .9 has more aliasing artifacts but has much better contrast and is easier on my eyes. Edit: Smooth gradients are much better in v1.0 though, look at the background differences here: https://raw.githubusercontent.com/bloc97/Anime4K/master/results/Comparisons/0.9-1.0/0_RC.png https://raw.githubusercontent.com/bloc97/Anime4K/master/resu...

miles 7y ago

Here is a colorized version: https://www.youtube.com/watch?v=EqbOhqXHL7E https://www.youtube.com/watch?v=EqbOhqXHL7E

dredmorbius 7y ago
Upscaled further: https://invidio.us/watch?v=9-hcR2rjg3o https://invidio.us/watch?v=9-hcR2rjg3o
- jarrad2000 7y ago
  enhance!
ekianjo 7y ago
Not as convincing when clothes change colors between frames
- sk0g 7y ago
  Makes it feel like a psychedelic trip without having to take anything though, so that's a positive. Maybe.

intellijdd 7y ago

At 35 seconds, the woman running with the child in the direction of the camera, her face is almost completely blurred out. If you look really hard you can see an outline of her nose and what not but overall it is interesting to see how the neural net works in this region of the video. Not surprising, but still drives the point home that good ML/AI algorithms have a bottleneck at the input data.

mwcremer 7y ago
I think she is wearing a kind of veil, as are several other women.
- seanmcdirmid 7y ago
  I thought it was a veil also. This was France, so it could have been someone from one of their colonies at the time?
  - andrepd 7y ago
    Old French (and Italian and Portuguese and etc) women commonly wear veils/babushkas, even today.
    - Udik 7y ago
      They're wearing a veil that covers the entire face though. Eyes included- I assume a very thin see-through veil.
      - azepoi 7y ago
        yes it's a see-through veil to protect their faces from the sun. Like this https://www.dhgate.com/product/wholesale-fashion-foldable-summer-veil-sun/451965835.html https://www.dhgate.com/product/wholesale-fashion-foldable-su... or http://www.coastalchrome.ca/womens-sun-hats-wkmovizpxj-6/abusun-sunscreen-hat-veil-sun-hat-summer-hat-outdoor-sunscreen-hat-for-women-b071gnb7hg-gmsmpjzkyy-1164.html http://www.coastalchrome.ca/womens-sun-hats-wkmovizpxj-6/abu...

Izkata 7y ago

Can someone point at specific things to see the upscaling at work? The two videos look the same to me, both when taken as a whole and when zoomed in to compare the same points in the videos...

kick 7y ago
Look at the ground in the thumbnail, for example, or the hill, for that matter, as well as the building in the background. And the fence, too. The details are much more distinct in the upscale. It actually surprised me, I was just expecting it to effectively be a de-speckle with edge sharpening, but it's exceeded my expectations.
- grogenaut 7y ago
  It's sharper, but it's the type of sharper that happens in a dream, look at the text on the left building (quonset hut with a hole in the side). The text flutters as the net fever dreams context from it. Like a dream it has more detail the more you look but it also isn't real. It's like how I can read shit even why my eyes are very dry or it's beyond the extent of my vision, it's tossing it's guesses into a viewable object, but it's just extrapolations, like how I can tell the score in a nfl game across the bar, I know the game isn't 88 to 9, it's likely 28 to 7. This doesn't make it any less cool, just putting it in context. Zoom and enhance away.
  - dredmorbius 7y ago
    The inference resolves to a high-detail and plausible result, but not necessarily one that has any fidelity with the ground truth. Which is somewhat like a dream, in ways. Additional information -- say, a close-up image of the quonset structure's signage, or even knowing what the text was, might result in additional detail. A method that was specifically designed for extracting textual information out of blurred images (several exist, and they're frighteningly accurate -- any image-hiding based on only pixelation or blurring without additional noise introduction is not particularly secure) might also be able to identify a text from the original film source.
  - 7y ago

krackers 7y ago

Are there techniques to ensure temporal consistency when performing this kind of upscaling? Currently it seems like each frame was upscaled independently leading to weird artifacts.

eyegor 7y ago
> it seems like each frame was upscaled independently leading to weird artifacts Yep, they used an off the shelf nn (gigapixel) to upscale each individual frame. You could try to achieve your proposal using a guassian convolution windowed in 3d across frames. I suspect it may make a mess of the video's framerate, but the motion appears smooth enough that it might work.
- ma2rten 7y ago
  You could also train a model that conditions on the previous frame in addition on the surrounding pixels.
- egypturnash 7y ago
  It’s not as if the source material’s got a steady frame rate, there’s a lot of places where it jumps about 2-4 frames. Which is quite possibly due to lapses in the hand-cranking of the film through the camera back in 1896, or maybe due to frames getting lost when the printed reel of film got damaged and spliced in the hundred and twenty-odd years between then and now. Possibly both, one could probably work out some of the cranking speed variations by looking at how the train’s frame-to-frame motions change, at least until it’s come to a stop.
- dTal 7y ago
  I like that the two sibling comments suggest "train a model" and "model a train", respectively.
- BubRoss 7y ago
  That would just create a slightly better onion skinning of overlayed frames, it wouldn't align them.
7y ago

jcims 7y ago

There's a sort of swimming/breathing effect in the video that's nauseating to me in the same way that TV motion smoothing is. It looks like artifacts from trying to perfectly crop the video when the source film likely has become distorted over time, but it's hard to watch.

Bnshsysjab 7y ago
Could be the frame rate, I feel a similar way when I first started watching 60fps. Now I feel like it’s a more substantial jump than 4K->8k
- bitL 7y ago
  Rolling shutter is one possible explanation of "wavy" effects if they digitized cinematic footage improperly. However, in this case I think we see aliasing effects of static Gigapixel AI.
- gwbas1c 7y ago
  You can see the problems in the side-by-side clip at the end. A guy walks through the divide and the upscaled side of his body does not move with the un-enhanced side of his body.
quickthrower2 7y ago
Yes it did seem weird.
- mxfh 7y ago
  For this very movie it would be presumeable quite trivial to correct the varying frame rate caused by the hand cranked camera with the help of the quite stable deceleration of the train and it's position to determine a corrected timestamp for each frame and then interpolate from there. It's fair to assumem that at the time, a skilled presenter knowing the material was able to compensate for that too to some extent, since the Cinematograph was camera and projector in one, both manually operated. Stable frame rates were simply not a thing in earliest film. https://en.wikipedia.org/wiki/Cinematograph https://en.wikipedia.org/wiki/Cinematograph

bitL 7y ago

Using Deep Learning models trained on static images is usually not a good idea. But one can simply extend the model to use 3D convolutions instead of 2D, use time as 3rd axis and then feed a sequence of images for training, getting much better results out of it without the wavy effects.

londons_explore 7y ago
And have the amount of training data and time scale from n^2 to n^3... That's why nobody has achieved good results with this yet...
- bitL 7y ago
  Oh yes, but that might be comparable to attention-based models for NLP, complexity-wise. Using C3D is quite common since 2015 actually: http://vlg.cs.dartmouth.edu/c3d/ http://vlg.cs.dartmouth.edu/c3d/ https://arxiv.org/pdf/1412.0767.pdf https://arxiv.org/pdf/1412.0767.pdf Training a great model is expensive these days, FB routinely pays 6 figures per training run.

takk309 7y ago

A thought I had while reading about this was what is the limit to upscaling via this method? By that I mean if one where to just keep upscaling and had a monitor capable of displaying the image, how long until the image became unrecognizable? Surely upscaling an already upscaled image would result in some strange artifacts.

macrolime 7y ago
There's no limit when you upscale with neural nets. Make one good enough and it can fill in the blanks properly.
- tlobes 7y ago
  Enhance
- rnestler 7y ago
  > Make one good enough and it can fill in the blanks properly. I would say it can fill in the blanks likely or plausibly. But one can't guarantee that it is the proper or real content, since the neural net is imagining / inventing the upscaled version.
  - macrolime 7y ago
    Sure. I'm not saying it will represent reality 100%, but it could be good enough that you probably couldn't tell the difference. Current neural nets might perhaps not be able to upscale with no limit and still make it look good, but it's not like it can't be done. A good human artist could take a small grainy image and given enough effort draw an upscaled version that would be a very reasonable representation of what a larger image would have looked like, there's no reason to believe that artificial neural networks can't do the same.
tedunangst 7y ago
You'll end up seeing more and more of the training set. You might zoom in on a newspaper on a desk through a window across the street reflected off the windshield of a car and get perfectly legible text, but it's just going to have whatever lorem ipsum headlines were used to train the net.

agentbellnorm 7y ago

I think the framereate is what makes it feel more vivid. I didn’t experience that big of a difference with regards to resolution.

l33tman 7y ago
Yeah, it seems the NN in this particular implementation just does edge sharpening (almost like a "dumb" algorithm), the interior of textures and faces etc. still seem to lack detail.
chooseaname 7y ago
Right. And it makes some things worse. In the original the lady who runs across at about :27, you can make our her nose and mouth (barely). Her again in the upscaled version at :38 (it's hard to stop it at the exact same spot), she is virtually featureless.

dusted 7y ago

That's a lot worse than the original.

bscphil 7y ago
I agree. The upscaled version is both more smoothed and more mottled. This matches the "look" of a lot of low bitrate HD video, but I would argue that information has actually been lost in the upscaling process, not gained. It would be much easier to tell with before / after screenshots, I think, but at any rate I don't see a lot of reason to hope neural net upscaling will be useful for many "real world" purposes. The HFR was pretty convincing to me, however.
- shirman 7y ago
  You can see comparison at the end of the video
  - bscphil 7y ago
    You're right. Unfortuate that the comparison clip is so short, and there's really no replacement for being able to flip between individual upscaled frames. From that very limited comparison, I think I still prefer the original.
- pastrami_panda 7y ago
  I agree with your points, but I'd also argue that Nvidias DLSS implementation is a real world purpose, no?
  - bscphil 7y ago
    That's a great point. Using approaches like this as a less computationally complex method of resampling for the purpose of anti-aliasing (and not upscaling) seems worthwhile, definitely. I was thinking more of attempts to create neural nets that map from e.g. the set of 480p images to the set of 1080p images. The "best case" results seem to be trained on low bitrate HD video, which gives results that "look good" to many people (especially those who grew up watching Youtube) but in terms of real detail are worse than a simple upscale (with e.g. Lanczos). I haven't yet seen results where content aware upscaling provides a real improvement over "dumb" algorithms for this purpose.

Joe-Z-L 7y ago

We can finally csi pictures then!? Yeah

mrb 7y ago

Honestly, I watched the original and the 4k and don't really notice any big difference. The wider aspect ratio of the 4k is more pleasant, but this has nothing to do with the enhanced resolution.

Thorrez 7y ago
The aspect ratio looks like it was just accomplished by cropping.
shirman 7y ago
Have not you notice FPS boost?

galacticaactual 7y ago

Strange to think none of these people knew what was coming their way 18 years hence...

excalibur 7y ago
This was filmed at La Ciotat, on the southern coast of France. This region actually survived WWI relatively unscathed. It did see a bit more action in 1944. But the war would have been a major event in these people's lives, even if the location wasn't directly in the middle of it.
- mytailorisrich 7y ago
  The town did not see any action but, as all towns in France, all fit men were sent to the front lines, and many never came back, as illustrated by the monuments listing the deads of WWI and WWII in all French towns and villages.
ThinkingGuy 7y ago
See also: "A Trip Down Market Street," filmed just 4 days before the 1906 San Francisco earthquake. https://en.wikipedia.org/wiki/A_Trip_Down_Market_Street https://en.wikipedia.org/wiki/A_Trip_Down_Market_Street

valiant-comma 7y ago

There are a number of speed-corrected vintage film clips on Youtube. Here’s an example from late-1890s France: https://youtu.be/NjDclfAFRB4 https://youtu.be/NjDclfAFRB4 While not upscaled, they are still pretty impressive, and interesting to watch.

ekianjo 7y ago
Peter Jackson made a whole ww1 movie using these techniques (documentary).
bangboombang 7y ago
Somewhat OT: Can someone enlighten me why exactly we need to "speed correct" them in the first place? It always struck me as odd that those old b/w clips were often shown playing too fast on TV. Isn't it more like they were simply recorded at lower fps rates? Then how can you call this speed correcting, when you merely just played them at the wrong frame rate before, just because TV is 25/30/50/60/59.4i² frames/fields per second and you didn't want to convert it in any way. And then suddenly a few years ago these speed corrected versions started to pop up on the internet, like "hey we just fixed the past, people didn't actually move twice as fast back then!" I guess I just don't like the term, but maybe there is more to it.
- Tehdasi 7y ago
  Early film cameras were hand cranked, so the actual speed of the film was all over the place. I assume it was just too much of a hassle to manually correct the speed of these films, so they didn't bother.
  - felixyz 7y ago
    It wasn't all over the place (in professional productions). Rather, it was an important part of the craft of cinematography to adjust the speed of cranking to fit the mood of a scene. Experienced cinamatographers would vary the speed even during a take, envisioning how the film would look when projected at standard speed - slower crank, faster movement, and vice versa.

kstenerud 7y ago

One thing I've never understood is why people keep playing back old film at too high a frame rate. It seems that the film cameras of the day recorded at a lower-than-expected framerate, and the playback machines played at a higher frame rate, resulting in the telltale weird effect of everyone moving too fast. If they'd recorded audio along with it, everyone would be talking like chipmunks.

anigbrowl 7y ago
Correct. Hand-cranked cameras are obviously extra-problematic (though fun to work with), but the first automatically cranked cameras worked at 18fps. Most broadcast systems work at ~30, 25, or 24 fps (and there's a system for doing 24fps playback on 30fps TV). Retiming video used to be ridiculously tedious/labor-intensive, and it's only in the last 5 years or so that it's become sufficiently easy to pull it off and have it look good.
Animats 7y ago
Well, before everybody had frame interpolation, early 16FPS film tended to be copied to new film run at 24FPS. There were horrible hacks like "3-2 pulldown", to convert 24FPS film to 30 FPS TV. (30 FPS TV is 60 interlaced half-frames, so one 24FPS frame was shown for 3 half frames, then the next for 2 half-frames. This was done mechanically in the projector. Then that got copied to VHS...
ghaff 7y ago
Most of the footage we're talking about is before synchronized sound, so speech doesn't really come into the equation. But, as others have noted, adjusting film speed today in digital may just be a matter of hitting a few keys. However, with physical film stock, it's a lot harder. Something you might do for the restoration of a famous film, but not something you're going to do routinely.

01100011 7y ago

Totally unrelated but I'm curious if anyone else experienced this phenomenon. When I was a kid, people in old pictures and movies(say, pre 1920s) looked genuinely strange. The structure of their faces, the shape of their bodies... it just looked really weird to me. Now that I'm older, I find people in old media very familiar, like I know people who look enough like them and I can suddenly relate to them on a new level. I'm guessing my younger brain just hadn't seen enough faces yet. In any case it always struck me as odd.

drcode 7y ago
I think part of it is that people back then we're far more likely to not have diverse ancestors- the people in the OP video probably all had parents from the same small town, very different from our modern era. This probably enhanced a lot of distinct body and facial features that have mostly washed away in modern times. However, as an adult you've met enough people to still be able to recognize some of them.
kingkawn 7y ago
Culture transforms us
adrianmonk 7y ago
"Middle age is when you've met so many people that every new person you meet reminds you of someone else." - Ogden Nash
- MaxBarraclough 7y ago
  How does he define old age?
  - deleted 7y ago
    [deleted]
  - nkrisc 7y ago
    When everyone you meet reminds you of someone dead?

krtong 7y ago

Oh this makes me happy. Im stoked with the 4k results. The gravel under the tracks looks so good. I feel like it's a definite improvement and I know it's only going to get better.

WalterBright 7y ago

This is pretty awesome! But those dresses the women wore look awfully hot an uncomfortable.

mytailorisrich 7y ago
Not least if you consider than it's very sunny and 30+ degrees (C) during summer in that area.

skywal_l 7y ago

Seeing this little girl, long born and dead, jumping and smiling got me quite emotional...

feintruled 7y ago
Watching old movies is always a very melancholy experience for me for this reason. To think that everyone in the film you are watching is dead, when they seem so alive before your eyes. (Interesting thing I read once about the Hanna-Barbera cartoons with the canned laugh track - it was using some old recording that had been knocking around since the 20s. So if you watch them now, the whole action is ringing with the laughter of the long dead. Now how's that for a macabre thought!)

shirman 7y ago

I am an author of this video, you can feel free to ask anything – I covered algos that I have used in video description, you can find them there

janekm 7y ago
I was wondering if you tried to run something like warp stabiliser on it first to avoid the wavy effect?
- shirman 7y ago
  I wanted to see what could happen with a video without any manual editing, wavy effect could be removed in after effect, it is not a hard task, but just a question of a manual work :)
jacobwilliamroy 7y ago
Why did you crop it? 25% of the picture is gone.
- dylan604 7y ago
  This is an age old question when converting 4:3 footage to 16:9. Do you keep the original aspect ratio (OAR) in the converted footage to keep the purists happy (small percentage of audience), or do you fill the frame with as much picture as possible to avoid mattes to keep the average content consumer happy (much larger % of audience)? At the end of the day, the producers will make the decision that they feel will please the largest group of viewers. I have personally been involved with these types of projects. The purists are the smaller section, but they are much more vocal on the internet. While reading forums and tweets and other soapboxes, these purists will eviscerate you publicly. However, using other metrics (sales figures, independent surveys, etc), the numbers show that the overall product is well received. There are so many people online that believe they know everything, but know nothing; yet these people are the most vocal. I was once accused of uprezing existing video tapes rather than re-scanning the film like the company claimed. Never mind that I was personally carrying the film to the transfer house. We took pictures of the employees with the film, took shots of the film on the shelves in the vaults, and wound up making "making of" type documentaries showing the process of the film on the telecine, interviewing the colorist transferring the film, showing the artists doing the restoration, and the sound department working with the audio from the original optical track, etc. Still, the know-it-alls kept on yelling.

p1necone 7y ago

The upscaled version looks worse to me. Seems like it's lost information, not gained.

isaacrolandson 7y ago

Call me crazy but I prefer watching the original

anjc 7y ago

Can people in this thread genuinely not see a difference between the before and after or are you just being contrary? The change in resolution should be clear enough, but the FPS change is amazing and gives a great sense of presence to me when watching. There are some jarring artifacts, such as the ladies' hats glitching at 10sec, but these artifacts are in the original and simply become clear in the upscaled version. It's very impressive. Side note, They Shall Not Grow Old is just stunning, with its restoration pipeline. I can't wait until all media can be updated in such a fashion, and I assume it's just a matter of time before e.g. YouTube and Spotify have filters to offer restored VHS footage, old phonograph recordings, etc

aedron 7y ago
I honestly couldn't see much difference. I did not study it in detail to find the differences, just a casual view of both clips. I did look at the comparison clips at the end of the 4K version but still didn't see much of a difference. A problem may be that the source material is simply too high quality to begin with? I watched the original first and thought "wow, 1896, this enhancement is great", only turns out that was the original clip. They should have chosen something that has more room for improvement (which would also give them a lot more interesting material to choose from).
- anjc 7y ago
  > A problem may be that the source material is simply too high quality to begin with? I watched the original first and thought "wow, 1896, this enhancement is great", only turns out that was the original clip. Funny, I did the same

pcf 7y ago

Could this be used to generate "missing" frames in stop-motion animation as well? For example for this sequence from "The Terminator" (1984): https://www.youtube.com/watch?v=wSXl_XKXZAI https://www.youtube.com/watch?v=wSXl_XKXZAI

Erlich_Bachman 7y ago
There is a user-friendly mature software for this already (which technically did not even use ML in the recent sense because it is much older): SVP project. You can connect it to many players and play any video of your choice in realtime. Then there are high quality tools in video editing software like Premiere or Davinci Resolve which do this also (pixel flow framerate adjustment)... Then there is hardware accelerated version of this in NVIDIA graphics cards which can be used in some video players too... This is mostly a solved problem, it has existed for many years, no real need for ML here. (Possibly for some quality improvements?)

melling 7y ago

I’ve seen lots of old sports and news footage from the 1970’s that looks bad. The color looks particularly bad, for example. It would be nice to automatically fix that.

brownbat 7y ago

If we're killing garbage in garbage out, that's going to take us some weird places.

onhn 7y ago

It may appear higher resolution but I don’t enjoy looking at the extra details knowing that they are probably fictitious.

Renaud 7y ago

I can't help to think that we will -one day, probably soon- have enough 3D face models to just replace those in these movies with some that will fit the original version. The resulting faces won't be an exact match due to the low quality of the original, but unless you knew these people, no-one will be able to tell. Same with pretty much every other aspect of the scene: have good models that fit the original scene and re-create a full 3D version of the whole movie, at any level of definition. Once this process can be automated, we'll be able to recreate all the classics, shot for shot.

jacobwilliamroy 7y ago
For what purpose?
- hammerbrostime 7y ago
  VR and re-editing come to mind.
dylan604 7y ago
It has become pretty standard in Hollywood where at the beginning of a production, the actors have headshots taken from various angles specifically for use in post. Enough actors have died during production to make this necessary to complete the project. I wouldn't be surprised if it has become a mandatory practice insurance companies require (this is pure speculation on my part) to minimize payouts.

chriskanan 7y ago

The same approach is being used to enhance the textures of older games, ranging from Doom, Witcher 2, Final fantasy 7, and others. People have turned them into mods [0,1]. I actually showed a few examples on day one of the deep learning course I'm teaching this semester as an emergent application of the technology. [0] https://www.pcgamer.com/heretic-looks-stunning-with-neural-network-texture-improvements/ https://www.pcgamer.com/heretic-looks-stunning-with-neural-n... [1] https://www.pcgamer.com/AI-upscale-mod-list/ https://www.pcgamer.com/AI-upscale-mod-list/

stared 7y ago
See "AI Neural Networks being used to generate HQ textures for older games (You can do it yourself!)" https://www.resetera.com/threads/ai-neural-networks-being-used-to-generate-hq-textures-for-older-games-you-can-do-it-yourself.88272/ https://www.resetera.com/threads/ai-neural-networks-being-us... (Dec 2018) - people create it for old games, it is easy to set in on one's own computer, or on Colab. Also, super-resolution is being used not only to statically enhance textures, but also - work on the flight. See Atomic Heart RTX Tech Demo, with ray-tracing and deep learning super-sampling (DLSS) https://mundfish.com/en/rtxtechdemo/ https://mundfish.com/en/rtxtechdemo/.
metadaemon 7y ago
I'm pumped about the Dark Souls Remastered up-scaling! Thanks for the links.

72deluxe 7y ago

It states that humanity's standards for video have risen in 125 years. Viewing the film, it seems that the standard for clothing to wear whilst going outside has done the opposite, quite drastically.

cm2187 7y ago
And if there was sound, you could argue the same with the quality of the language. Idiocracy!
- raverbashing 7y ago
  From a linguistic perspective this couldn't be further from the truth. We only perceive older language as more polite, but it doesn't mean it was. What matters is the level of rudeness perceived by the receiver at the time. If someone was called a 'Scoundrel' they might have been shook while today it's a laughable insult.
  - 72deluxe 7y ago
    You scoundrel!
  - cm2187 7y ago
    I completely disagree. If you look at speeches or interviews from the 30s or the 40s, in French or English, the quality of the spoken language is significantly better than today. Sentences are better constructed, richer vocabulary. And if you look at XVIII french literature, it is night and day with anything written today.
    - raverbashing 7y ago
      > If you look at speeches or interviews from the 30s or the 40s, in French or English, the quality of the spoken language is significantly better than today... And if you look at XVIII french literature There's probably some selection bias there. I'm not disagreeing there might have been a reduction in vocabulary, etc, but we tend to see older (usage) words as more erudite as well. Compare the vocabulary/grammar on social media with current newspapers or a book and there's a big difference as well.

amelius 7y ago

Has anyone else experienced this: when watching old movies (or just black/white movies), it feels uncomfortable for only the first few minutes, and then you completely forget about the bad image quality. It's like the brain performs the upscaling/coloring for you.

mlvljr 7y ago
Via a neural network, no less!
reportgunner 7y ago
What makes the difference for me is the sound quality. If the sound track is synced to the video well, I don't care too much about the video quality/depth/colors/contrast.
LeonM 7y ago
I remember experiencing this with the stop-motion movie 'chicken run' from 2000 [0]. The movie has a rather low frame-rate (due to the stop-motion production) and when I started watching my immediate thought was: I can't watch this, this is terribly stuttery. But after about 10 minutes or so my brain adjusted and I no longer noticed the stuttering. [0] https://en.wikipedia.org/wiki/Chicken_Run https://en.wikipedia.org/wiki/Chicken_Run
GordonS 7y ago
I'm colour blind, and strangely when I start watching something in black and white, I don't know it's in black and white! - I only realise when someone else mentions it, and even then it takes a while for me to see it. It's weird, like my brain is automatically filling in colours that aren't there.

anonymous_fun 7y ago

Math upscale'd it, and my internet connection downscale'd it. I suppose it was math on either end.

ubercow13 7y ago

It is common in video player software these days to use neural network based filters to upscale and downscale, eg see https://artoriuz.github.io/mpv_upscaling.html https://artoriuz.github.io/mpv_upscaling.html

squarefoot 7y ago

I wonder if a similar technique could be used to turn old 4/3 videos to widescreen just by learning the surroundings of a scene through other shots, then mapping them back and forth to build a complete widescreen scene on all frames.

aphextim 7y ago
Off the top of my head a reverse content aware type system maybe could work. https://www.theverge.com/2019/4/3/18293526/adobe-after-effects-content-aware-fill-remove-unwanted-objects-videos https://www.theverge.com/2019/4/3/18293526/adobe-after-effec...
KMnO4 7y ago
I've watched a few movies in what's called ScreenX[0]. It's a cinema experience where they project the movie on the main (front) screen, and project the surroundings on the two side walls. The result is an immersive viewing experience. Obviously the film is not recorded in the ridiculously wide aspect ratio, so usually the side images are computer generated. For landscape imagery, this is fairly seamless. You can't really tell that the images aren't real. However, during a scene with people, the side walls projected static images that drifted around slightly (as if they were cardboard cutouts). I guess the idea is that most things happening in your peripheral vision are lost anyway, so no need to make them detailed. [0]: https://www.cineplex.com/Theatres/ScreenX https://www.cineplex.com/Theatres/ScreenX
dylan604 7y ago
Depending on the specific shot, you can borrow data from frames before/after the current frame rather than just flat out recreating from nothing. For instance, a shot where the camera pans left to right you can borrow frames from later in the shot to fill in the content on the right half of the image. Or a shot that zooms in/out will have actual footage from the scene that can be used as well. This kind of technique is used frequently for other things as well like when removing objects from a scene or even fixing/restoring film with scratches, holes, dust, etc.

monktastic1 7y ago

Anyone know of similar projects for audio? Music within a genre tends to be similar enough that you ought to be able to upsample. I tried doing this once myself but got nowhere.

the_dripper 7y ago

AMAZING!

cal97g 7y ago

Okay, this application of ML is actually cool.

jtbayly 7y ago

Am I the only one that finds the "improved" version much worse? There are digital artifacts all over the place, like the steam at the beginning, the people's faces, any place there is movement. I see that some things don't move as "jerky" but I think the original is far superior to watch. Edited to add: Here's a link to a quick comparison image. These aren't exactly the same frame or size, but close enough to see what I'm talking about. https://imgur.com/a/A3roH54 https://imgur.com/a/A3roH54

godson_drafty 7y ago
I agree, the improved version is worse. At first glance, it looks better than the original, but then numerous artifacts become apparent. For example, check out the little girl's face. It's jumbled in the "improved" version, but very clear in the original. It looks like most of the perceived improvement comes simply from the upgrade to 60fps, which is achieved trivially through interpolation. I wonder if the video would have gotten as much attention had AI not been mentioned.
tachyonbeam 7y ago
Someone posted a colorized[0] version and it had somewhat "jumpy" colors that change over time. I think one problem is you need temporal consistency for this to really work properly. If you pick a color for an object, or you upscale an object a certain way, it has to stay that way over a potentially unlimited number of frames. That's already hard to achieve because neural networks are typically bad at long-term dependency, but it's even more difficult if the object moves, because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with. At the moment, neural networks don't have sophisticated models of the world. [0]: https://www.youtube.com/watch?v=EqbOhqXHL7E https://www.youtube.com/watch?v=EqbOhqXHL7E
- sangnoir 7y ago
  >...because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with. I agree with most of what you said, but disagree with the need to know underlying fundamentals: most 5 year olds can throw and catch a ball without needing to understand calculus or Newton's laws of gravity.

chriscatoya 7y ago

I think it's great we're developing this capability, but I prefer the original-- we're seeing the film as it was originally seen then. The captured scene isn't made better by higher fidelity, it's the opposite for me. As with most advancements in technology and art, we'll see a proliferation of people using the new technique with mixed results at first. It's the shiny new toy. Then over time, people will figure out how to use it and apply it with intent. The upscaling projects we've seen in gaming have generally produced spectacular results, probably because the intent in gaming graphics has always been for the sort of pixel density we're finally able to start achieving. Remember the huge disparity between the box art and the game? lol. Gaming aside, I don't think we've left the toy phase just yet.