7 ms·

Lumière’s Train in 4K

113 points by ecliptik 7y ago

lokl 7y ago
Related: is there software that, given low-resolution video of people, can produce high-resolution images of their faces by analyzing multiple frames?
- CommieBobDole 7y ago
  I'm not aware of anything that will do this offhand, but I think that's probably the future of upscaling and restoration - software that looks at a video, identifies all of the objects, learns as much as possible about their shapes, textures, lighting and movement by analyzing all the frames available, then makes an 'educated guess' about what it can't figure out and essentially renders a new scene in whatever resolution and framerate is requested. I can only imagine the amount of input data and processing power required to do that, though. Probably a decade or two away.
- acuozzo 7y ago
  What you're looking for is a temporal sharpening algorithm. These exist in crude forms because they're considerably more challenging to program than temporal noise reduction algorithms. Problem #1 is automatic image registration/alignment which is still largely an open problem of great interest to astronomers and those in medical/surgical fields. It's challenging because we're taking 2D images of objects moving in a 3D space. Problem #2 is classification. It's very hard at the moment to suss out which high frequency details are important vs. which are noise.
deleted 7y ago
[deleted]
acqq 7y ago
It seems the "restorations" are all around, e.g. published on Blu Ray in 2015: https://www.amazon.fr/LUMIERE-%C3%89dition-Prestige-blu-ray/dp/B00SVF132A https://www.amazon.fr/LUMIERE-%C3%89dition-Prestige-blu-ray/...
jonplackett 7y ago
Is it just me or does the non-4k digital version look better. The 4K one just looks kinda fake.
- tiborsaas 7y ago
  I'm on the other end, the 4k 60fps makes it look alive and more relatable.
  - josteink 7y ago
    Look at people's faces: They all look the same, because the AI has interpolated them to be the same. Especially the male faces with moustaches. To me, it creates this eerie kind of feeling where everything looks real-ish, but still has that deceptive feeling of actually being fake. Maybe I'm just overly exposed to Hollywood CGI and have a good filter for these things? The original looks low-quality, but real, and so it is clearly better.
    - mytailorisrich 7y ago
      > but still has that deceptive feeling of actually being fake. I think that's key when it comes to AI for these use-cases. It's not restoring by filtering out noise and interpolating. It's really creating something entirely new. Even if they perfect the technology so that the result looks perfectly real, it will still be completely fake.
    - stronglikedan 7y ago
      In the context of the main subject being the train, and not the peoples' faces, the 4k 60fps version makes me feel like I'm standing right next the real train as it pulls into the station. I was taken aback at first, as I was expecting less.
    - ethagnawl 7y ago
      > To me, it creates this eerie kind of feeling where everything looks real-ish, but still has that deceptive feeling of actually being fake. https://en.wikipedia.org/wiki/Uncanny_valley https://en.wikipedia.org/wiki/Uncanny_valley
p1esk 7y ago
Quality improvement from the first video to the second is impressive, but from the second to the third is not.
- jonplackett 7y ago
  Yeah I agree. And I don't mind the grain in the second one. The third one looks unnatural - just looks like the smart sharpen tool in photoshop, rounding off the highlights and flattening texture. Don't get what all the fuss is about.
  - jerf 7y ago
    You brain subconsciously picks up on cues that you don't even necessarily realize. The low frame rate combined with "black and white" screams old timey, and that detaches us from the people in the footage by classifying them as "old timey", which given the ways this footage is used in our culture, ends up classifying them adjacent to "cartoony". Probably courtesy of all the montages of old failing flying machines, which also gives them a "goofy" vibe, and the fact we tend to run the clips about twice as fast as we should, making people jump around and generally look silly rather than human. For me, YMMV, the digitally-restored version still triggers that classification in my head. However, the upscaled one does not. It looks like normal people boarding a normal train, except in black and white. It's really the frame rate that does it for me. This allows a more visceral connection to a time period, and allows us to replace our mental image of "goofy people who are probably just about to drive the train off a cliff or something" with real human beings from a long time ago, doing something mundane and normal. It isn't there yet, but it's a promising start. Add a decent, realistic colorization and we'll really be cooking with fire. YMMV, because you may not have this subconscious associations with oldtimey footage or you may be triggering on cues still present in the new footage and still classifying it mentally as "old timey", so you don't see any change. For instance, there's still a very large cue coming out of "black and white video" that may result in your subconscious still classifying it as "old timey" for you. But for those who do have our subconscious classify it differently, as I do, it's an impressive effect. None of this is meant to be critical of you personally, merely an explanation of why it may be that some people are having a different reaction than you. People are different. I will also criticize myself and say that consciously, I'm well aware that my mental model of "old timey" is completely broken and they were real people back then, just like today, but being consciously aware doesn't affect my feelings anywhere near as much as seeing the upscaled footage does. Humans gonna human. The tech will only get better.
BuildTheRobots 7y ago
Discussion from yesterday: https://news.ycombinator.com/item?id=22242478 https://news.ycombinator.com/item?id=22242478
raphaelj 7y ago
Is it me, or the first and second video are totally different shots ? The train, people and station look totally different.
- oupo 7y ago
  I don't know why he didn't link the right scene but here it is: https://youtu.be/4nj0vEO4Q6s?t=265 https://youtu.be/4nj0vEO4Q6s?t=265
  - jacobush 7y ago
    It looks like it's the crudest form of telecine, a video camera pointed at a projection screen on the wall. Maybe even the "weaving" motion is from a slightly swaying cloth projection screen? I'd be interested to see what the Lumiere train movie would look like if properly laser scanned in 4k with a wetgate.
    - acqq 7y ago
      It's unclear what the correct "starting" versions for what are. Specifically, as I've posted in a separate post, there were already more "digital" "restoration" works, and whatever we see on youtube is most probably already processed by somebody and some computers and algorithms, more than once. And it's most probable that the previous kinds of processing were with bad tools or with some other deficiencies. The proper path would indeed be somebody hi-res scanning frame by frame, all grain and scratches kept, each frame more than once with different lighting etc, and then starting from there and comparing the "restoration" results, always with the original(s). Just as an example, a lot of techniques for removing grain remove the existing information. The different lighting scans allow a lot of advanced algorithms to restore what our eyes could discern from the real film but most of the processes remove etc.
ElijahLynn 7y ago
This development should be on the WORLD news! Wow. Just wow!
CommieBobDole 7y ago
I've been following this, and I have to say this video looks awful. Everything seems to change resolution frame-by-frame, tons of detail appears to be missing (but comes and goes throughout the video), etc. If you posted this video today and said "hey, here's a cool old-timey video I took of the recent 1800's train re-enactment society meeting", people would want to know what kind of '90s phone you used and how many times you dropped it before you took the video. But (and the 'but' is important here) according to this article, this is not a 4k upscale of a high-quality restoration of the original film, it's based off the second Youtube video on the linked page, which is a terrible low-quality scan of a bad print, haphazardly 'restored' to remove most of the scratches and holes, then uploaded to youtube and compressed to the point that almost no detail remains. Given the incredibly poor quality of the source material, this is actually pretty impressive.
erulabs 7y ago
Ack! Over resource limit and invalid SSL :( This is a fantastic blog, and I'd be happy to sponsor/fix its hosting (I run a web-hosting company) - if the writers behind this site read this, feel free to reach out!
ethagnawl 7y ago
Be sure to select 4K from the "settings > quality" menu. When I first watched the video, it was defaulting to 360P.
superpermutat0r 7y ago
Looks a little bit like Jinc upscaling and some heavy denoising. I know it's NNs but still.

113 points by ecliptik 7y ago

lokl 7y ago

Related: is there software that, given low-resolution video of people, can produce high-resolution images of their faces by analyzing multiple frames?

CommieBobDole 7y ago
I'm not aware of anything that will do this offhand, but I think that's probably the future of upscaling and restoration - software that looks at a video, identifies all of the objects, learns as much as possible about their shapes, textures, lighting and movement by analyzing all the frames available, then makes an 'educated guess' about what it can't figure out and essentially renders a new scene in whatever resolution and framerate is requested. I can only imagine the amount of input data and processing power required to do that, though. Probably a decade or two away.
acuozzo 7y ago
What you're looking for is a temporal sharpening algorithm. These exist in crude forms because they're considerably more challenging to program than temporal noise reduction algorithms. Problem #1 is automatic image registration/alignment which is still largely an open problem of great interest to astronomers and those in medical/surgical fields. It's challenging because we're taking 2D images of objects moving in a 3D space. Problem #2 is classification. It's very hard at the moment to suss out which high frequency details are important vs. which are noise.

deleted 7y ago

[deleted]

acqq 7y ago

It seems the "restorations" are all around, e.g. published on Blu Ray in 2015: https://www.amazon.fr/LUMIERE-%C3%89dition-Prestige-blu-ray/dp/B00SVF132A https://www.amazon.fr/LUMIERE-%C3%89dition-Prestige-blu-ray/...

jonplackett 7y ago

Is it just me or does the non-4k digital version look better. The 4K one just looks kinda fake.

tiborsaas 7y ago
I'm on the other end, the 4k 60fps makes it look alive and more relatable.
- josteink 7y ago
  Look at people's faces: They all look the same, because the AI has interpolated them to be the same. Especially the male faces with moustaches. To me, it creates this eerie kind of feeling where everything looks real-ish, but still has that deceptive feeling of actually being fake. Maybe I'm just overly exposed to Hollywood CGI and have a good filter for these things? The original looks low-quality, but real, and so it is clearly better.
  - mytailorisrich 7y ago
    > but still has that deceptive feeling of actually being fake. I think that's key when it comes to AI for these use-cases. It's not restoring by filtering out noise and interpolating. It's really creating something entirely new. Even if they perfect the technology so that the result looks perfectly real, it will still be completely fake.
  - stronglikedan 7y ago
    In the context of the main subject being the train, and not the peoples' faces, the 4k 60fps version makes me feel like I'm standing right next the real train as it pulls into the station. I was taken aback at first, as I was expecting less.
  - ethagnawl 7y ago
    > To me, it creates this eerie kind of feeling where everything looks real-ish, but still has that deceptive feeling of actually being fake. https://en.wikipedia.org/wiki/Uncanny_valley https://en.wikipedia.org/wiki/Uncanny_valley

p1esk 7y ago

Quality improvement from the first video to the second is impressive, but from the second to the third is not.

jonplackett 7y ago
Yeah I agree. And I don't mind the grain in the second one. The third one looks unnatural - just looks like the smart sharpen tool in photoshop, rounding off the highlights and flattening texture. Don't get what all the fuss is about.
- jerf 7y ago
  You brain subconsciously picks up on cues that you don't even necessarily realize. The low frame rate combined with "black and white" screams old timey, and that detaches us from the people in the footage by classifying them as "old timey", which given the ways this footage is used in our culture, ends up classifying them adjacent to "cartoony". Probably courtesy of all the montages of old failing flying machines, which also gives them a "goofy" vibe, and the fact we tend to run the clips about twice as fast as we should, making people jump around and generally look silly rather than human. For me, YMMV, the digitally-restored version still triggers that classification in my head. However, the upscaled one does not. It looks like normal people boarding a normal train, except in black and white. It's really the frame rate that does it for me. This allows a more visceral connection to a time period, and allows us to replace our mental image of "goofy people who are probably just about to drive the train off a cliff or something" with real human beings from a long time ago, doing something mundane and normal. It isn't there yet, but it's a promising start. Add a decent, realistic colorization and we'll really be cooking with fire. YMMV, because you may not have this subconscious associations with oldtimey footage or you may be triggering on cues still present in the new footage and still classifying it mentally as "old timey", so you don't see any change. For instance, there's still a very large cue coming out of "black and white video" that may result in your subconscious still classifying it as "old timey" for you. But for those who do have our subconscious classify it differently, as I do, it's an impressive effect. None of this is meant to be critical of you personally, merely an explanation of why it may be that some people are having a different reaction than you. People are different. I will also criticize myself and say that consciously, I'm well aware that my mental model of "old timey" is completely broken and they were real people back then, just like today, but being consciously aware doesn't affect my feelings anywhere near as much as seeing the upscaled footage does. Humans gonna human. The tech will only get better.

BuildTheRobots 7y ago

Discussion from yesterday: https://news.ycombinator.com/item?id=22242478 https://news.ycombinator.com/item?id=22242478

raphaelj 7y ago

Is it me, or the first and second video are totally different shots ? The train, people and station look totally different.

oupo 7y ago
I don't know why he didn't link the right scene but here it is: https://youtu.be/4nj0vEO4Q6s?t=265 https://youtu.be/4nj0vEO4Q6s?t=265
- jacobush 7y ago
  It looks like it's the crudest form of telecine, a video camera pointed at a projection screen on the wall. Maybe even the "weaving" motion is from a slightly swaying cloth projection screen? I'd be interested to see what the Lumiere train movie would look like if properly laser scanned in 4k with a wetgate.
  - acqq 7y ago
    It's unclear what the correct "starting" versions for what are. Specifically, as I've posted in a separate post, there were already more "digital" "restoration" works, and whatever we see on youtube is most probably already processed by somebody and some computers and algorithms, more than once. And it's most probable that the previous kinds of processing were with bad tools or with some other deficiencies. The proper path would indeed be somebody hi-res scanning frame by frame, all grain and scratches kept, each frame more than once with different lighting etc, and then starting from there and comparing the "restoration" results, always with the original(s). Just as an example, a lot of techniques for removing grain remove the existing information. The different lighting scans allow a lot of advanced algorithms to restore what our eyes could discern from the real film but most of the processes remove etc.

ElijahLynn 7y ago

This development should be on the WORLD news! Wow. Just wow!

CommieBobDole 7y ago

I've been following this, and I have to say this video looks awful. Everything seems to change resolution frame-by-frame, tons of detail appears to be missing (but comes and goes throughout the video), etc. If you posted this video today and said "hey, here's a cool old-timey video I took of the recent 1800's train re-enactment society meeting", people would want to know what kind of '90s phone you used and how many times you dropped it before you took the video. But (and the 'but' is important here) according to this article, this is not a 4k upscale of a high-quality restoration of the original film, it's based off the second Youtube video on the linked page, which is a terrible low-quality scan of a bad print, haphazardly 'restored' to remove most of the scratches and holes, then uploaded to youtube and compressed to the point that almost no detail remains. Given the incredibly poor quality of the source material, this is actually pretty impressive.

erulabs 7y ago

Ack! Over resource limit and invalid SSL :( This is a fantastic blog, and I'd be happy to sponsor/fix its hosting (I run a web-hosting company) - if the writers behind this site read this, feel free to reach out!

ethagnawl 7y ago

Be sure to select 4K from the "settings > quality" menu. When I first watched the video, it was defaulting to 360P.

superpermutat0r 7y ago

Looks a little bit like Jinc upscaling and some heavy denoising. I know it's NNs but still.