5 ms·

This is interesting, but it's still not the same as a proper remaster. You can't get information out of SD video that wasn't there before with upscaling, so th

by magduf 8y ago

This is interesting, but it's still not the same as a proper remaster. You can't get information out of SD video that wasn't there before with upscaling, so the algorithm is just interpolating. It's too bad that Paramount is so fan-hostile; they could just get the original film footage (assuming it was shot that way; ST:TOS and ST:TNG were, which is why we have excellent remastered versions today), scan it in as raw video, let the fans have at it, and they'd make the high-def FX for free.

skocznymroczny 8y ago
In the blog post, it says: "While the popular Original Series and The Next Generation were mostly shot on film, the mid 90s DS9 had its visual effects shots (space battles and such) shot on video. While you can rescan analog film at a higher resolution, video is digital and can't be rescanned."
- amaranth 8y ago
  The Next Generation also had its visual effects shot on video, this was a major hurdle for their remastering efforts. They had to recreate all of the effects so it was a slow an expensive process. On the positive side, the effects end up looking much better since this isn't the 80s and we can throw a lot of CPU power at creating them.
  - TomMarius 8y ago
    Could you please expand on why CPU and not GPU? Edit: why am I getting downvoted for asking a question? Sorry :-(
    - tracker1 8y ago
      For what it's worth, the implications are about the same... though in practice, it would likely be ray-tracing frame by frame, then overlays... though the process in the article could be trained with very high detail models and fill in the blanks from that.
    - kelnos 8y ago
      The parent was likely using "CPU" to just mean "processing power", not being prescriptive toward whether that processing is done on a CPU or a GPU. I assume your downvoting is because people (uncharitably) believed you were being an annoying pedant instead of asking a genuine question.
neuralk 8y ago
No, it is not just interpolating. The underlying algorithm uses machine learning by applying a trained deep neural network. So there is value added besides a mere upscale. You're ultimately right, though, and that a true HD is only going to come from the raw film content. What the neural network gives us are essentially plausible higher-res hallucinations. Edit: as per the other comment, if the original exists only on video and not film, perhaps this is the best we're going to get.
- cwyers 8y ago
  Is there any evidence for this? Showing bad 480p DVD rips alongside 1080p upresed video isn't really a fair comparison -- comparing it to a real TV upscaler's output would be fairer. And honestly, even the unfair comparsion doesn't show a whole lot of benefits to me.
  - basch 8y ago
    Its not upscaling. Its taking what it knows about other non ST pictures and creating new texture and information.
    - cwyers 8y ago
      Right, but how much does it actually add over a decent upscaler?
      - basch 8y ago
        are you saying this specific program, or the idea in theory? http://screenshotcomparison.com/comparison/132311 http://screenshotcomparison.com/comparison/132311
        Matticus_Rex 8y ago
        Wow! That's a bigger difference than I expected.
        8y ago
rictic 8y ago
> You can't get information out of SD video that wasn't there before with upscaling, so the algorithm is just interpolating. In the fully general case of arbitrary video this is true, but in practice it isn't. You can gather information over time to do superresolution, and if you want to get super fancy you can build a world model (e.g. get more information about what an actor's face looks like from a close up shot, and apply that knowledge to less detailed shots). I expect ML based upscaling to eventually produce some truly stellar results.
- basch 8y ago
  a combination of a world model, along with non-source enhancements. i imagine upscaling the Phantom Menace podrace to 4k, and giving the model a bunch of NEW rock texture info to use to create new detail. Kinda of an automated way to combine these two ideas. http://www.framecompare.com/image-compare/screenshotcomparison/JCNNN8TU http://www.framecompare.com/image-compare/screenshotcomparis... https://techcrunch.com/2019/03/18/nvidia-ai-turns-sketches-into-photorealistic-landscapes-in-seconds/ https://techcrunch.com/2019/03/18/nvidia-ai-turns-sketches-i... What you dont want is every upscale to start looking homogeneous, so it would be best for a design team to specifically map old to new texture sets, giving each upscale a unique look.
- ocdtrekkie 8y ago
  I was thinking particularly about the space battle shots, where you could provide 3D models of each of the ships, so that the software had the higher detail information, and would just need to match position, movement, and lighting.
- tracker1 8y ago
  I would expect multi-view rendered models of ships/stations etc to also help significantly as well... vs. re-rendering sfx, it would be upscaling via AI with knowledge of detailed views/models for the actors... training stills could go a significant way.
jahewson 8y ago
> You can't get information out of SD video that wasn't there before with upscaling But you can infer visual detail from the information that is already there. Especially because ML uses information from the training set to help make sense of the information that is already there. e.g. if I show you a picture of a key then you can figure out what the lock looks like because the key contains that information, even if it’s not visible.
kriro 8y ago
AFAIK, STTNG has one other advantage. They used real models and DS9 mostly switched to CGI. Old CGI is harder to beautify than models.
- tracker1 8y ago
  They could generate higher detail model compositions to train the AI.
- mortenjorck 8y ago
  In theory at least, a neural network with sufficient training in the right domain should be able to add the slight imperfections and subtle light/texture cues our eyes expect to see, even to too-perfect 1990s CGI scenes. This would, however, require a network that can maintain continuity between frames so these details don’t appear to jump around.
- magduf 8y ago
  Um, that's not what I read at all. What I read was that TNG's FX was all rendered with computers at the time, at NTSC resolution, so it was all useless when they did the remastering, and they had to make all-new FX shots. The place where the film was invaluable was for all the live-action stuff (i.e., anything with actors and sets, or on-location). It's much like what they did with ST:TOS, except TOS actually did film real models, but they were so awful by today's standards they redid them with modern CGI.
  - cfmcdonald 8y ago
    That's true of some effects, but in TNG the core spaceship / space station models were shot on film, and then composited with other effects like phaser shots. In DS9 / Voyager as I understand it, they transitioned from using models to directly generating the whole shot with CGI at NTSC resolution. See https://memory-alpha.fandom.com/wiki/CGI#Acceptance https://memory-alpha.fandom.com/wiki/CGI#Acceptance This means that in TNG they just had to recomposite the film with a newly created high-res phaser shot; whereas for DS9/Voyager they would have to recreate the whole shot in high-res CGI.
arendtio 8y ago
> You can't get information out of SD video that wasn't there before... Well, that is the difference between regular upscaling and upscaling with neural networks. With a neural network, the additional information is being stored within the network during training and added to the video during the upscaling process. Ultimately, you could argue, that this is just interpolating too, but the quality of the interpolation depends on the training material. If you would train it on an original and use such a neural network to upscale a lower resolution version you could end up with the original (a perfect interpolation). So it all comes down to the quality of the training material and while AI Gigapixel seems to have quite good material, I wonder if the result could be improved by transforming the video as a whole and not just frame by frame, as that would give the NN even more information to interpolate on.
- avisser 8y ago
  > With a neural network, the additional information is being stored within the network I've seen a few people say this in the thread. It doesn't seem accurate to me. Information is being created/hallucinated/interpolated. Re-scanning filmstock gets new information. If not, it doesn't matter how sophisticated your algorithm is (naive upscaling or deep learning), you're still interpolating values.

by magduf 8y ago

skocznymroczny 8y ago

In the blog post, it says: "While the popular Original Series and The Next Generation were mostly shot on film, the mid 90s DS9 had its visual effects shots (space battles and such) shot on video. While you can rescan analog film at a higher resolution, video is digital and can't be rescanned."

amaranth 8y ago
The Next Generation also had its visual effects shot on video, this was a major hurdle for their remastering efforts. They had to recreate all of the effects so it was a slow an expensive process. On the positive side, the effects end up looking much better since this isn't the 80s and we can throw a lot of CPU power at creating them.
- TomMarius 8y ago
  Could you please expand on why CPU and not GPU? Edit: why am I getting downvoted for asking a question? Sorry :-(
  - tracker1 8y ago
    For what it's worth, the implications are about the same... though in practice, it would likely be ray-tracing frame by frame, then overlays... though the process in the article could be trained with very high detail models and fill in the blanks from that.
  - kelnos 8y ago
    The parent was likely using "CPU" to just mean "processing power", not being prescriptive toward whether that processing is done on a CPU or a GPU. I assume your downvoting is because people (uncharitably) believed you were being an annoying pedant instead of asking a genuine question.

neuralk 8y ago

No, it is not just interpolating. The underlying algorithm uses machine learning by applying a trained deep neural network. So there is value added besides a mere upscale. You're ultimately right, though, and that a true HD is only going to come from the raw film content. What the neural network gives us are essentially plausible higher-res hallucinations. Edit: as per the other comment, if the original exists only on video and not film, perhaps this is the best we're going to get.

cwyers 8y ago
Is there any evidence for this? Showing bad 480p DVD rips alongside 1080p upresed video isn't really a fair comparison -- comparing it to a real TV upscaler's output would be fairer. And honestly, even the unfair comparsion doesn't show a whole lot of benefits to me.
- basch 8y ago
  Its not upscaling. Its taking what it knows about other non ST pictures and creating new texture and information.
  - cwyers 8y ago
    Right, but how much does it actually add over a decent upscaler?
    - basch 8y ago
      are you saying this specific program, or the idea in theory? http://screenshotcomparison.com/comparison/132311 http://screenshotcomparison.com/comparison/132311
      - Matticus_Rex 8y ago
        Wow! That's a bigger difference than I expected.
      - 8y ago

rictic 8y ago

> You can't get information out of SD video that wasn't there before with upscaling, so the algorithm is just interpolating. In the fully general case of arbitrary video this is true, but in practice it isn't. You can gather information over time to do superresolution, and if you want to get super fancy you can build a world model (e.g. get more information about what an actor's face looks like from a close up shot, and apply that knowledge to less detailed shots). I expect ML based upscaling to eventually produce some truly stellar results.

basch 8y ago
a combination of a world model, along with non-source enhancements. i imagine upscaling the Phantom Menace podrace to 4k, and giving the model a bunch of NEW rock texture info to use to create new detail. Kinda of an automated way to combine these two ideas. http://www.framecompare.com/image-compare/screenshotcomparison/JCNNN8TU http://www.framecompare.com/image-compare/screenshotcomparis... https://techcrunch.com/2019/03/18/nvidia-ai-turns-sketches-into-photorealistic-landscapes-in-seconds/ https://techcrunch.com/2019/03/18/nvidia-ai-turns-sketches-i... What you dont want is every upscale to start looking homogeneous, so it would be best for a design team to specifically map old to new texture sets, giving each upscale a unique look.
ocdtrekkie 8y ago
I was thinking particularly about the space battle shots, where you could provide 3D models of each of the ships, so that the software had the higher detail information, and would just need to match position, movement, and lighting.
tracker1 8y ago
I would expect multi-view rendered models of ships/stations etc to also help significantly as well... vs. re-rendering sfx, it would be upscaling via AI with knowledge of detailed views/models for the actors... training stills could go a significant way.

jahewson 8y ago

> You can't get information out of SD video that wasn't there before with upscaling But you can infer visual detail from the information that is already there. Especially because ML uses information from the training set to help make sense of the information that is already there. e.g. if I show you a picture of a key then you can figure out what the lock looks like because the key contains that information, even if it’s not visible.

kriro 8y ago

AFAIK, STTNG has one other advantage. They used real models and DS9 mostly switched to CGI. Old CGI is harder to beautify than models.

tracker1 8y ago
They could generate higher detail model compositions to train the AI.
mortenjorck 8y ago
In theory at least, a neural network with sufficient training in the right domain should be able to add the slight imperfections and subtle light/texture cues our eyes expect to see, even to too-perfect 1990s CGI scenes. This would, however, require a network that can maintain continuity between frames so these details don’t appear to jump around.
magduf 8y ago
Um, that's not what I read at all. What I read was that TNG's FX was all rendered with computers at the time, at NTSC resolution, so it was all useless when they did the remastering, and they had to make all-new FX shots. The place where the film was invaluable was for all the live-action stuff (i.e., anything with actors and sets, or on-location). It's much like what they did with ST:TOS, except TOS actually did film real models, but they were so awful by today's standards they redid them with modern CGI.
- cfmcdonald 8y ago
  That's true of some effects, but in TNG the core spaceship / space station models were shot on film, and then composited with other effects like phaser shots. In DS9 / Voyager as I understand it, they transitioned from using models to directly generating the whole shot with CGI at NTSC resolution. See https://memory-alpha.fandom.com/wiki/CGI#Acceptance https://memory-alpha.fandom.com/wiki/CGI#Acceptance This means that in TNG they just had to recomposite the film with a newly created high-res phaser shot; whereas for DS9/Voyager they would have to recreate the whole shot in high-res CGI.

arendtio 8y ago

> You can't get information out of SD video that wasn't there before... Well, that is the difference between regular upscaling and upscaling with neural networks. With a neural network, the additional information is being stored within the network during training and added to the video during the upscaling process. Ultimately, you could argue, that this is just interpolating too, but the quality of the interpolation depends on the training material. If you would train it on an original and use such a neural network to upscale a lower resolution version you could end up with the original (a perfect interpolation). So it all comes down to the quality of the training material and while AI Gigapixel seems to have quite good material, I wonder if the result could be improved by transforming the video as a whole and not just frame by frame, as that would give the NN even more information to interpolate on.

avisser 8y ago
> With a neural network, the additional information is being stored within the network I've seen a few people say this in the thread. It doesn't seem accurate to me. Information is being created/hallucinated/interpolated. Re-scanning filmstock gets new information. If not, it doesn't matter how sophisticated your algorithm is (naive upscaling or deep learning), you're still interpolating values.