13 ms·
Remastering Star Trek: Deep Space Nine with Machine Learning
- zackkrida 8y agoYou may find the original blog post from the creator here: https://captrobau.blogspot.com/2019/03/remastering-star-trek-deep-space-nine.html https://captrobau.blogspot.com/2019/03/remastering-star-trek...
- WrtCdEvrydy 8y agoInteresting, I wonder if we could create a web-based system where people could upload videos and get 4k footage back... Edit: Okay, new plan, anyone wanna make a cli for AI Gigapixel?
- jasonvorhe 8y agoSounds intriguing. What's your experience with AI Gigapixel so far?
- basch 8y agoDownload the trial and upscale a photo. Its pretty cool, it seems a little better than old fractile upscale/sharpening techniques. I changed the exposure too, but this is what i was able to achieve using both their JPEG to RAW and their Gigapixel, along with some color correction. (theres still a tiny bit of blocking id like to clean up, but its not bad for a test run.) http://screenshotcomparison.com/comparison/132311 http://screenshotcomparison.com/comparison/132311 If anyone has used Photoshops Match Color feature before, I do expect a day where I can open some textures and apply them to similar low rez textures in a photo, and use the samples to repaint the picture. Similar in end result to how texture packs work for games. I also think by taking HDR footage and compressing it to 6-8bit color space, and training a model to reproduce the original, that AI will be pretty good at upconverting older footage into high color depths.
- ukyrgf 8y agoThe URL for this post should really be changed to that. It's nothing more than a two sentence blogspam submission with a link to the original creator.
- sctb 8y agoDone! Updated from https://www.turingtribe.com/story/star-trek-fan-remasters-star-trek-deep-space-nine-in-hd-using-machine-learning-yEHAXy7S7PxGZEQE9 https://www.turingtribe.com/story/star-trek-fan-remasters-st....
- ericsoderstrom 8y agoI don't see a huge difference between the two. Mostly the Ferengi forehead creases are sharper I guess, but unclear that the overall effect is much of an improvement. Cool proof of concept though!
- izzydata 8y agoYou need to have both full screened on a 1080p screen to actually compare the two. Also video quality is difficult to determine from still frames. Video compression is more about how it moves from 1 frame to another. You would want to watch an actual clip of both to see the difference better.
- ericsoderstrom 8y agoYeah that's probably it. Was wondering whether half a screen of 1080p would be enough to properly show the difference
- onli 8y agoWhat? Everything is sharper in the 1080p version, the effect is far from subtle. Is it possible you did not load the 1080p version of the Youtube video?
- NedIsakoff 8y agoI honestly really don't see the difference beyond that.
- eropple 8y agoA lot of the CG (ships, etc.) looks sharper to my eye. I didn't notice at first, though, because YouTube defaulted to 720p (and since it's a 1080p render downscaled to be side-by-side, you do need those pixels). The creator says that 4K-vs-480p renders (that is, rendering the YouTube video at 4K) are on the way.
- QuackingJimbo 8y agoSame. It's certainly not objectively "amazing" (to use the author's own verbiage)
- magduf 8y agoThis is interesting, but it's still not the same as a proper remaster. You can't get information out of SD video that wasn't there before with upscaling, so the algorithm is just interpolating. It's too bad that Paramount is so fan-hostile; they could just get the original film footage (assuming it was shot that way; ST:TOS and ST:TNG were, which is why we have excellent remastered versions today), scan it in as raw video, let the fans have at it, and they'd make the high-def FX for free.
- skocznymroczny 8y agoIn the blog post, it says: "While the popular Original Series and The Next Generation were mostly shot on film, the mid 90s DS9 had its visual effects shots (space battles and such) shot on video. While you can rescan analog film at a higher resolution, video is digital and can't be rescanned."
- amaranth 8y agoThe Next Generation also had its visual effects shot on video, this was a major hurdle for their remastering efforts. They had to recreate all of the effects so it was a slow an expensive process. On the positive side, the effects end up looking much better since this isn't the 80s and we can throw a lot of CPU power at creating them.
- TomMarius 8y agoCould you please expand on why CPU and not GPU? Edit: why am I getting downvoted for asking a question? Sorry :-(
- tracker1 8y agoFor what it's worth, the implications are about the same... though in practice, it would likely be ray-tracing frame by frame, then overlays... though the process in the article could be trained with very high detail models and fill in the blanks from that.
- 8y ago
- ocdtrekkie 8y agoI kinda hope that perhaps, if CBS doesn't find value in doing a TNG-level remaster of DS9 and Voyager, that something like this might give them a "good enough" version for a Blu-ray release. At the very least, one could imagine using this technique where applicable to at least, massively cut down the conversion cost in scenarios where re-cutting the scene would be particularly cost prohibitive.
- makerofspoons 8y agoI do wonder if having DS9 and Voyager available on so many streaming platforms right now cuts down on the incentive to remaster them. I'm not sure if a remaster would be enough of a draw to CBS All Access if you can already watch all the episodes on the other major streaming services, but perhaps Star Trek Discovery is successful enough they might consider it?
- ocdtrekkie 8y agoOn the contrary, I'd think that a remaster would give them a reason to charge all those streaming services more for the shows. No reason to hold exclusivity over a twenty year old show.
- rbanffy 8y agoI think that changing the price is sensitive, but having better quality content would entice people to watch it (reducing pressure on producing new more costly content)
- aetherspawn 8y agoI don’t think it’s fair to call ST: Discovery a part of the franchise at this point. It’s this bizarre smodge of minority group agendas, dark horror-like scenes and modern political ideas disguised as the family friendly Star Trek that you could watch with your kids that we all actually loved. And are the plot lines intelligent? Do they make you think about interesting (non overdone ie non-left) philosophical issues? Not really, and they’re strung along in an awkward and long story arc that’s pretty dull and predictable and takes away from the per-episode storytelling :/
- conception 8y agoPeople are using this same technique for old games - https://venturebeat.com/2019/01/18/machine-learning-is-rescuing-old-game-textures-in-zelda-and-final-fantasy/ https://venturebeat.com/2019/01/18/machine-learning-is-rescu...
- CharlesW 8y agoST:DS9 was originally shot on film like ST:TNG, but the market performance of the TNG remaster makes it very unlikely that DS9 will receive the same treatment. "Essentially, for the HD release of Star Trek, all people had to do was scan each episode. For The Next Generation, they would have to scan all those original pieces of film and then edit together each episode again, themselves. It’s more difficult, more expensive, and much more time-consuming. "Unfortunately, it wasn’t actually worth it. Sales of the extravagant TNG remaster—original retail price $118 for just season one—failed to reach CBS and Paramount’s expectations. A similar process would have to be done for both DS9 and Voyager—and would actually be even harder." https://io9.gizmodo.com/the-detailed-depressing-reason-deep-space-nine-and-voy-1791962332 https://io9.gizmodo.com/the-detailed-depressing-reason-deep-...
- nerpderp82 8y agoBut you could take the raw footage and the edited video and recreate the edit lists (which they should still have btw). Or just scan each film frame, then match the video to film frame by frame and re-assemble the list of frames. All software done by one person. No creativity or judgement needed.
- jjeaff 8y agoSend out a call for volunteers. I guarantee you, you'd have a group of fans with the know how to do it. And they would do it for free.
- rbanffy 8y agoAnd if the technology doesn't exist, they'll probably invent it too. This is Star Trek we are talking about. Their fans are dead serious.
- jjeaff 8y agoYa, I can't think of any show with such a long lasting, rabid fan base. Just look at that recent "fan film" that CBS tried to sue over. Production quality was off the charts for a fan film.
- dbcooper 8y agoAnyone compared this with MadVR?
- cpeterso 8y ago"it took me about two days to get everything extracted, upscaled and put it back together in a way that was pleasing. This resulted only in the first five minutes of the episode being done." Maybe someone at Google can use spare cloud time to remaster the whole series as their 20% time project? This remastering process would be easily parallelizable.
- cpeterso 8y agoI forgot to add a :) because remastering Star Trek is clearly not Google's business, but it could be someone's business: video remastering/upscaling services for Hollywood. There's a lot of popular old film and TV content just waiting to be sold again to fans on Blu-ray or steaming.
- AcerbicZero 8y agoOn the one hand I'd love to see them remastered and touched up, but on the other hand, I originally watched them on a ~27 inch, old mid 90's TV....so the visuals aren't really the part which hits my nostalgia nerve.
- jaimebuelta 8y agoInterestingly, there's a project for a documentary about DS9 that's close to completion. https://www.indiegogo.com/projects/what-we-left-behind-star-trek-deep-space-nine-doc https://www.indiegogo.com/projects/what-we-left-behind-star-... It has been delayed due to master the old footage (it's one of the last updates) I'll leave this work it as a comment on the project.
- themagician 8y agoI had no idea about this! How did they manage to get the entire cast, EXCEPT Avery?
- sdrothrock 8y agoAvery Brooks has distanced himself from DS9 intentionally; he considers that part of his career done.
- modzu 8y agoim not discounting ML being able to do this better. kudos to the op. but for fun i tried it using conventional methods (NR+sharpen); similar results but an order of magnitude faster. you be the judge: https://pasteboard.co/I6tvYmP.png https://pasteboard.co/I6tvYmP.png
- ChrisArchitect 8y agohttps://news.ycombinator.com/item?id=19406060 https://news.ycombinator.com/item?id=19406060
- Jerry2 8y agoI recently tried to rewatch Babylon 5 but the resolution was terrible and it looked bad on my TV so I gave up pretty quickly. I hope someone does the same treatment to Babylon 5 episodes!
- rosser 8y agoUnfortunately, that's probably the "last best chance" for a not-awful watching experience. At some point after the show wrapped, they lost all of their 3d models, so they will never be able to re-render a single ship. EDIT: I didn't mention fan models as a possible approach because this is WB/PETN we're talking about here. They aren't Paramount-dumb, but they seemed take that as a challenge, at least vis à vis B5.
- coryrc 8y agoThere weren't enough polygons on them to look good at higher resolutions; they'd really have to be redone anyway.
- Zardoz84 8y agoyou are pretty wrong . Watch the original meshes at HD resolution : https://youtu.be/uHAuK_lDkk0 https://youtu.be/uHAuK_lDkk0
- coryrc 8y agoDid we watch the same video? It looks like a 2002-era video game when rendered at 1080p. It does not realistic at all; it is nothing like improving a film video of a model (film is basically ~5k equivalent and a model is.. infinite?).
- boomlinde 8y agoI agree that it doesn't look particularly good. The question is whether it's due to a lack of polygons.
- 8y ago
- kriro 8y agoWhat is the licensing situation for remastering like? It would be a pretty interesting competitive edge if a company like Netflix could improve their entire library with some inhouse remastering magic. Or even a bargaining tool for buying new licenses...sure, we'll add your stuff to our collection and remaster it "for free".
- tracker1 8y agoWould definitely get me more interested in handing over my cash... Though, in this case, I think it would more likely be Amazon Video.
- entropicdrifter 8y agoAlright, now I want to do this with Babylon 5 since it seems like the HD remaster from WB/Amazon petition has flopped. Heck, even just upscaling the SFX shots from 360p to full DVD quality would be a worthwhile project.
- skybrian 8y agoI see a lot of comments from people getting excited about superficial quality improvements, and I'm mystified. It's nice I guess, but the story and acting are unchanged by this sort of thing, and isn't that what matters to Star Trek fans?
- luckydata 8y agothe low visual quality is becoming more and more visible as TV screens get bigger and better. It would be great to have those old shows in a quality that fits our new screens, that's all.
- pbhjpbhj 8y agoAre films/programmes now preparing for future re-rendering in any way?
- moultano 8y agoThis seems like it would really benefit from a model trained specifically to upscale video. There's a lot of information you can get from the structure between frames that you can't exploit with a still image.
- sigi45 8y agoHoly shit, you did exactly what i wanted to do after i read about the final fantasy results!11 :D
- MayeulC 8y agoAs I see more of these machine learning applications, where a neural network is trained on a dataset, then applied to a similar dataset, I wonder how well it would help if the network was trained on part of the new dataset as well, as some kind of "specialization". One could take a generic "AI Gigapixel" net, and retrain it on some of the newer Star Trek content, for which an upscaled/remastered "ground truth" exists. My guess is that it would help a lot with features that are specific to the Star Trek universe. Taking this further, one could take the resulting samples, and "rate" them, before feeding it back in the training engine. This would make some kind of "adversarial" feedback loop, but instead of the GAN, humans are involved in the loop (which could be used to train an adversarial network as well, that said). My hope is that it would converge to much better results in a shorter time, and with less input data. My apologies if those are common concepts in ML. If so, I'd love to look at some references that could further my understanding on these topics, and get me to use the right terminology.
- nominated1 8y agoI'm curious as to why the author chose "AI Gigapixel". What made it a better choice than say NNEDI3 or SuperRes or any other NN upscaler (there’s a bunch of them out there)? Were others even considered?
- Balgair 8y agoPro-tip: You have to change the YouTube setting so that you are watching in 1080 HD, not the 360p that it can default you too. You know, I think there is some real promise here! With some HD remasters, you can start to really see the makeup, the little pores in everyone's skin, the smudges and uncleanliness of real film-making. Film-makers and directory choose the lighting and the focus with the end-product in mind. They know that the screen won't capture certain things and so they know where they can skimp and save [0]. When you re-master it, you're going against the 'vision' of the directors. Not in a big way at all, it's very subtle. But it's still there. With ML techniques, you get the 'idea' that the director was going for, without seeing all the screw-ups that they knew they could get away with. It's crisper but the idea and vision are the same [0]. Peter Jackson's recent 'They Shall No Grow Old' is another great example of using ML too. In that case it was to preserve the old WW1 footage, bring it's frame-rates up to modern 24 fps, 3-D it, and colorize it. The results are literally breath-taking. Personally, I gasped when it finally hits; it's that good. Not to geek out too much here, but Jackson is literally changing history with that film. He changed the way we all view old footage, as something all herky-jerky and grainy, to something that is modern and real. Those 16 year old child-soldiers become real people again. Though Jackson's work is a lot different than this effort, I think we all know that ML and the movies are here to stay. It's relatively cheap to update, takes little time, and be profitable (Remember the Disney Vault gimmick?). How long will it be before Chaplin's 'City Lights' is ML'd and remastered into 24 fps, 3-D, color, and with sound? Maybe 5 years? Hell, I'd pay to see the best of old cinema brought back to modern standards like that. [0] I know no film-makers, this is supposition.
- dractori 8y agoThere's a Criterion Collection blu ray of City Lights that I would imagine was done very well given their reputation and the reviews https://www.criterion.com/films/27558-city-lights https://www.criterion.com/films/27558-city-lights
- crazygringo 8y agoWow, so this is fascinating. Some observations: - Many of the frames look like someone just applied a "sharpen" filter -- there's (as expected) no real new information, it's just sharper... so it doesn't seem like a big deal, like I could do it in Photoshop trivially - But then there are a few spots where new details are truly seamlessly added... the fireball in the spaceship explosion, and forehead wrinkles. Stunning, they're absolutely seamless and believable, with detail that is simply not there in the original... that's literally magic - But at the same time, characters in the background that are slightly out-of-focus get oversharpened when they're supposed to be blurry, like it can't tell when moderate blurriness is due to resolution or focus Overall, I'm pretty shocked that the effect is so seamless across frames -- I totally would have expected this to produce weird discontinuities in time, but I didn't see any at all. I mean, this actually seems like it's already production-ready to throw into TV's or VLC. Which is crazy. I wonder how much of this is "general-purpose", or to what extent this works on this episode because it's trained on this episode? E.g. the neural network is learning from a close-up of the Ferengi face or spaceship, to apply specifically to a later smaller version. Or to what extent this will work well across TV shows, across actors, across genres, without prior training, or with sparse training?
- ch_sm 8y agoHe’s using an off-the-shelf commercial app called "AI Gigapixel", which in turn uses a process called image super-resolution. It is general purpose (works for all kinds of images) and fairly thoroughly explored by now. hope this helps
- bitL 8y agoAnyone knows how AI Gigapixel differs from DLSS? (outside Topaz Labs ofc). Thanks!
- microDude 8y agoAny Friends of DeSoto here?
- craigmcnamara 8y agoStar Trek: Deep Space Nine aka, Cheers in Space!