13 ms·
Nvidia Canvas
- MileyCyrax 5y agoI wonder how difficult it would be to make something similar that generated 3D models. Most of the examples look like they'd make good video game levels.
- brundolf 5y agoRelated: https://www.dungeonalchemist.com/ https://www.dungeonalchemist.com/
- spoiler 5y agoDungeon Alchemist seems really cool (I'm a backer), but I'm not entirely sure that it is related. DA is basically procedurally generated furnishing (with a few params), but it doesn't create 3D models from what I understand, it "just" shuffles around furniture.
- mm983 5y agoFirst time I've heard of this. Is it for tabletop games like d&d? Or for game development? Looks really cool
- bredren 5y agoI wondered the same. There is some solid competition in this area right now, without AI assisted asset generation. Unreal 5 has a new, free, 3d model library integrated as Quixel Bridge. [1] Kitbash 3D, a company selling modular 3D sets used regularly in Beeple’s 2d provides mid-res, theme-based sets for customized use. Neither take into account the idea of fully featured 3d objects being built from basic primitive using ML. It makes sense that it will go this direction though, because it means designers can get unique 3D assets customized to the size and dimensions with less work. Couple this with Apple’s photogrammetry in iOS 15 it seems original 3D assets available for training data will swell greatly. [1] https://youtu.be/d1ZnM7CH-v4 https://youtu.be/d1ZnM7CH-v4 @ 4:34
- Cthulhu_ 5y agoI think the theory's all there, it just needs reference material on the one hand and the work to be put in on the other. With the new Unreal 5 engine, I think there is a lot of room for technology where an artist sketches out a rock and tools come in to generate the small details - much like there's tools like speedtree and co nowadays to procedurally generate content.
- fsloth 5y agoWell, I think there is enough interesting research to put things in place. Not in single model. But, we have 0. This neural thing, of course, to create landscape-like 2D projections of a plausible scene. 1. Wave-function collapse models that synthesize domain data quite nicely when parametrized with artistic care - this is a "simpler" example of the concept. https://github.com/mxgmn/WaveFunctionCollapse https://github.com/mxgmn/WaveFunctionCollapse 2. Fairly good understanding how to synthesize terrain. Terragen is a good example of this (although not public research, the images drive the point home nicely) https://planetside.co.uk/ https://planetside.co.uk/ So, we could use the source image from this as a 2D projection of an intended landscape as a seed to a wave-function collapse model that would use known terrain parametrization schemes to synthesize something usable (so basically create a Terragen equivalent model). I think that's it plausibly more or less. But it's a "research" level problem still, I think, not something one can cook up by chaining the data flow from a few open source libraries together.
- speps 5y agoHave a good read :) https://github.com/yenchenlin/awesome-NeRF https://github.com/yenchenlin/awesome-NeRF
- blondin 5y agothis looks amazing! i wonder why the beta is windows only though...
- yellowfish 5y agoWhy wouldn't it be?
- coolspot 5y agoBecause most real-world CUDA research happens on Linux with Python and Jupyter?
- alphachloride 5y agoMost end-users are on Windows, however.
- nonameiguess 5y agoThat may not be the case for much longer. The press release on their final financial report from last year: https://nvidianews.nvidia.com/news/nvidia-announces-financial-results-for-fourth-quarter-and-fiscal-2021 https://nvidianews.nvidia.com/news/nvidia-announces-financia... Data center revenue at $6.7 billion. Gaming at $7.7 billion. But data center grew 124%, gaming 41%. If that keeps up, data center passes gaming this year.
- jamra 5y agoI don't think data centers count as end users. End users are human beings that use a system.
- nonameiguess 5y agoThat doesn't make any sense to me. Human ML/AI researchers are also users and NVIDIA clearly intentionally targets them as a market segment. They don't only care about pleasing gamers running Windows.
- noway421 5y agoLooks great!! Looking forward to Mac OS support!
- zeusk 5y agosarcasm? This requires RTX cards and afaik Apple hasn't supported Nvidia hardware since like maxwell?
- asteroidbelt 5y agoI suspect they don't really need GPU to render it. It is usually training what requires a lot of GPU, not evaluation. So the Nvidia requirement is only to sell more cards.
- etaioinshrdlu 5y agoNot true. Big neural nets like these are still dog-slow on CPUs.
- asteroidbelt 5y agoMaybe they are. But I suspect 10 core i9 CPU is not much slower than the oldest Nvidia card they list as the requirement. Don't know much about GPU performance though except random links I have found online which tell that GPU is 3-5 time faster for ML.
- faeyanpiraat 5y agoThey list nvidia RTX as their minimum requirement. i9-7980XE: 1.3 teraflops RTX 2060: 52 teraflops
- deleted 5y ago[deleted]
- asteroidbelt 5y ago
- pizza 5y agoInteresting potential for dream journaling..
- andai 5y agoHeard someone say the other day they use an AI face generator to capture the faces of people they meet in their dreams.
- rainboiboi 5y agoWindows? Can we have linux support please!
- visarga 5y agoThe neural net was probably trained on Linux, they put in extra effort to package it for Windows.
- qayxc 5y agohttps://nvlabs.github.io/SPADE/ https://nvlabs.github.io/SPADE/
- programmarchy 5y agoThis occurred to me, too. Might be able to get it running on Wine or even Proton [1] [1] https://github.com/ValveSoftware/Proton/ https://github.com/ValveSoftware/Proton/
- jskybowen 5y agoCurious if this could be used to generate UI mocks. Pass in a whiteboard sketch and you get the mocks out.
- sp332 5y agohttps://sketch2code.azurewebsites.net/ https://sketch2code.azurewebsites.net/
- esjeon 5y agoThis looks still dang hard for my cursed paws. I’m pretty sure it’s not easy for most people, and it still can’t beat google image search considering the amount of images.
- w-ll 5y agomspaint to bryce 3d
- npunt 5y agoIncredible! They finally built a tool to 'Draw the Owl' [1] [1] https://knowyourmeme.com/memes/how-to-draw-an-owl https://knowyourmeme.com/memes/how-to-draw-an-owl
- visarga 5y agoAs incredible as it looks this feat has been demonstrated since a few years ago.
- tmabraham 5y agoIDK why this is being downvoted, it was indeed published two year ago, they just apparently repackaged this as an easier-to-use tool: https://arxiv.org/abs/1903.07291 https://arxiv.org/abs/1903.07291
- defaultname 5y agoBefore there was a dense scientific paper. Now they've released an incredibly simple tool that lets anyone draw photorealistic images with simple strokes. Saying that is repackaging something into an easier-to-use tool seems like quite a stretch. They didn't put a GUI on curl or something.
- nacs 5y ago> They didn't put a GUI on curl or something I don't think a GUI for curl would be as easy as you imagine. Curl has a lot of power with all the options and protocols it supports.
- bun_at_work 5y agoIsn't this a browser?
- lmohseni 5y agohttps://www.jensroesner.com/wgetgui/wgetgui.png https://www.jensroesner.com/wgetgui/wgetgui.png Here's an image of a wget gui. It's not quite a browser, interesting to look at nonetheless.
- jp0d 5y agoWow. This is amazing! I'm not holding by breath for Mac OS support as Apple isn't very fond of Nvidia. I'm sure there will be clones in future for MacOs/IOS, Linux and Android.
- qayxc 5y agoThe model itself is hardware-agnostic, so there's nothing preventing someone from building a frontend for their platform of choice. Granted, powerful hardware is still required to run inference at acceptable speeds (or at all - I don't know the memory requirements).
- mtobeiyf 5y agoThis reminds me of my "Sketch to Art" project made in 2018: https://github.com/mtobeiyf/sketch-to-art https://github.com/mtobeiyf/sketch-to-art The idea is pretty much the same, except Nvidia is using a more complex model.
- jiofih 5y agoAnd 327 other projects around the same time. The beauty of open research.
- grouphugs 5y agoreally wanted to move to amd, but ahem
- ksec 5y agoI am thinking if this Nvidia Canvas and the Apple Object Capture Capture [1] [2] will make Graphics or 3D Modelling cheaper or taking much less time. Instead of using tools on the computer which was never really good enough for human creation, they now create photos, painting, or models in real world and scan it in a computer for further editing. [1] https://www.youtube.com/watch?v=88rttSh7NcM https://www.youtube.com/watch?v=88rttSh7NcM [2] https://www.youtube.com/watch?v=SuNNyjs9BO8&t=1s https://www.youtube.com/watch?v=SuNNyjs9BO8&t=1s
- ackbar03 5y agoDoes anyone know if the backend code for this is made opensource on github or something? So it can run without windows?
- jachee 5y agoGiven its deep integration with their RTX APIs, I imagine even if the source code were open, the only way to get at the RTX ML-specific stuff is via their Windows driver.
- account_created 5y agoTo all non Windows users http://nvidia-research-mingyuliu.com/gaugan/ http://nvidia-research-mingyuliu.com/gaugan/
- EvanKnowles 5y agoLooks like it has CORS issue.
- littlestymaar 5y agoIt works for me. (FF Linux)
- fab1an 5y agoCompared to canvas, this looks more like a early 2010s style transfer tech demo :) Thanks for the link though!
- littlestymaar 5y agoAh, that's kind of reassuring for canvas, because I was really disappointed when I' tried to play a bit with it. I was like: “meh, how is that even worth a press release?”
- nomel 5y agoAfter painting for 5 minutes: This segmentation map may contain unsupported labels.
- JoblessWonder 5y agoWhoa. Not just all non-Windows users but browser based. That's neat. I was interested in trying it out but didn't want to download the software.
- amelius 5y agoThey implemented Bob Ross.
- eludwig 5y agoIt's more like handing off Bob Ross paintings to an overachieving photorealist who promptly paints over them.
- toomanyducks 5y agosolution looking for a problem? though if anyone does know of problems this solves I'd love to hear about them, this is an incredibly cool solution.
- chrsig 5y agoDoes every program need to be a solution to something? One might say the problem it solves is by satisfying one's desire for novelty. Put another way: It's just really cool, and that can be enough.
- xvector 5y agoI've often desperately wanted to put certain landscapes from my dreams into art, but I suck at drawing. There are some dreams that I remember years later because of how beautiful they were, and how they made me feel. This would be a godsend if it works as well as the demo pictures show.
- cblconfederate 5y agothe problem is demand for stock images. i m not sure the quality here is good enough, but there's no reason why image-generating ANNs won't keep getting better
- thunkshift1 5y agoAs of now maybe.. next version(s ?) will be probably animation to realistic movies.
- toomanyducks 5y agobut will it? my very limited experience with animation has been characterized by control: it's storytelling and creation where the creator is responsible for every fraction of a second. The value of this type of AI is in ceding control to an algorithm and letting it deal with the hard parts. My limited understanding is sort of pointing to a difference in goals of the two projects: one is for control, the other is for ease. And I don't think ease has a very stable place in animation.
- 5y ago
- nl 5y agoThe code behind this is available here: https://github.com/mcheng89/gaugan https://github.com/mcheng89/gaugan https://nvlabs.github.io/SPADE/ https://nvlabs.github.io/SPADE/ https://github.com/NVlabs/SPADE https://github.com/NVlabs/SPADE
- deleted 5y ago[deleted]
- winrid 5y agoI bet you could make some nice assets for games with this. Definitely will find a niche in the indie dev scene.
- aspaviento 5y agoFor the indie dev scene it would be more useful one that generates pixel art which is more common. Indie games rarely have realistic scenarios.
- Hamuko 5y agoJust run it through Photoshop's Mosaic filter.
- dannyw 5y agoalso pixel art is cheaper. realistic scenarios are more expensive. so more value here
- aspaviento 5y agoUsing realistic scenarios forces you to use realistic assets everywhere. If this tool only does backgrounds, it would raise costs for indie developers. Hence my previous comment.
- BiteCode_dev 5y agoNice for a prototype,but given model bias, it will create a creativity bubble like the google search bubble but for visuals.
- w1nk 5y agoWould you mind elaborating a bit more on what you mean? It's very fashionable to be concerned about model bias at the moment, but it's not clear to me what the issue you're describing would be? Something like: trees would end up looking too much like the same tree?
- jalk 5y agoThe “worry” here is that everything produced will look similar and hence will become boring at some point.
- w1nk 5y agoRight, that was the assumption I was alluding to at the end of my comment. That said, it still doesn't fully resolve the question and unfortunately leaves the statement in handwaving territory still. 'boring' isn't really a measurement we can take and discuss super effectively, but we do have actual metrics across visual datasets that span basically all of what you might see as a human. By chance, are you aware of any research on this topic?
- patrick451 5y agoI would say it leaves the statement in hypothesis territory, not hand-waving territory. If we can't discuss or measure "boring" effectively, that is a problem with our instruments and vocabulary, not the assertion. "Boring" is an absolutely crucial thing to be worried about when it comes to anything remotely artistic.
- w1nk 5y agoIt's not a hypothesis though, there's nothing that we can test in it. As presented, it's an assertion that has zero evidence to support it. That's the part I was hoping to get some actual clarification and precision on. When discussing machine learning, bias is a term that has a quantifiable definition in it's different contexts....boring doesn't. It's just FUD to snipe this kind of work with some kind of 'bias' fear without any additional evidence or thought.
- roschdal 5y agoThis is almost as nice as going outside in nature.
- thn-gap 5y agoDo authors own the copyright of their generated images? I can't see any mention in the FAQ.
- pulse7 5y agoRelated: Do authors of a digital camera own the copyright for their (preprocessed) images?
- bruce343434 5y agoWhy wouldn't they? It's just a tool. When I write something by pen, do I get the copyright or does the company that made the pen? Me, obviously.
- XCSme 5y agoBut what if the pen draws by itself? You just say "draw a dog", and it does. I do think you should always be the copyright owner, unless it's clearly stated in their terms that any image created using their tool is owned by nVidia.
- bruce343434 5y agoBut it's still just a tool that functions to your input. An IDE which insert a lot of boilerplate and autocompletions, does it get the copyright to your codebase? Nope.
- bellyfullofbac 5y agoInterestingly, if I employ an artist to produce a work (e.g. software code), usually the employment contract would say the copyright belongs to me and not that artist. "Hey pen, sign this contract."...
- jeffpeterson 5y agoI'd just like to point out that this line of inquiry is not some unanswered philosophical question. All of capitalism is focused on this question of ownership. Who owns the picture? The answer is always whoever the parties involved agreed would own it. Both options can exist and they'll have different prices. This same question often comes up with self-driving cars and "fault", and it seems to regress into the same trap. Ownership of _risk_ is one of the primary concerns of capitalism. The question is not, "who should be at fault?", it is instead "what is the cost of this risk?" and then we buy and sell that risk like everything else (which is also how we determine that cost). If the self-driving advocates are right and self-driving is safer, then the risk will likely cost less than your current insurance. Of course, it's not always clear. If the parties can't agree who owns a thing, they often use some legal mechanism to resolve their dispute.
- shreyshnaccount 5y agoOkay so this is like putting image segmentation data into a GAN and getting the opposite result, right? Or is there something I'm missing?
- riztaak 5y agoAbstract, abstract painting.
- fab1an 5y agoThis is Windows only, probably due to the lack of support for the relevant GFX stuff on Mac? Incidentally, does anyone know of a straightforward and quick Windows-in-the-cloud solution? A bit like GeForceNOW but giving you an entire VM without setup et al?
- anaisbetts 5y agoHere you go, works for AWS, Azure, and GCP - https://github.com/parsec-cloud/Parsec-Cloud-Preparation-Tool https://github.com/parsec-cloud/Parsec-Cloud-Preparation-Too.... Parsec is similar technology to Geforce Now (only better imho)
- fab1an 5y agoThanks!
- kleiba 5y agoI find the demo video terrible, though. Of course, it gets the general idea across but it's too fast-cut to see any details which - for something visual like painting - is kind of the whole point.
- nly 5y agoAnyone else remember when a 1.1GB download was a serious commitment? Now it's a coffee break curiosity.
- marcodave 5y agoI remember when downloading 10MB was a serious commitment shakes fist at clouds
- growt 5y agoWhen I got the first smartphone for my wife (then girlfriend) we set it up when we were in a hotel (without wifi). The 10mb mobile data used for setting up email etc. endet up costing more than the phone :(
- pell 5y agoI remember when downloading 10MB would sometimes take multiple tries. Maybe the internet felt so much more exciting to me back then because it was so much slower.
- myth_drannon 5y agoGetRight download manager to your service.
- bellyfullofbac 5y agoI was trying to remember that name. Tried to look for a video of it on YouTube, there's some pickup artist who calls himself Mr. GetRight instead... But hah, their website is still alive, and they're still selling it for $20: https://www.getright.com/screens.html https://www.getright.com/screens.html
- freedomben 5y agoIndeed. I remember downloading a 3 MB mp3 file and it would take numerous tries and quite some time. Winamp would play the partial though so you could put it on repeat and get a few more seconds of the song on each run. This was back when you could find websites that just offered a collection of mp3s for direct download! The internet was a wild lawless place of free information back then.
- Qiu_Zhanxuan 5y agoof course you need an RTX GPU, how come i'm not allowed to run it on a 1650 Super ? sigh
- deathtrader666 5y agoLooks like it is Windows only. The absence of nVidia GPUs on Macs is really making Macs a weaker dev machine for ML work.
- j4yav 5y agoIt would be really interesting/fun to run this on a frame by frame output of classic 8 bit video games and see what it does. I know it wouldn't make the game look real within it's own concept, but the ordered/familiar inputs to the AI might generate some interesting video outputs.
- toxik 5y agoYou’d need some kind of inter frame consistency also, would probably jiggle quite a lot.
- jffry 5y agoI'm not entirely sure how this would be achiveved. When you draw in the app, you use a brush and have to pick materials from a palette (like "sky" or "ground" or "stone wall" etc). It doesn't seem to have any sort of "import an image" feature because what would that even mean in their model. The approach I would probably consider is to use a modified ROM so that the different sprites in the game are different solid colors. Then I'd write some kind of mouse automation to use those captured images and draw the frame in the app, clicking on the various palette options based on color. The next challenge is that the Canvas app doesn't let you set individual pixels, the smallest brush is ~10px across on its ~550px canvas. Maybe I'd have to settle for picking a Z-ordering and just drawing everything approximately, or maybe you could do some sort of attempt at a path routing algorithm to draw along the edges of the shapes and fill in the centers.
- yeldarb 5y agoAn example is here (using the same NVIDIA GauGAN model that backs this Canvas app): https://twitter.com/jonathanfly/status/1144735290591981568?lang=en https://twitter.com/jonathanfly/status/1144735290591981568?l... Jonathan has played with GauGAN quite a bit (search twitter for "from:@jonathanfly gaugan" to see more).
- jffry 5y agoCool! For anybody looking for Mario specifically, here it is: https://twitter.com/jonathanfly/status/1158846045285232641 https://twitter.com/jonathanfly/status/1158846045285232641 The one with the Pole Position racing game looks pretty cool, with a surprising amount of stability between frames: https://twitter.com/jonathanfly/status/1146569133376573440 https://twitter.com/jonathanfly/status/1146569133376573440
- MrYellowP 5y agoAm I the only one who thinks this development is a bad idea? Just extrapolate the obvious into the future. When everyone can create good art, despite being actually completely unskilled and untalented, then good art ceases to exist. When everyone's an artist no one's an artist. It doesn't matter if we're not there yet, we will get there eventually and at that point it's too late. ... not that it's stoppable anyway.
- imvetri 5y agoThat is very true my friend. I share the same opinion. This AI art seems very menial to us, but not for the fresh minds. This is same, and applies to our generation. When we were given tools to make art, our previous generations would have thought the same.
- csomar 5y agoIt's already happening. All these logos, designs and brochures looks the same now.
- ramblerman 5y agowould you call them good art?
- Aerroon 5y agoWe'll just get different art instead. Compositions etc.
- tnecio 5y agoThe same used to be said about photography. What is most important in true art is not being skilled with a paintbrush or Photoshop but ability to evoke different emotions and thoughts.
- sva_ 5y agoExactly. Art has been past the point, where drawing photorealistic pictures was considered artistic talent, for over a century now.
- Jyaif 5y agoHow did they create the model? Did they take a bunch of reference pictures where they said "this part here is water, this part here is rocks, this part here grass, etc...", and somehow trained a model from that?
- jffry 5y agoFrom their FAQ [1]: Q: How does the AI in Canvas work? NVIDIA Canvas uses a GAN (Generative Adversarial Network) to turn a rough painting of a segmentation map into a realistic landscape image. 5 million photographs of landscapes were used to train the network on an NVIDIA DGX. Q: Is Canvas related to GauGAN? Canvas is built on the same research core that NVIDIA showed in GauGAN. [1] https://nvidia.custhelp.com/app/answers/detail/a_id/5105 https://nvidia.custhelp.com/app/answers/detail/a_id/5105
- 41209 5y agoGood find, I think I'm going to take this out with me to go drawing this weekend and see what I can do.
- ma2rten 5y agoThere is also an online version here: https://www.nvidia.com/en-us/research/ai-playground/ https://www.nvidia.com/en-us/research/ai-playground/
- sabujp 5y agoThis is awesome, now have it automatically build a 3d world with 3d assets
- seansaito 5y agoClicks "Download" on a Mac Me: "Why is it downloading an .exe?"
- qvrjuec 5y agoSomeone used the version of this available online last year to create a 3D video version of the output, super interesting: https://www.youtube.com/watch?v=ZXFmZsv0Ddw https://www.youtube.com/watch?v=ZXFmZsv0Ddw
- dirtyid 5y agoIndistinguishable from magic.
- mjgoeke 5y agoHmm slightly disappointed. I thought I'd make a mashup video game short with generated art. The output resolution is locked at 512x512. The "target style images" seem to be locked to that handful that come with the application. The brush materials don't include anything man-made. Am I doing it wrong?
- gfiorav 5y agoYou need an RTX card to run this, so don't bother with the "Driver update" prompt if you don't have one!