8 ms·
Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing
- RivieraKid 4y agoNot photorealistic at all.
- swframe2 4y agoCheck out https://zivadynamics.com/ https://zivadynamics.com/. It has the realism but the models need to be manually posed. Using the low-poly from efforts like this one, Unity will make it easier to pose the high res Ziva model. There are several papers on how to capture/simulate the clothing which should result in high res outputs. Also see DeepMotion.
- ncmncm 4y agoFunny how it is not good at heels. Who knew those would be hard?
- nighthawk454 4y agothat kind of high-frequency detail is easy to lose in a model and/or from smoothing techniques
- bj-rn 4y agoDoes this also apply to hands? They look mangled in these examples.
- doodlebugging 4y agoWe're probably lucky that they chose to model fully clothed subjects. Based on the smearing we see in some translated details like the fingers and examples of holes in skirts we could have some comically bad output.
- Fiahil 4y agoSame as many other instances of specialized ML models, their results look good generally, but completely fall apart under scrutiny. Remember GPT-3 blogpost writing? Although, they are getting better years after years, so maybe we'll be out of the uncanny valley at some point ?
- pingeroo 4y agoWhat was the issue with GPT-3 & blog posts?
- skybrian 4y agoThe generated text often seems okay if you're skimming, but if you think about what they're saying, it's rarely meaningful.
- plafl 4y agoI have been recently looking at several "3d scanning" from monocular images/videos solutions and although they make for impressive demos I'm afraid they are a few years away from robust results. On one hand you have traditional photogrammetry based on classical image descriptors (from the beginning of the 2000s). A nice open source solution is Meshroom [1]. I have very little experience as I have just tinkered a little but I would say they work OK with geometry of medium complexity and detailed textures. They fail horribly with untextured objects, like for example a candle. They are semi-automatic: you upload some photos and run the pipepiline but for best results it's easy to tweak as there are a lot of knobs to try. On the other hand you have these deep learning research papers. A lot of them have Colab notebooks you can try yourself (which is awesome, thank you). They can give you these very nice demos for some kind of objects (the ones we have a lot of training data, like human bodies), but they are not truly general and sometimes will generate artifacts. There is some opportunity here if they are integrated in a semi-supervised workflow, like this [2]. Anyway, just curious and not an expert. [1] https://alicevision.org/ https://alicevision.org/ [2] https://keentools.io/products/facebuilder-for-blender https://keentools.io/products/facebuilder-for-blender
- deleted 4y ago[deleted]
- aresant 4y agoWhat is the timeline between these papers and an app that can take me from a photo to doing an animated dance flip to impress my kids as in their output example?
- xtagon 4y agoI wonder how this compares with https://shunsukesaito.github.io/PIFuHD/ https://shunsukesaito.github.io/PIFuHD/ (which has published code)?
- recuter 4y agoThat is mentioned in the paper.
- throwaway16374 4y agoYou know, this is just one step away from someone making x-ray goggles
- bcraven 4y agoI don't understand how? Surely estimating someone's bone structure does not need a 3D model. A image-to-skeleton pipeline might even be useful for 2D sprite rigging.
- robbrown451 4y agoPretty sure that's not what is intended. They used to sell supposed x-ray googles to gullible people, the ads implying you'd be able to see people naked.
- kevin_thibedeau 4y agoThe killer app that will take AR mainstream.
- thaumasiotes 4y ago> They used to sell supposed x-ray googles to gullible people, the ads implying you'd be able to see people naked. This is actually technology we already have today. You wouldn't want X-ray goggles, but infrared "goggles" would do the job just fine. Like X-rays, infrared light will pass right through clothes; unlike X-rays, it won't pass through skin. Hook up an IR camera to a VR headset and you're there.
- ge96 4y agoOne day I was messing around with an IR camera for RPi and was surprised to see the magnets inside of my black plastic closed headphones.
- Brian_K_White 4y agoIt isn't about skeletons, it's about clothing. If an ai can reconstruct a view of the far side of something from data like the near side view and the shadows & other effects on the surrounding environment, then it can do the same thing to reconstruct a view of what's underneath clothing, theoretically, which may be reasonably restated as eventually.
- keithwhor 4y agoI'd love to see indie game developers use these tools in combination with DALL-E to recreate the style of the PlayStation-era Final Fantasy games. 3d characters in 2d pre-rendered worlds. Presumably with the right tooling -- creating imagined geometry in DALL-E and generating boundaries programmatically or by hand, character models from friends / cosplayers found via craigslist and open animation sets -- you'd be able to generate a game of Final Fantasy VIII's caliber with team as small as a lone developer in only a few years, maybe shorter.
- janekm 4y agoI would propose that the reason you are nostalgic for the FF games of past is because of the unique creative vision of those games, which is not something that DALL-E can help you with much yet, though it's absolutely astonishing at mimicking other's creative styles. (there may still be a way it could help, if an artist created part of the style and used DALL-E to augment the visual production pipeline)
- avaer 4y agoDALL-E can't help in art direction but (somewhat surprisingly) GPT-3 plus a human scanning Pinterest can be an art director. I've spent hundreds of OpenAI dollars on this because it works. To me, that means a version DALL-E which can do art direction without me in the loop at all is not too far off.
- truckerbill 4y agoWhat does the GPT-3 element do?
- avaer 4y agoAs an lone indie dev making an AI-generated FOSS Final Fantasy, I can confirm it's possible :D. That era of games really fits well with the kind of content that AI is capable of generating today. I've been primarily tackling the synthesis from the GPT-3 + 3D animation side but the 2d prerender idea is genius. I'll need to try to hook it up one of these weekends. The only hard part, it seems, is finding the navmesh.
- dylan604 4y agoHow is this reconstruction and not just construction? There is no data there to reconstruct. It's just made up based on some hinted guessing and a lot of assumptions.
- robotresearcher 4y agoThere’s no accepted threshold between the two. The input image is obviously highly constraining the output model - the pose and clothing color are consistent - so this is within the domain norm to call this a reconstruction.
- dylan604 4y agoRight, but who's to say that the designer shirt doesn't have some sort of pattern or logo or totally different color on the back? If you're not recreating something from actual data elsewhere, it's just being made up on intuition level guessing. My concern is what the purpose of this "tech" is being used for, and more specifically, how this "tech" will be mis-used for other purposes.
- blenderdt 4y agoThis can be really useful for architectural visualizations. For archviz you don't need an accurate 3D model but one that looks good and casts shadows.
- Spivakov 4y agoImpressive. Not exactly the same thing but I have been thinking about the idea of 3D construction of cloth/jeans you want to purchase onto your own body, so that you have better idea before placing order, which could be really useful for online shopping. I was thinking in game graphic way (bottom-up), such as building 3D model of myself, shopping website providing 3D models of their clothes, and then it would be a "fitting" problem to properly place clothes onto the body. From this paper it seems that it is not necessary to have low-level data to achieve the objective.
- blueblisters 4y agoI think you would still need some form of 3D modeling if you want to achieve a good/accurate fit for apparel on your body.
- est 4y agophones with depth camera surely can help.
- spyder 4y agoIt's called virtual try on, there are lot of research papers about it: https://github.com/minar09/awesome-virtual-try-on https://github.com/minar09/awesome-virtual-try-on
- namibj 4y agoUnfortunately the human body deforms from clothing... I have most faith in energy-dispersive CT or MRT to provide the required information about bone shapes and tissue types to extrapolate how a garment works while moving. Even much more so for shoes that I think need shape customization the most.
- blincoln 4y agoIn the early 2010s, I did a security evaluation of Styku's Kinect-based fitting room. Back then, it was in a similar vein as what you described - the sensors would build a 3D model of your body, and then recommend the jeans that would fit best. Their original idea was even better, IMO. It was supposed to send the model to an on-demand automated clothing factory that would custom-sew entire pieces of clothing to your measurements. I think the technology wasn't quite there yet at the time. Even with four first-gen Kinects, the body model was probably too inaccurate to compare to an actual tailor. They seem to have moved away from clothing in general since then.
- jayd16 4y agoIs it just me or does it look like the AI is just pasting a blurry version of the front to fill in the back? You can see hints of necklines on backs, or the red from a dress showing where a black overcoat should be.
- esquivalience 4y agoGreat observation
- scoopertrooper 4y agoIt knows the neck area can’t be the same between front and back, but it doesn’t know if it should sample the skin or cloth textures to fill the gap, so it just takes a blend of both. I guess, by hedging this way, it managed to improve its evaluation score without having to learn how clothes actually work.
- SimplyUnknown 4y agoIt's definitely not perfect, but I'd argue that it is better than a "blurry front paste". For example, it doesn't place breasts on the back. Also, for the polo shirt example the model is doing significantly better than copying the v-line of the front of the shirt and shows a round collar. I think it is pretty impressive and one could argue the model has somewhat learned what a human looks like from the back given the clothing of the front, even though it is quite blurry.
- detay 4y agofunny how some low-poly 3d models wrapped with photos are being called "photorealistic" nowadays.
- lewispollard 4y agoYeah, it's photorealistic if your photo is about 200px high and you present the 3d models at the same size. As soon as you zoom in, it loses some of the wow factor, they look like models from The Sims.
- swframe2 4y agoThis will eventually link up with MetaHuman, DeepMotion and ZivaDynamics to get the high res end result. There a lot of money to be made. It is will happen in the next few years. We're very close.
- whoevercares 4y agoSpeaking as a former employee of a 3D startup that successfully exit - those technologies are never a game changer, they need a great business foundation to operate , such as a successful retail bussiness. It limits the business value of such startup. To make things worse, we have to always expecting the next big thing will come, like meta verse today, Kinect 10 years ago.
- qayxc 4y agoThey could be a great application for niches, though. Prototyping 3d games or movies can be helped greatly by such techniques. If you can take a photo (effort required: seconds) and turn it into a textures low- to mid-poly model within seconds, that's a huge time saver right there. Combined with other techniques [1], [2] this could indeed present a viable business model - again for certain niches. It's the same old routine: identify potential, present solution, profit (if only for a limited time). On its own this might not look too useful/impressive, but I'm certain there's people out there who could already benefit greatly from this work. It's not always about "the next big thing" or "turning an entire industry on its head". I could see this work its way into various products without much fanfare and gazillion dollar valuations. [1] https://github.com/Shimingyi/MotioNet https://github.com/Shimingyi/MotioNet [2] https://deepai.org/publication/end-to-end-learning-for-3d-facial-animation-from-raw-waveforms-of-speech https://deepai.org/publication/end-to-end-learning-for-3d-fa...
- gmueckl 4y agoCombine this with a webcam pointed at a busy street, some pose estimation and an auto rigger and you can steal people's identities with this in an automated way. Example? A fashion billboard that doesn't use model photos but renders passers-by in fancy new clothes. Of course, the gathered data gets syphoned of into a database somewhere.
- qayxc 4y agoThe example actually sounds quite amazing. There's much simpler ways (that are already in use, btw.) to "steal" people's identities. Smartphones and CCTV cameras are much more effective sources of information than a smart billboard anyway. Smart billboards, however, sound quite awesome.
- gmueckl 4y agoYou have to consider who the collected data is available to. CCTV in public spaces is usually the domain of law enforcement and thus the related data is available to law enforcement and potentially other government agencies, but not normally to private companies. A smart billboard and related data would be owned by a private company who may go on and process and sell it behind your back. It's complex. Computing and related tech is an enabler of so many new possible applications. It has become really tough to develop something useful for consumers that can be commercially successful, but doesn't at least have the potential to be turned into something creepy.
- macinjosh 4y agoIt is unnerving how many people are OK with governments using surveillance tech but draw the line with private corporations. At the end of the day they are just organizations made up of fallible humans. Either using advanced surveillance technology worries me deeply. Corporations have an increased ability to do things w/o oversight, but the government can literally do whatever it wants when it wants as long as those in power allow it. Government has the monopoly on power and violence. This concern is increased even more in democracies where there is great division. One side can abuse government to enforce their aims on an unwilling opposing side. Additionally, corporations generally just want to make money so they are fairly predictable and easy to contain (i.e take your business somewhere else if you don't want to deal with them). Government operates at the beck and call of political whims which, let's face it, are more often than not emotional and knee-jerk reactions. Lastly, only government can make it illegal to avoid surveillance and put you in jail for it. The worst a corp can do is close your account or something.
- est 4y agoSomeone combine this with a good apparel physics engine. Different materials should move differently and shines with different light source.
- Dave3of5 4y agoWhat would much cooler is if you had a drawing and you could convert that to a 3d Model. For gamedev you often have a drawing of a Character made up which then gets scuplted, retopo, texture ...etc. This is very time consuming especially for characters that are in the background. If you could create a bunch of characters from drawings that would make large crowd scenes much easier to create.
- sandworm101 4y agoIf fills in the backs, but it cannot do buts. Most of these pictures are professional models. They are attractive people. But their backsides are all flat as pancakes and thier cloaths obviously looser in the back then the fronts, which does a disservice to all the time they likely put in at the gym.
- mywacaday 4y agoHow long before actors become redundant, especially background extras.
- tiahura 4y agoI'd think there would be a lot more interest in without clothing.
- causality0 4y agoI'd love to see this released as usable software. I've wanted to print my own Eugene Goodman statue for a while.