3 ms·
First: sigh. First-coda: past tense Second: I worked alongside a brilliant team with rare knowledge of geometry reconstruction. We took the 70 images of people
by peteforde 7y ago
First: sigh.
First-coda: past tense
Second: I worked alongside a brilliant team with rare knowledge of geometry reconstruction. We took the 70 images of people, algorithmically removed the backgrounds and then used a proprietary process to create a mesh geometry from the intersection masks of those shapes in 3D space, which all had to be carefully calibrated on the regular. That geometry was smoothed to create a valid smooth human mesh. We then created a texture which would be mapped to the physical geometry of the mesh. All lighting and camera sensors had to be carefully measured and corrected, with ample algorithmic color smoothing along the seams of this texture. Then we ran our proprietary auto-rigging algorithm on the mesh, and exported both a 3JS compatible obj+jpg as well as a Unity bundle. The entire process was optimized to take under 60 seconds. This meant that you could step out of the booth, watch the magic happen, and then immediately put on a Vive and interact with yourself in a mirror using your fully articulated new avatar.
Third: VCs were unwilling to fund something with a hardware component
Fourth: sigh
- abrookewood 7y agoSorry to hear that it didn't work out, but it's still an impressive achievement. 60s to produce something that lifelike is incredible. I'm surprised no game companies were interested.
- peteforde 7y agoBelieve it or not, a significant aspect of the struggle was with my partners. I wanted to focus on games and they wanted to make emoji meme apps. Some day, I will write a book. It’ll be a doozy.
- mmcwilliams 7y agoWas any of the geometry reconstruction or texture mapping done on the Pis themselves? I can imagine with 70 of them you’d have a nice little cluster to work with. Cool project, either way!
- peteforde 7y agoWe definitely tried to do on-board processing, as we were still bottlenecked by the blazing “gigabit over USB 2.0” NIC speed. However, while I have a hunch that the new Pi 4 might be more than up to some of the crunching, the real problem is that the spatial nature of the domain means that each Pi would need access to the sparse clouds and shape masks generated by the rest in order to proceed. We could do color correction and remove lens distortion on the Pi, but that’s it. Reflecting on this conversation, it occurs to me that I should strongly consider publishing a string of #1 on HN deep dives on what I learned to pay it forward. ;)
- mmcwilliams 7y agoI'd definitely read those if you wrote them. I build small machines that use Pi Zeros as motor controllers and I'm a sponge for domain knowledge about using them in practice. Your point about voltage fluctuation affecting clock speed already has me re-evaluating my current solution. Also I'm really curious about your custom cameras. Thanks for sharing!
- JVIDEL 7y agoWell that sucks, I could see this being used in VR arcades
- peteforde 7y ago"yep"