5 ms·

RealityScan Beta: Mobile Photogrammetry

88 points by dagmx 5y ago

adampk 5y ago
What types of features are you using for the SfM pipeline? Algorithmic (SIFT) or ML based?
ilovefood 5y ago
This looks very interesting. Lately I've been playing a lot with the new photogrammetry API that Apple provides in Monterey on M1 (cloud instance) and it's working exceptionally well already compared to desktop based solutions to scan cars and humans. It fits my workflow of just using a phone, sending a video and getting a 3D model back (of course with some plumbing). I wonder how this compares and most importantly how the licensing model will look like? Is the processing done on the device or on a remote server farm? However, my congratulations on the release of the beta! I'm downloading it as we speak.
- deleted 5y ago
  [deleted]
- axg11 5y ago
  Are there any existing apps that use this API or another easy way for me to test this?
  - gecko39 5y ago
    If you have a compatible mac there are several free apps like this one: https://apps.apple.com/us/app/3d-scanner-app/id1419913995 https://apps.apple.com/us/app/3d-scanner-app/id1419913995
  - secretsatan 5y ago
    It's trivially easy to build one if you have a mac, they even have a demo project https://developer.apple.com/documentation/realitykit/creating_a_photogrammetry_command-line_app/ https://developer.apple.com/documentation/realitykit/creatin...
fxtentacle 5y ago
That's the power of Epic Games' deep pockets. Many people developing UE games need 3D assets, so Epic bought Quixel, a 3D asset store: http://www.cgchannel.com/2019/11/epic-games-acquires-quixel/ http://www.cgchannel.com/2019/11/epic-games-acquires-quixel/ Also people need to share their 3D models for colleagues, marketing, fans, etc. to look at. Quixel was using them internally anyway, so Epic bought Sketchfab: https://www.epicgames.com/site/en-US/news/sketchfab-is-now-part-of-epic-games https://www.epicgames.com/site/en-US/news/sketchfab-is-now-p... And people want to create their own 3D assets, so Epic bought the company that Quixel was using for 3D scanning, called Capturing Reality: https://www.epicgames.com/site/en-US/news/capturing-reality-is-now-part-of-epic-games https://www.epicgames.com/site/en-US/news/capturing-reality-... And now you can go from reality to virtual twin to realtime 3D visualization without ever leaving Epic Games' servers.
- Melatonic 5y ago
  People do not realize how big this is going to be - Epic is set up (and already is) to be huge in this space for cinema, TV, gaming, etc.
  - warning26 5y ago
    Anyone remember Epic's early DOS games, like Epic Pinball? How far they've come.
    - zactato 5y ago
      Exactly! At no point while playing ZZT in 1991 did I think "I bet this company is going to dominate the film industry 30 years from now"
- hunterb123 5y ago
  > without ever leaving Epic Games' servers That's probably a major driving point to the strategy after the Xbox Underground hacks. Fun read of the tale: https://www.blackhatethicalhacking.com/articles/free-access/hacking-stories-xbox-underground/ https://www.blackhatethicalhacking.com/articles/free-access/... They got into Epic's network and that gave them access to many other companies. Funny enough it wasn't the Apache simulator stuff that broke the camel's back, but the Gears of War 3 leak and Xbox One prototypes. I remember the Gears 3 leak and at the time the conspiracies that Microsoft leaked it themselves: https://kotaku.com/is-this-where-gears-of-war-3-was-leaked-from-5819239 https://kotaku.com/is-this-where-gears-of-war-3-was-leaked-f...
jamesmcintyre 5y ago
Anyone have suggestions for the easiest software to build a 3d environment (like in their demo) and drop in scanned models? Or at least a path to accomplishing this with the lowest barriers to entry?
- spicybright 5y ago
  Maybe find some demos in the engine of your choice then import the model in?
- danuker 5y ago
  About building the 3D model: There is Regard3D, if by barriers to entry you mean sensor hardware: http://www.regard3d.org/index.php/documentation/introduction http://www.regard3d.org/index.php/documentation/introduction If by barriers to entry you mean technical skills, there are SaaS solutions like Beholder: https://beholder.vision/ https://beholder.vision/ About embedding the demo in a web page, I have no idea.
- TheaomBen 5y ago
  Hi, I'm currently working on a project in that area. The main workflow in the system is take a bunch of pictures, SIFT & match them, then generate a sparse model of 2d features (image coords), 3d features (point cloud in relative coords) and camera positions. We then shard and re-combine different models to synthesize larger or more precise reconstructions. I'm building with clojurescript & Three.js[0] in the frontend, and can recommend the stack Back to your comment I'd love to hear what you're looking to do and what you'd need, sounds like I may have explored the same problem space. [0] https://threejs.org/ https://threejs.org/
Xenixo 5y ago
The chair looks awesome! Can't wait :) I digitized my flat with an m1 ipad and that looks already okay. It's funny to see the flat layout after doing the 3d scan. An interesting way to persistent the flat memory.
ge96 5y ago
Hope iPhone use wasn't related to using Lidar aside from the images.
- deleted 5y ago
  [deleted]
- andybak 5y ago
  And I hope someone has finally found a way to enhance photogrammetry with depth sensor data because i'm pretty sure both combined has the potential to produce better results than either alone.
  - longtimelistnr 5y ago
    The photogrammetry api in macOS Monterey does this I believe
smoldesu 5y ago
I'm a little torn on this. On the one hand, this is really impressive for on-the-fly photogrammetry. On the other hand though, this asset kinda just looks... bad? From a distance it's pretty passable, and in a relative vacuum of visual information it looks fine. However, the material itself is really sketchy. Whatever their default shader settings are makes the leather look toy-ish, caught somewhere between a reflective and diffuse material. The texturing is pretty low-quality (as expected), but the really annoying part is how it handles the ambient occlusion on the surface: it just looks like self-shadowing is baked into the texture. It's a cool proof-of-concept, but I'm not quite sure who could use it in it's current state. Even by video game standards, these assets would probably look really out-of-place...
- mkaic 5y ago
  As a 3D artist who uses a lot of photoscans -- the scan quality looks fantastic to me! Photoscanned assets aren't usually meant to hold up up close, they're more intended for quickly building out a larger scene. Additionally, the material you see straight out of the app can be enhanced significantly with some common shader tricks like faking a bump/roughness map. Lastly, good lighting makes a world of difference when using photoscanned assets. Personally, I've just joined the beta and expect I'll be taking advantage of it heavily for several upcoming CGI projects.
  - smoldesu 5y ago
    All good! I'm not trying to detract from anyone's enjoyment/use of the product if they have one :p I'm just a 3D hobbyist, and I don't really do a lot of photo scans. My thoughts on the matter are coming purely from a CG/asset generation perspective, if you're an artist with a little more agency with how you represent 3D objects then I bet you could work wonders with this (especially the geometry portion). Perhaps it was silly for me to expect these scans to rival hand-made assets, but my larger point was that we simply have a long ways to go before we reach parity with the two. In the time that's passed, I actually did come up with a use for it though. It might be fun to use Blender's layer functionality to "trace" a 3D scan into a higher-res production model, just to get proportions right. I'll probably still hold off until this reaches general availability though.

88 points by dagmx 5y ago

What types of features are you using for the SfM pipeline? Algorithmic (SIFT) or ML based?

This looks very interesting. Lately I've been playing a lot with the new photogrammetry API that Apple provides in Monterey on M1 (cloud instance) and it's working exceptionally well already compared to desktop based solutions to scan cars and humans. It fits my workflow of just using a phone, sending a video and getting a 3D model back (of course with some plumbing). I wonder how this compares and most importantly how the licensing model will look like? Is the processing done on the device or on a remote server farm? However, my congratulations on the release of the beta! I'm downloading it as we speak.

deleted 5y ago
[deleted]
axg11 5y ago
Are there any existing apps that use this API or another easy way for me to test this?
- gecko39 5y ago
  If you have a compatible mac there are several free apps like this one: https://apps.apple.com/us/app/3d-scanner-app/id1419913995 https://apps.apple.com/us/app/3d-scanner-app/id1419913995
- secretsatan 5y ago
  It's trivially easy to build one if you have a mac, they even have a demo project https://developer.apple.com/documentation/realitykit/creating_a_photogrammetry_command-line_app/ https://developer.apple.com/documentation/realitykit/creatin...

fxtentacle 5y ago

That's the power of Epic Games' deep pockets. Many people developing UE games need 3D assets, so Epic bought Quixel, a 3D asset store: http://www.cgchannel.com/2019/11/epic-games-acquires-quixel/ http://www.cgchannel.com/2019/11/epic-games-acquires-quixel/ Also people need to share their 3D models for colleagues, marketing, fans, etc. to look at. Quixel was using them internally anyway, so Epic bought Sketchfab: https://www.epicgames.com/site/en-US/news/sketchfab-is-now-part-of-epic-games https://www.epicgames.com/site/en-US/news/sketchfab-is-now-p... And people want to create their own 3D assets, so Epic bought the company that Quixel was using for 3D scanning, called Capturing Reality: https://www.epicgames.com/site/en-US/news/capturing-reality-is-now-part-of-epic-games https://www.epicgames.com/site/en-US/news/capturing-reality-... And now you can go from reality to virtual twin to realtime 3D visualization without ever leaving Epic Games' servers.

Melatonic 5y ago
People do not realize how big this is going to be - Epic is set up (and already is) to be huge in this space for cinema, TV, gaming, etc.
- warning26 5y ago
  Anyone remember Epic's early DOS games, like Epic Pinball? How far they've come.
  - zactato 5y ago
    Exactly! At no point while playing ZZT in 1991 did I think "I bet this company is going to dominate the film industry 30 years from now"
hunterb123 5y ago
> without ever leaving Epic Games' servers That's probably a major driving point to the strategy after the Xbox Underground hacks. Fun read of the tale: https://www.blackhatethicalhacking.com/articles/free-access/hacking-stories-xbox-underground/ https://www.blackhatethicalhacking.com/articles/free-access/... They got into Epic's network and that gave them access to many other companies. Funny enough it wasn't the Apache simulator stuff that broke the camel's back, but the Gears of War 3 leak and Xbox One prototypes. I remember the Gears 3 leak and at the time the conspiracies that Microsoft leaked it themselves: https://kotaku.com/is-this-where-gears-of-war-3-was-leaked-from-5819239 https://kotaku.com/is-this-where-gears-of-war-3-was-leaked-f...

jamesmcintyre 5y ago

Anyone have suggestions for the easiest software to build a 3d environment (like in their demo) and drop in scanned models? Or at least a path to accomplishing this with the lowest barriers to entry?

spicybright 5y ago
Maybe find some demos in the engine of your choice then import the model in?
danuker 5y ago
About building the 3D model: There is Regard3D, if by barriers to entry you mean sensor hardware: http://www.regard3d.org/index.php/documentation/introduction http://www.regard3d.org/index.php/documentation/introduction If by barriers to entry you mean technical skills, there are SaaS solutions like Beholder: https://beholder.vision/ https://beholder.vision/ About embedding the demo in a web page, I have no idea.
TheaomBen 5y ago
Hi, I'm currently working on a project in that area. The main workflow in the system is take a bunch of pictures, SIFT & match them, then generate a sparse model of 2d features (image coords), 3d features (point cloud in relative coords) and camera positions. We then shard and re-combine different models to synthesize larger or more precise reconstructions. I'm building with clojurescript & Three.js[0] in the frontend, and can recommend the stack Back to your comment I'd love to hear what you're looking to do and what you'd need, sounds like I may have explored the same problem space. [0] https://threejs.org/ https://threejs.org/

Xenixo 5y ago

The chair looks awesome! Can't wait :) I digitized my flat with an m1 ipad and that looks already okay. It's funny to see the flat layout after doing the 3d scan. An interesting way to persistent the flat memory.

ge96 5y ago

Hope iPhone use wasn't related to using Lidar aside from the images.

deleted 5y ago
[deleted]
andybak 5y ago
And I hope someone has finally found a way to enhance photogrammetry with depth sensor data because i'm pretty sure both combined has the potential to produce better results than either alone.
- longtimelistnr 5y ago
  The photogrammetry api in macOS Monterey does this I believe

smoldesu 5y ago

I'm a little torn on this. On the one hand, this is really impressive for on-the-fly photogrammetry. On the other hand though, this asset kinda just looks... bad? From a distance it's pretty passable, and in a relative vacuum of visual information it looks fine. However, the material itself is really sketchy. Whatever their default shader settings are makes the leather look toy-ish, caught somewhere between a reflective and diffuse material. The texturing is pretty low-quality (as expected), but the really annoying part is how it handles the ambient occlusion on the surface: it just looks like self-shadowing is baked into the texture. It's a cool proof-of-concept, but I'm not quite sure who could use it in it's current state. Even by video game standards, these assets would probably look really out-of-place...

mkaic 5y ago
As a 3D artist who uses a lot of photoscans -- the scan quality looks fantastic to me! Photoscanned assets aren't usually meant to hold up up close, they're more intended for quickly building out a larger scene. Additionally, the material you see straight out of the app can be enhanced significantly with some common shader tricks like faking a bump/roughness map. Lastly, good lighting makes a world of difference when using photoscanned assets. Personally, I've just joined the beta and expect I'll be taking advantage of it heavily for several upcoming CGI projects.
- smoldesu 5y ago
  All good! I'm not trying to detract from anyone's enjoyment/use of the product if they have one :p I'm just a 3D hobbyist, and I don't really do a lot of photo scans. My thoughts on the matter are coming purely from a CG/asset generation perspective, if you're an artist with a little more agency with how you represent 3D objects then I bet you could work wonders with this (especially the geometry portion). Perhaps it was silly for me to expect these scans to rival hand-made assets, but my larger point was that we simply have a long ways to go before we reach parity with the two. In the time that's passed, I actually did come up with a use for it though. It might be fun to use Blender's layer functionality to "trace" a 3D scan into a higher-res production model, just to get proportions right. I'll probably still hold off until this reaches general availability though.