7 ms·

Hi I'm Marcus (one of the creators). Feel free to hit me up if you have any questions :-) We are "very alpha" right now. Edit: We have an API over at: https:/

Hi I'm Marcus (one of the creators). Feel free to hit me up if you have any questions :-) We are "very alpha" right now. Edit: We have an API over at: https://usdz.app/api https://usdz.app/api

genpfault 5y ago
What photogrammetry pipeline are you using?
- schappim 5y ago
  It is a stupidly long AWS based pipeline w/ the photogrammetry part is in swift.
- deleted 5y ago
  [deleted]
porkbrain 5y ago
If it's not a secret, I'd love to read some design document about the backend behind this app. What's the blackbox that takes set of images as an input and produces a 3D model as an output?
- schappim 5y ago
  We've very much built on the shoulders of giants: - Frontend & API (Rails + jumpstartrails.com) - Images --> USDZ (Swift Code on a Cloud hosted Mac) - USDZ --> GLTF (Swift + Custom Ruby) - CDN (AWS Cloudfront) Lots of Glue: - AWS S3 / Cloudfront - AWS SQS
  - deleted 5y ago
    [deleted]
- nobbis 5y ago
  It uses the Object Capture API introduced in the latest version of macOS. The API allows you to create 3D models from images with just a few lines of code. https://developer.apple.com/augmented-reality/object-capture https://developer.apple.com/augmented-reality/object-capture
  - schappim 5y ago
    If we're talking Swift frameworks, ModelIO (underrated) and ObjectCapture are certainly part of things.
    - nobbis 5y ago
      He asked for the “blackbox that takes set of images as an input and produces a 3D model as an output.” That black box is Object Capture API, which takes a folder of images and outputs an USDZ (or OBJ) file. Model I/O isn’t necessary. Kudos to Apple for releasing a photogrammetry pipeline that allows web and app developers to build tools like this, with zero knowledge of 3D reconstruction required.
      - schappim 5y ago
        You're not wrong (althought we do use ModelIO a lot). I just saw that you're from https://www.aboundlabs.com https://www.aboundlabs.com and metascan.ai Your scans look great! So how does your black box work?
      - secretsatan 5y ago
        I’ve just been trying out the photogrammetry session, it’s super simple to use. I think Apple’s photogrammetry software isn’t as advanced as other’s, but it is free and works well small scale. I work in the same field, making a capture app for photogrammetry software
        schappim 5y ago
        Yeah what Apple's has is a clear lead on the time required to build objects. I think that the photogrammetry side of things, the "give me an object now" will shift towards mobile device camera apps. It is only serverside now due to power requirements.
armagon 5y ago
Hi Marcus. It would be nice to have some idea going in for what the requirements are. - I have to admit, I don't understand why I can't preview the 3D experiences right on my computer. - Likewise, when I try the QR code on my iPhone 6 ... well, I'm hardly surprised to not see anything in 3D, but it wouldn't hurt to have a little more detail than "Scan this QR code on your iPhone or iPad." I suppose I could mention that I've done some game development and a little bit of VR/AR, but have never heard of the USDZ file format before.
- schappim 5y ago
  Hi there! USDZ is the format that Apple is backing for VR/AR delivery (developed by Pixar)[1]. We're betting that it is what is going to be used in any Apple headset. Right now you can't easily view USDZs in the browser (the format is a moving target), however Apple is making strides to make it available with the proposed Model element[2]. We were doing some USDZ --> GLTF converstion (GLTF being a format that will work in the browser), but it is jankey (technical term). I have not tried an iPhone 6 yet. I will have to pry it from my children's hands and remove a layer of cheese and excretions before I do. I've made a video on how to do this via the browser here[3]. TBH, the experience a heap better in the iOS App (currently in testflight) [4]. [1] https://graphics.pixar.com/usd/release/wp_usdz.html https://graphics.pixar.com/usd/release/wp_usdz.html (seem to have cert issues) [2] https://github.com/WebKit/explainers/tree/main/model https://github.com/WebKit/explainers/tree/main/model [3] https://usdz.app/screencasts/getting-started https://usdz.app/screencasts/getting-started [4] https://testflight.apple.com/join/VteIxFd7 https://testflight.apple.com/join/VteIxFd7
  - schappim 5y ago
    Oh, I should also mention, HEICs are your friend (as they have depth, and accelerometer data embedded in the file format).
nemacol 5y ago
Anyway to take a model that was created and get it into CAD / modeling software? I don't know much about VR/AR but I know it is a pain in the bum to replicate an object / create a model for 3d printing. Thank you.
- schappim 5y ago
  We allow you to download as USDZs (and sometimes do conversions to GLTFs). You can import both of these into Blender and export as Obj (and I'm pretty sure STLs). You may want to use our API to do the Images ---> USDZ conversion[1]. [1] https://usdz.app/api https://usdz.app/api
  - nemacol 5y ago
    Excellent. Thank you for the reply.
tluyben2 5y ago
This is really great, thanks for doing this!
ianbutler 5y ago
Hey very cool stuff, one piece of feedback it would be good to be able to delete models on the beta iPhone app, or if we can it’s not particularly obvious how to do so.
- schappim 5y ago
  You're right! Right now it is not possible (via the phone). It should be possible via the API or https://usdz.app https://usdz.app
thih9 5y ago
Pretty cool! How are you handling spam or other kinds of content moderation? Is it easier because users don’t upload a regular text/image and need to create something custom?
- schappim 5y ago
  There hasn't been anything humans can't handle. If it gets out of hand, we'll look at something similar to Akismet, or an ML based solution. Right now we're hand picking experiences to make it to the front page.
  - onara 5y ago
    At early stage you are not going to get spam anyway. Once it grows (and i hope it does) you will get fare share of spam. As you said at that point you could check solutions like Akismet, OOPSpam.
  - thih9 5y ago
    Thanks for answering! Hand picking sounds like a really good approach, simple and effective.
    - schappim 5y ago
      So funny you should mention this, yesterday we had our "first porn" moment. Thankfully if one throws a bunch of random images at the API and trys to make a model, the algorithm just fails. However we're thinking it might be fun to do something w/ NSFW image detection and display a silly model...
hpen 5y ago
Hey have you guys considered licensing an API?
- schappim 5y ago
  We don’t think is a good long term business model (with the obvious path of functionality moving to the client).
  - toomuchtodo 5y ago
    Smart. Looking forward to this being an on device capability.

Hi I'm Marcus (one of the creators). Feel free to hit me up if you have any questions :-) We are "very alpha" right now. Edit: We have an API over at: https://usdz.app/api https://usdz.app/api

genpfault 5y ago

What photogrammetry pipeline are you using?

schappim 5y ago
It is a stupidly long AWS based pipeline w/ the photogrammetry part is in swift.
deleted 5y ago
[deleted]

porkbrain 5y ago

If it's not a secret, I'd love to read some design document about the backend behind this app. What's the blackbox that takes set of images as an input and produces a 3D model as an output?

schappim 5y ago
We've very much built on the shoulders of giants: - Frontend & API (Rails + jumpstartrails.com) - Images --> USDZ (Swift Code on a Cloud hosted Mac) - USDZ --> GLTF (Swift + Custom Ruby) - CDN (AWS Cloudfront) Lots of Glue: - AWS S3 / Cloudfront - AWS SQS
- deleted 5y ago
  [deleted]
nobbis 5y ago
It uses the Object Capture API introduced in the latest version of macOS. The API allows you to create 3D models from images with just a few lines of code. https://developer.apple.com/augmented-reality/object-capture https://developer.apple.com/augmented-reality/object-capture
- schappim 5y ago
  If we're talking Swift frameworks, ModelIO (underrated) and ObjectCapture are certainly part of things.
  - nobbis 5y ago
    He asked for the “blackbox that takes set of images as an input and produces a 3D model as an output.” That black box is Object Capture API, which takes a folder of images and outputs an USDZ (or OBJ) file. Model I/O isn’t necessary. Kudos to Apple for releasing a photogrammetry pipeline that allows web and app developers to build tools like this, with zero knowledge of 3D reconstruction required.
    - schappim 5y ago
      You're not wrong (althought we do use ModelIO a lot). I just saw that you're from https://www.aboundlabs.com https://www.aboundlabs.com and metascan.ai Your scans look great! So how does your black box work?
    - secretsatan 5y ago
      I’ve just been trying out the photogrammetry session, it’s super simple to use. I think Apple’s photogrammetry software isn’t as advanced as other’s, but it is free and works well small scale. I work in the same field, making a capture app for photogrammetry software
      - schappim 5y ago
        Yeah what Apple's has is a clear lead on the time required to build objects. I think that the photogrammetry side of things, the "give me an object now" will shift towards mobile device camera apps. It is only serverside now due to power requirements.

armagon 5y ago

Hi Marcus. It would be nice to have some idea going in for what the requirements are. - I have to admit, I don't understand why I can't preview the 3D experiences right on my computer. - Likewise, when I try the QR code on my iPhone 6 ... well, I'm hardly surprised to not see anything in 3D, but it wouldn't hurt to have a little more detail than "Scan this QR code on your iPhone or iPad." I suppose I could mention that I've done some game development and a little bit of VR/AR, but have never heard of the USDZ file format before.

schappim 5y ago
Hi there! USDZ is the format that Apple is backing for VR/AR delivery (developed by Pixar)[1]. We're betting that it is what is going to be used in any Apple headset. Right now you can't easily view USDZs in the browser (the format is a moving target), however Apple is making strides to make it available with the proposed Model element[2]. We were doing some USDZ --> GLTF converstion (GLTF being a format that will work in the browser), but it is jankey (technical term). I have not tried an iPhone 6 yet. I will have to pry it from my children's hands and remove a layer of cheese and excretions before I do. I've made a video on how to do this via the browser here[3]. TBH, the experience a heap better in the iOS App (currently in testflight) [4]. [1] https://graphics.pixar.com/usd/release/wp_usdz.html https://graphics.pixar.com/usd/release/wp_usdz.html (seem to have cert issues) [2] https://github.com/WebKit/explainers/tree/main/model https://github.com/WebKit/explainers/tree/main/model [3] https://usdz.app/screencasts/getting-started https://usdz.app/screencasts/getting-started [4] https://testflight.apple.com/join/VteIxFd7 https://testflight.apple.com/join/VteIxFd7
- schappim 5y ago
  Oh, I should also mention, HEICs are your friend (as they have depth, and accelerometer data embedded in the file format).

nemacol 5y ago

Anyway to take a model that was created and get it into CAD / modeling software? I don't know much about VR/AR but I know it is a pain in the bum to replicate an object / create a model for 3d printing. Thank you.

schappim 5y ago
We allow you to download as USDZs (and sometimes do conversions to GLTFs). You can import both of these into Blender and export as Obj (and I'm pretty sure STLs). You may want to use our API to do the Images ---> USDZ conversion[1]. [1] https://usdz.app/api https://usdz.app/api
- nemacol 5y ago
  Excellent. Thank you for the reply.

tluyben2 5y ago

This is really great, thanks for doing this!

ianbutler 5y ago

Hey very cool stuff, one piece of feedback it would be good to be able to delete models on the beta iPhone app, or if we can it’s not particularly obvious how to do so.

schappim 5y ago
You're right! Right now it is not possible (via the phone). It should be possible via the API or https://usdz.app https://usdz.app

thih9 5y ago

Pretty cool! How are you handling spam or other kinds of content moderation? Is it easier because users don’t upload a regular text/image and need to create something custom?

schappim 5y ago
There hasn't been anything humans can't handle. If it gets out of hand, we'll look at something similar to Akismet, or an ML based solution. Right now we're hand picking experiences to make it to the front page.
- onara 5y ago
  At early stage you are not going to get spam anyway. Once it grows (and i hope it does) you will get fare share of spam. As you said at that point you could check solutions like Akismet, OOPSpam.
- thih9 5y ago
  Thanks for answering! Hand picking sounds like a really good approach, simple and effective.
  - schappim 5y ago
    So funny you should mention this, yesterday we had our "first porn" moment. Thankfully if one throws a bunch of random images at the API and trys to make a model, the algorithm just fails. However we're thinking it might be fun to do something w/ NSFW image detection and display a silly model...

hpen 5y ago

Hey have you guys considered licensing an API?

schappim 5y ago
We don’t think is a good long term business model (with the obvious path of functionality moving to the client).
- toomuchtodo 5y ago
  Smart. Looking forward to this being an on device capability.