5 ms·

What's in your hands? 3D Reconstruction of Generic Objects in Hands

28 points by lnyan 5y ago

hwers 5y ago
I know even publishing the code for stuff like this is generous but it'd be so cool if the standard practice included publishing a little wasm binary blob as a library independent network with weights to play with without having to go through colab. (Many frameworks have a way of creating that automatically already, e.g. torchscript or jax jit.)
- quanto 5y ago
  I am not as familiar with the state of the art; are there NN frameworks that export the NN model such that it can be run within WASM?
  - hwers 5y ago
    Yup. In torch you can use torchscript and in jax you can 'jit' a function and both turn your NN function into a fairly easily compliable form which could then be translated to wasm. Here's a cool project that does something like this (though I believe it's basically superseded by torchscript): https://github.com/podgorskiy/tensor4 https://github.com/podgorskiy/tensor4 Here it is running stylegan in the browser (via wasm): http://podgorskiy.com/static/stylegan/stylegan.html http://podgorskiy.com/static/stylegan/stylegan.html
- goodmattg 5y ago
  Note of reference for readers who may not be as familiar. One (pytorch) process is: 1) Use torch.onnx to convert the model to ONNX 2) USe `onnxruntime` built for Web Assembly (WASM) to run the NN in the browser
  - hwers 5y ago
    Well my point was that you could extract it into a wasm blob without onnx (library independent). See tensor4 as linked in the other thread.
quanto 5y ago
Instead of working from a single 2D image as this work does, what is the state of the art for 3D reconstruction from multiple (non-calibrated) images? e.g. some casual shots of an object using my phone.
- jgod 5y ago
  Photogrammetry apps, right?
- theschwa 5y ago
  It depends on the goal, but the whole family of NeRF techniques have been achieving some really interesting results.
- mkaic 5y ago
  probably Nvidia's latest Instant Neural Graphics Primitives paper (as far as I'm aware). they demonstrate near-instantaneous training of NeRF models: https://nvlabs.github.io/instant-ngp/ https://nvlabs.github.io/instant-ngp/
daenz 5y ago
Dashcam threat sensors identified the suspect was holding a handgun-shaped object... -A press conference in the near future

28 points by lnyan 5y ago

I know even publishing the code for stuff like this is generous but it'd be so cool if the standard practice included publishing a little wasm binary blob as a library independent network with weights to play with without having to go through colab. (Many frameworks have a way of creating that automatically already, e.g. torchscript or jax jit.)

quanto 5y ago
I am not as familiar with the state of the art; are there NN frameworks that export the NN model such that it can be run within WASM?
- hwers 5y ago
  Yup. In torch you can use torchscript and in jax you can 'jit' a function and both turn your NN function into a fairly easily compliable form which could then be translated to wasm. Here's a cool project that does something like this (though I believe it's basically superseded by torchscript): https://github.com/podgorskiy/tensor4 https://github.com/podgorskiy/tensor4 Here it is running stylegan in the browser (via wasm): http://podgorskiy.com/static/stylegan/stylegan.html http://podgorskiy.com/static/stylegan/stylegan.html
goodmattg 5y ago
Note of reference for readers who may not be as familiar. One (pytorch) process is: 1) Use torch.onnx to convert the model to ONNX 2) USe `onnxruntime` built for Web Assembly (WASM) to run the NN in the browser
- hwers 5y ago
  Well my point was that you could extract it into a wasm blob without onnx (library independent). See tensor4 as linked in the other thread.

quanto 5y ago

Instead of working from a single 2D image as this work does, what is the state of the art for 3D reconstruction from multiple (non-calibrated) images? e.g. some casual shots of an object using my phone.

jgod 5y ago
Photogrammetry apps, right?
theschwa 5y ago
It depends on the goal, but the whole family of NeRF techniques have been achieving some really interesting results.
mkaic 5y ago
probably Nvidia's latest Instant Neural Graphics Primitives paper (as far as I'm aware). they demonstrate near-instantaneous training of NeRF models: https://nvlabs.github.io/instant-ngp/ https://nvlabs.github.io/instant-ngp/

daenz 5y ago

Dashcam threat sensors identified the suspect was holding a handgun-shaped object... -A press conference in the near future