3 ms·

Thank you for the suggestions! 1. I'm ordering a microscope mount right now! We went with 3D printing because we could get a very good coupling between the sti

Thank you for the suggestions! 1. I'm ordering a microscope mount right now! We went with 3D printing because we could get a very good coupling between the stick and the robot mount. See this figure for a comparison: https://i.imgur.com/vWopcFB.jpeg https://i.imgur.com/vWopcFB.jpeg 2. Our dataset (and dataset export code) gives the depth as numpy files and trajectory information in a JSON file with calibration and trajectory information! The r3d export format from the Record3D app we used is simply a .zip file with renamed extension so we were ok with using it. Do you have suggestion as to which cinematic format you are thinking?

beiller 3y ago
I think you will find openexr surprisingly well supported across many platforms.
- MahiShafiullah 3y ago
  OpenEXR looks very interesting! Especially if OpenEXR can compress depth data nearly as efficiently as we can compress image data by making everything a video, it would be a game changer. For example, our RGB dataset is 814 MB right now while the RGB-D dataset is 77 GB, even though depth "just" adds one new channel to our data.
  - startupsfail 3y ago
    Yes, that’s another thing. When Apple is recording RGBD in a cinematic format, it is tiny.
startupsfail 3y ago
Yes, Record3D is an option. The problem with it, you’ll never get large scale data in the format recorded by a custom app. If you use some standard app and format that is used by general public, there is a bit higher chance. For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app. In a longer term the stick is likely not needed too. Ideally you’d want a human demonstrating with regular human tool, like pliers to generalize. This is a particular mount that I’ve used. https://www.amazon.com/Starboosa-Smartphone-Microscope-Universal-Telescope/dp/B082WY3BF5 https://www.amazon.com/Starboosa-Smartphone-Microscope-Unive... It’s pretty solid, but it doesn’t open completely, so you need to disassemble and reassemble the stick to put it on (which is not that difficult).
- MahiShafiullah 3y ago
  > For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app. I agree with you there. My impression was that the cinematic format is closed, unlike .r3d which is a bunch of open format files in a .zip. Do you know whether Apple has published the specs for the cinematic video format anywhere, or if there are there good libraries for handling it?
saulrh 3y ago
Also look into the file formats that the robotics community use, stuff like rosbag and MCAP. These are formally more like point clouds than cinematic video captures and they won't integrate well with the video-processing side of things, but if you're dumping depths and trajectories from numpy and shuttling them to other robotics stuff then they might be a better fit than a 3d video format. In particular, if you end up adding anything other than video-like data streams - temperature, internal estimates from SLAM, etc - you'll have a much better time adding it to an mcap or rosbag than you would trying to shoehorn it into a video stream format.
- dhon_ 3y ago
  Rosbag is just a capture of published Ros topics which can include point clouds, image/video data, imu, pose etc

beiller 3y ago

I think you will find openexr surprisingly well supported across many platforms.

MahiShafiullah 3y ago
OpenEXR looks very interesting! Especially if OpenEXR can compress depth data nearly as efficiently as we can compress image data by making everything a video, it would be a game changer. For example, our RGB dataset is 814 MB right now while the RGB-D dataset is 77 GB, even though depth "just" adds one new channel to our data.
- startupsfail 3y ago
  Yes, that’s another thing. When Apple is recording RGBD in a cinematic format, it is tiny.

startupsfail 3y ago

Yes, Record3D is an option. The problem with it, you’ll never get large scale data in the format recorded by a custom app. If you use some standard app and format that is used by general public, there is a bit higher chance. For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app. In a longer term the stick is likely not needed too. Ideally you’d want a human demonstrating with regular human tool, like pliers to generalize. This is a particular mount that I’ve used. https://www.amazon.com/Starboosa-Smartphone-Microscope-Universal-Telescope/dp/B082WY3BF5 https://www.amazon.com/Starboosa-Smartphone-Microscope-Unive... It’s pretty solid, but it doesn’t open completely, so you need to disassemble and reassemble the stick to put it on (which is not that difficult).

MahiShafiullah 3y ago
> For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app. I agree with you there. My impression was that the cinematic format is closed, unlike .r3d which is a bunch of open format files in a .zip. Do you know whether Apple has published the specs for the cinematic video format anywhere, or if there are there good libraries for handling it?

saulrh 3y ago

Also look into the file formats that the robotics community use, stuff like rosbag and MCAP. These are formally more like point clouds than cinematic video captures and they won't integrate well with the video-processing side of things, but if you're dumping depths and trajectories from numpy and shuttling them to other robotics stuff then they might be a better fit than a 3d video format. In particular, if you end up adding anything other than video-like data streams - temperature, internal estimates from SLAM, etc - you'll have a much better time adding it to an mcap or rosbag than you would trying to shoehorn it into a video stream format.

dhon_ 3y ago
Rosbag is just a capture of published Ros topics which can include point clouds, image/video data, imu, pose etc