9 ms·
Oculus: Hand Tracking Updates
- melling 6y agoWith the worry about Coronavirus, hand tracking with cameras or something like Google Soli[1] might be useful in ATM’s, elevators, kiosks, etc so people can avoid physically touching the devices. [1] https://atap.google.com/soli/ https://atap.google.com/soli/
- texascloud 6y agoAre there cameras on the sides of the occulus so peripheral hand movements can be recognized?
- krasin 6y agoYes, pretty much: https://thumbor.forbes.com/thumbor/960x0/https%3A%2F%2Fblogs-images.forbes.com%2Fsolrogers%2Ffiles%2F2019%2F05%2F190402_Quest_FH_dark_color-1200x675.jpg https://thumbor.forbes.com/thumbor/960x0/https%3A%2F%2Fblogs...
- jackbrookes 6y agoAs well as SLAM and tracking the controllers.
- gegtik 6y agoI really want to know why this is feasible for the Quest but not the Rift S, given that both have Inside-Out tracking (ie. IR cameras mounted on the headset). In fact, the Rift S has 5 cameras to the Quests's 4 so it should be at least as capable shouldn't it?
- Impossible 6y agoInternal company politics and prioritization of Quest. It's possible that hand tracking will come to Rift S, but it might be a side effect of wanting to support it with the Quest link cable.
- whywhywhywhy 6y agoJust the reality that headsets like Quest are the true future of VR, PCVR is an evolutionary dead end a very small percentage of people who can afford them want a gaming PC tower in their home, PCVR is a fraction of a fraction of percentage. I own both and demoing my Rift on a high spec system my friends were blown away and the first question was "Can it run on my macbook?" Then thats where the idea of owning VR died for them. Demoed my Quest and one person bought one within 2 minutes of taking the headset off, took his to a family BBQ and two more people there bought it after trying it for a few minutes. I still enjoy many PCVR games but this is just reality.
- Miraste 6y agoPCVR capabilities are too high for it to be a dead end. More niche, absolutely, because mobile VR/AR has somewhere from console to smartphone-level growth potential--but a smaller, higher-paying market for premium experiences is going to continue.
- Impossible 6y agoIt's unlikely that PC will ever be mass market, but when/if VR makes more in roads in productivity applications PC will be key. There is a lot of promise in art creation, game development and 3D content creation, and of course HN's traditional programmer dream of infinite terminals, but the real killer app is telepresence for remote meetings. All of those use cases need better software and hardware
- whywhywhywhy 6y ago>There is a lot of promise in art creation, game development and 3D content creation Absolutely agree with this, literally anyone who can use the headset is able to sculpt a 3D model in Oculus Medium that resembles what they want. Same can't be said for any non-vr 3D modelling software, huge potential in the creative industries. Still niche and not mainstream though.
- deleted 6y ago
- MikusR 6y agoQuest does the processing in the headset and it is very probably accelerated.
- boolcow 6y agoRift S is connected to a far more powerful gaming PC. It seems like Oculus simply treats the Rift S and their customers as a second-class.
- mentat 6y agoLatency even over a few feet can matter.
- mrlala 6y agoBy that argument aren't the controllers experiencing more latency on RiftS than Quest already? I realize there is more post-processing with hand tracking, but given that it already experiences the latency of having to process the control actions... not really sure how this could affect it that much. Also, you could argue the hand tracking calculations could be done faster on a PC, so even if there is some perceived latency it could balance out or be better... would really need numbers for all of this, but it does just seem very suspect that they are trying to push more things on the Quest intentionally..
- rubicon33 6y agoThis is definitely looking more and more true. Consider the fact that the Rift-S has been out of stock for months. Oculus continues to re-stock the Quest (which sells out within 24h), but we haven't seen a restock of the Rift-S in probably 3 months? Their "Del Mar" which is coming up, is very likely not going to be a tethered headset. My money is it will be standalone like the Quest. Oculus is trying to get the masses to adopt VR and I salute them for it. There will always be headsets like the Valve index for enthusiasts.
- saurik 6y ago
- shafyy 6y agoI would be suprised if it's primarily a technological reason. My thinking is: The Rift S is 99.99% used for "serious" gaming and PCVR will always be that way. For serious gaming you'll probably need some kind of controllers for the near future, since you want that fidelity and haptic feedback. Now, controllers could be replaced some day by braceletes or what not, but this is not going to happen next year. The Quest and its successors (i.e. stand-alone VR), on the other hand, is and will be more and more used for lighter gaming, video, browsing, productivity, fitness, social etc. Often times, hand-tracking will be enough without the need for controllers. Therefore, they focus their resources on bringing hand-tracking only to the Quest.
- aylmao 6y ago+1. Also worth noting, the implementation would be fundamentally different. On the Quest it's running on-device on specific chips, whereas on the Rift S this processing would be done on the user's PC. It's doable, but you probably can't just drop the implementation for hand-tracking on one platform and have it work on the other.
- zmmmmm 6y agoMy guess is it's partly that but also mixed in with aligning to their mass market consumer ambitions. I've given my Quest to countless people to try out and while every single person is blown away, many / most of them are challenged to use the controllers as anything more than simple pointing devices without significant training. With hand tracking that all goes away. You just use your hands how you normally use them and it "just works". It's similar to how the original iPhone captured everybody's imaginations because it offered such intuitive gestures to control its UI. If you see the Quest as their "mass market" device and then you consider that their top priority is to knock over every barrier that stops it spreading like wildfire, then it's a no brainer to eliminate the controllers.
- Caligatio 6y agoThe Quest can link to your computer and essentially act as a Rift using a USB cable (needed to be USB 3.0, a recent beta update knocked that down to USB 2). I can easily see the Rift being removed from the product line in the future.
- ccktlmazeltov 6y agoWhat's the point of a Rift since you can stream to your Quest now?
- rpdillon 6y agoIt's a quirk of the Qualcomm chipset they used in the Quest. It has an extra DSP accelerator that was originally unused, but they managed to repurpose it for hand tracking. It was an opportunistic play, as I understand it.
- jachee 6y agoCouldn't they tap into the Host PC's compute power for tethered headsets? It feels like they're trying to exclude anyone who wants to play non-Oculus-published games.
- SimpleMinds 6y agoCould you link to a source about that dsp being specifically used for hand-tracking? My quick google didn't found a result that could confirm that and it would explain a bit. Thank you!
- corndoge 6y agohttps://www.youtube.com/watch?v=PMIDaomx0GA https://www.youtube.com/watch?v=PMIDaomx0GA It's in here somewhere. It's also extensively covered in the hour breakout session by the folks who implemented it, which is uploaded somewhere though I can't find the link atm.
- rpdillon 6y agoCarmack's keynote that corndoge linked to is exactly my source. I'm not sure about one that's more searchable, unfortunately.
- xdxdx 6y agoAlso referenced in this paper (section 5): https://research.fb.com/publications/machine-learning-at-facebook-understanding-inference-at-the-edge/ https://research.fb.com/publications/machine-learning-at-fac...
- SimpleMinds 6y agoThank you everyone!
- TwoBit 6y agoRift S could do hand tracking better than Quest, but it requires a separate effort that isn't currently a high priority. Source: I worked on it at Oculus.
- corndoge 6y agoHow was / is your overall experience at Oculus?
- rock_hard 6y agoDifferent camera setup on both device would have required twice the amount of training data for the models So they rather doubled down on the more promising form factor
- nkurz 6y agoDoes the Hand Tracking API allow for tracking other objects? In particular, could a long straight object be tracked as a pointer based on tracking two points along its axis? This seems like it would be computationally easier than tracking hands, and would open up a lot of possibilities for using real world objects as props within the VR environment.
- evan_ 6y agoNot at this time, hands only. I imagine it might be difficult to distinguish a pointer from other straight lines in the room, hands are a little more distinctive.
- GraffitiTim 6y agoI imagine some kind of VR tracker for the Quest will come out eventually. Something like the Vive trackers but smaller. Agree, it would open up all kinds of interesting applications.
- pharke 6y agoThey could be wildly successful if instead of $100 pucks they could just sell $1 stickers that you could attach to whatever you want. With enough of them you could probably use it to aid scanning in geometry of real world objects
- whoisjuan 6y agoObjects would create occlusion which will block the camera from being capable to track finger vectors.
- latchkey 6y agoOh nice! I just bought Quest over the weekend from a guy off craigslist who was selling them at cost and still wrapped in plastic. Fantastically impressed with things. Totally worth the $500. The games are amazing, but what is really impressing me is all the vr video content. The ability to "stand" on a stage while the orchestra plays Mozart all around you is mind blowing. It is neat to see them pushing the boundaries with updates.
- rubicon33 6y agoProps to the guy selling them at cost. Easily could have made an additional $50-$100 if he wanted to.
- gfodor 6y agoIt's Facebook -- clearly they expect to make up that margin somehow. It's obvious how.
- latchkey 6y agoYea, I was surprised. He had them in boxes that were clearly shipped from Best Buy. I asked if he got them cheaper and he said no. Didn't ask why he was selling them at cost, but I thought it was nice.
- rgovostes 6y agoApple's first published ML paper[1] described how they trained a neural network to estimate eye gaze. To augment the limited data set of labeled eye photos, they rendered synthetic eyes and then used adversarial training to make them match the real-life data set. This paper came out several months before the iPhone X introduced Face ID with eye gaze tracking, so it can be seen as a hint of things that were to come. What's intriguing to me is that the paper also talks about doing the same with synthetic depth maps of hands, adding details such as occlusion and imperfectly-captured edges. With Apple investing heavily in ARKit, and rumors of an AR headset, it seems like hand tracking is inevitable. 1: https://arxiv.org/abs/1612.07828 https://arxiv.org/abs/1612.07828
- sytse 6y agoWith hand tracking they mean finger tracking, seeing you 'click' with your index finger and thumb. The Oculus has hand tracking today via controllers already.
- Impossible 6y agoThey mean handtracking. Oculus touch is a tracked controller, which means the controller is tracked and the tracking system has no concept of a hand. You could put the controller on your foot, or the floor or on your dog and it would track in the same way as your hand. Hand tracking uses computer vision to recognize hands, for most humans fingers are included in that
- TeMPOraL 6y agoAnd Quest has that proper hand tracking already (had it for something like 5 months now), it's just been an experimental feature until now. I haven't used it much; for some reason the tracked hands seem to be placed lower than the fake hands when I'm using controllers, which makes the latter a better experience for me.
- Impossible 6y agoIncreased latency due to using complex, deep learning based, entirely computer vision driven hand tracking vs. tracked controllers which partially use accelerometer data to correct drift and account for latency, and use a much faster classic computer vision algorithm for position tracking (https://developer.oculus.com/blog/tracking-technology-explained-led-matching/ https://developer.oculus.com/blog/tracking-technology-explai...)
- TeMPOraL 6y agoDo Deep Learning-based methods introduce all that much latency? My understanding of neural network is that the training is expensive, but using them is essentially free - you have to push your pixels through a few layers of matrix multiplications and you have the results; it's just tuning those matrices that takes time.
- mark_l_watson 6y agoMy Oculus Quest is the best toy that I have bought for myself in years, even more enjoyable since I did VR for SAIC and Disney about 20+ years ago, so super exciting to see such a fun low cost consumer VR device. I can’t wait to try a game/experience using hand tracking. Lot’s of great content, my favorites being the Darth Vader trilogy and the RX racket ball and ping pong games.
- bsanr2 6y agoAt the risk of being a downer: I'll be so happy once we move past the "preschool playset" era of hand-tracked software. I wonder what it is that's keeping us from moving past these dull tech demos. Is the issue the interaction design being too simplistic ("Look, I can pick [thing] up!")? Or perhaps the opposite: that, without suitable haptics, too much energy has to be devoted to a very unnatural way of interacting (essentially with phantoms). It seems that games like BeatSaber and apps like TiltBrush have cracked the code, but you have to wonder how.
- ninenines 6y agoNot exactly as powerful but something similar and that can run on a browser, opening up to a lot of creative applications, handpose powered by tf.js(similar model exists for tflite) Demo: https://storage.googleapis.com/tfjs-models/demos/handpose/index.html https://storage.googleapis.com/tfjs-models/demos/handpose/in... Repo: https://github.com/tensorflow/tfjs-models/blob/master/handpose/README.md https://github.com/tensorflow/tfjs-models/blob/master/handpo... Tflite model: https://github.com/google/mediapipe/tree/master/mediapipe/models/hand_landmark_3d.tflite https://github.com/google/mediapipe/tree/master/mediapipe/mo... there is lighter 2d model as well