9 ms·

Ultraleap – tactile sensations projected onto your hands

155 points by leontrolski 6y ago

leontrolski 6y ago
Here's a link to a recently published paper - "Using Spatiotemporal Modulation to Draw Tactile Patterns in Mid-Air" - https://link.springer.com/chapter/10.1007/978-3-319-93445-7_24 https://link.springer.com/chapter/10.1007/978-3-319-93445-7_...
frfl 6y ago
What affect does this have on dogs (if any)?
- Blammar 6y ago
  I'm thinking the same thing that happened with the Navy sonar and the beached beaked whales... (see https://royalsocietypublishing.org/doi/pdf/10.1098/rspb.2018.2533 https://royalsocietypublishing.org/doi/pdf/10.1098/rspb.2018...)
deleted 6y ago
[deleted]
reilly3000 6y ago
Man I've been harping on haptic holograms being the next UI mode since 2014, and this takes us half-way there. Congrats for getting some products to market in this space! I feel like there are a couple more years of VR stagnation, but by 2023-5 when 3mn SOCs are pretty common and affordable I think it will all hit at once and haptics will be a huge mode of interaction.
- mywacaday 6y ago
  Why does 3nm make a big difference?
  - reilly3000 6y ago
    Incredibly low power draw in the low end, which allows for more mobility and diverse sources of energy. Like if you could wear meaningful compute that is powered with body kinetic energy or some flexible solar thing, I think that unlocks computing to happen in many places with many physical and virtual things working in concert. ESP-32 on BLE sleep can sip tiny amounts of energy, but cannot do any meaningful compute, and a low top end speed. I want a chip that can burst to 5GHZ and rest at 0.05mha. That can make generating or streaming a massive bitrate from a wearable. Then you can do sick-ass offline-first edge compute and be able to drive one of these haptics devices, projectors, VR, whatever with a tiny machine or local cluster. Wearable k3s if you will. If you could drive any display from your person, with local and remote indexes and objects of everything you want or need, interfaces to people and things, replay of everything, totally offline as you want or need. If it was fast enough, and didn’t require a whole backpack to drive and power it, then I think a computer like that would feel like a superpower.
    - meekrohprocess 6y ago
      Hang on, is that actually true though? I was under the impression that shrinking process nodes represented a trade-off in terms of power consumption on the low end. There must be reasons why nobody makes a 14nm microcontroller, right? Shrinking the process node means that you get less power consumption per transistor flip, but it can also increase the amount of static leakage current, which hurts designs that aim for energy budgets under say, 100uA. I could be wrong, and I think that leakage can be mitigated by the lower operating voltages on smaller process nodes, but I don't believe it is as simple as "smaller process = more power efficient". If you're talking about GHz-scale application processors it holds true, but getting that sort of chip to idle at 0.05mA might be hard. RAM can also consume a lot of power, if you have gigabytes of it. So until we have cheap high-density NVRAM, you might need a sort of 'hibernate' mode to get really low power consumption. And if you did that, you'd need to burn a bunch of energy to wake up and go back to sleep...busy, busy, busy.
      - reilly3000 6y ago
        I really am out of my depth to speak to specifics of 3nm or how the hell physics even works at that scale. I can just assert that mass adoption of ubiquitous and tactile computing has dependency on speed, locality, and energy consumption. That said, I found this really interesting: https://www.pcgamesn.com/samsung-3nm-production-performance https://www.pcgamesn.com/samsung-3nm-production-performance They seem to have shipped a novel gate design rather than just shipping a smaller-yet FitFET. You're spot on that CPU power consumption is a just a part of the equation!
drfuzzyness 6y ago
I really wish it were easier for artists and small operations to get a hold of these. Ultraleap acquired LeapMotion a while back[1] and made it against their development TOS[2] to give public demonstrations which killed most of the energy I saw in the interactive arts space. A year ago, I looked into using Ultrahaptics for interactive exhibits and the yearly licensing they quoted was really high for making a single exhibit so my company ended up dropping that pitch. I understand they want to keep their tech close to their chest, prevent competitors, and focus on large distribution deals, but it feels like they're holding it back from the people who might make the most interesting examples. [1]: https://www.theverge.com/2019/5/30/18645604/leap-motion-vr-hand-tracking-ultrahaptics-acquisition-rumor https://www.theverge.com/2019/5/30/18645604/leap-motion-vr-h... [2]: https://www.ultraleap.com/licensing/#licensingforhaptics https://www.ultraleap.com/licensing/#licensingforhaptics
- PragmaticPulp 6y ago
  You can buy the hardware from retail distributors now. $2500: https://www.mouser.com/ProductDetail/Ultraleap/USX/?qs=wnTfsH77Xs6Oy1gMSijNgA%3D%3D https://www.mouser.com/ProductDetail/Ultraleap/USX/?qs=wnTfs...
ArtDev 6y ago
I would love to be able to manage windows and tabs with this. So sci fi!
- UShouldBWorking 6y ago
  I have one and it is not even close to useful. I had similar ideas but it is light years away from being as good as a keyboard shortcut. I imagined waving my hands around like tom cruise in minority report but it jumps around way too much and is horribly imprecise.
jtagen 6y ago
I have one of these sitting in my closet, the pre-acquisition development module I got off ebay for a few students to work with. The sensation is very strong, but the novelty wore off quickly.
- iamleppert 6y ago
  How much do you want for it?
  - jtagen 6y ago
    Think I paid $400, still in "like new" condition, everything works. Make an offer. jtagen at gmail.
modeless 6y ago
I'm curious if the people excited about this have tried it. The sensations are IMO not compelling. They definitely don't transform Minority Report-style air gestures into a viable mode of interaction. I think the cooler trick that ultrasonic phased arrays can perform is levitating 3D patterns of tiny styrofoam spheres. For those wanting to experiment with hand controlled UI, an Oculus Quest is probably the best entry point. Development is very accessible; you can access the hand tracking in the browser from JavaScript and go to town. And if you need more than hand tracking can provide, you get controllers too.
- spuz 6y ago
  Have you tried it?
  - modeless 6y ago
    Yes, I have owned multiple Leap Motion devices before their acquisition and used multiple demonstrations of ultrasonic phased arrays at SIGGRAPH and GDC, including Ultraleap specifically. It's true that my experience is somewhat limited as I haven't owned an ultrasonic phased array to play around with it for longer periods of time.
    - zaroth 6y ago
      I think this is not an experience which can be properly described as “somewhat limited”.
- rdw 6y ago
  I got the chance to try something like this a few years ago at GDC. It was barely even detectable. But I still hope that the technology gets researched and built out as far as it can go, because _something_ is necessary for virtual interfaces to work.
  - m463 6y ago
    I wonder if you get more sensitive if you spend time practicing. I've tried "reading" braille - feeling it - and it seems next to impossible to pick out the dots. But somehow blind people can read it, quickly.
cromwellian 6y ago
I feel like this is a dead end. Every year, haptic force-feedback gloves are getting better and better. I don't think tactile feedback without force feedback is the future.
- ipsum2 6y ago
  > haptic force-feedback gloves are getting better and better Any examples of commercial products that you think are compelling? Also, gloves are limiting - you need to put it on, figure out correct sizing, etc.
  - purple-again 6y ago
    This is my current favorite approach. https://www.google.com/amp/s/www.euronews.com/living/amp/2020/04/30/revolutionary-new-device-simulates-the-feeling-of-touch-in-virtual-reality https://www.google.com/amp/s/www.euronews.com/living/amp/202...
    - kroltan 6y ago
      Non-AMP link: https://www.euronews.com/living/2020/04/30/revolutionary-new-device-simulates-the-feeling-of-touch-in-virtual-reality https://www.euronews.com/living/2020/04/30/revolutionary-new...
HammadB 6y ago
The only mainstream UX for haptics I see making sense is direct neural input of some kind. Ultrasonic based approaches are cool, but probably limited to art installations, museums and the like
- deleted 6y ago
  [deleted]
deepnotderp 6y ago
IMHO phased arrays like this are too bulky and expensive, what you need for this to go mainstream is a Spatial Ultrasound Modulator like this https://www.nature.com/articles/s41467-020-18347-2 https://www.nature.com/articles/s41467-020-18347-2 There are also MEMS piston based devices that could also work
29athrowaway 6y ago
Does it make dogs insane?
runawaybottle 6y ago
I always thought the thing we’d need to simulate in a simulation is physical touch. I don’t know of any chemicals that can set off a pressure sensation. So if we are in a simulation, there’s got to be some kind of tech that creates physical touch sensation as we interact with the virtual world.
waiseristy 6y ago
So, what are the chances we'll see a ShowHN post about someone turning this into a sex toy?
codezero 6y ago
I got burned by Leap Mothion I won’t touch a thing they create. I think my most fundamental complaint was when Leap Motion launched instead of being prescriptive about how it should work they deferred all drivers to app specific functionality when they could have at least had a reference generic driver that had consistent behavior across apps that could then be modified. The device was a hog on system resources and worked only in super niche areas and I think I am mostly disappointed they didn’t go another route with a more generic input device. I’ve got no indication this device takes a different approach but tbh I don’t have the energy to do the work to check.
- SMFloris 6y ago
  I actually reverse engineered most of the code for the drivers for LeapMotion for my undergraduate project. Turns out it was illegal as per their TOS so my professor had me quit that project and take it offline. The biggest problem I had encountered back then is that there was a packet transmitted with a certain signature in order to turn on the cameras that I could not reverse for the life of me (had something to do with the app version). Other than that: basically two IR cameras that stream in an interleaved pixel format (2 x 640) x 480 iirc. It was a fun device to use and hack, but the first time I had ran into the brick-wall that is an enclosed format.
  - duncansw 6y ago
    Head of the Leap tracking team here - we actually unlocked the UVC drivers for our cameras a while back! https://github.com/leapmotion/leapuvc https://github.com/leapmotion/leapuvc > LeapUVC gives you access to the Leap Motion Controller image data through the industry standard UVC (Universal Video Class) interface. This gives you low level controls such as LED brightness, gamma, exposure, gain, resolution, and more.
    - SMFloris 6y ago
      Oh wow! That is great news! Great job, will dig out my leap motion once I get back and try it out!
- 6y ago
deleted 6y ago
[deleted]
Garlef 6y ago
I guess this could be enhanced to also serve as an input device. Combined with either microphones reacting to the reflections of the ultrasound or a high res theremin perhaps?
- MertsA 6y ago
  You'd be much better off with existing hand tracking or EMG systems. https://backyardbrains.com/experiments/RobotHand https://backyardbrains.com/experiments/RobotHand
nyanpasu64 6y ago
I tried putting my finger into a humidifier's ultrasound port once, exposing it to high levels of vibration. After a second or so, it caused a sharp pain in my finger. I wonder about the safety of this technology at various power levels, to skin as well as ears.
bambax 6y ago
I'm currently toying with making an "air" MIDI controller that uses ultrasonic distance sensors (HC SR04). Those sensors work around 40kHz, well beyond normal human hearing (human range is 20Hz to 20kHz and narrows with age) and I certainly don't hear anything. Yet my children, who are between 10-15, swear they can hear them (and I have at least verified they can tell by ear when they are on or off). I wonder how an array of 256 of these would sound to them? (From the product image: 16x16=256).
- deleted 6y ago
  [deleted]
- codezero 6y ago
  I suspect they could be hearing echos that fall into a lower frequency outside of your hearing band.
  - bambax 6y ago
    Yes it's impossible they are hearing anything at 40kHz. Either the generated sounds are not really 40kHz, or there is something vibrating or resonating that produces something they can hear. But the question about running 256 of those simultaneously remains.
    - codezero 6y ago
      Absolutely!
MildlySerious 6y ago
I'm still disappointed in the Leap Motion, although mostly because of the drivers. They are so locked down that it's basically impossible to play with the hardware without going through their SDK, which, last time I checked, did not work with ARM devices and I never had any luck getting it to work on Ubuntu.
rad_gruchalski 6y ago
I am looking forward to the day when car manufacturers shove this stuff into cars to introduce tactile feedback for touch screens to solve the urging problem of getting rid of physical controls.

155 points by leontrolski 6y ago

Here's a link to a recently published paper - "Using Spatiotemporal Modulation to Draw Tactile Patterns in Mid-Air" - https://link.springer.com/chapter/10.1007/978-3-319-93445-7_24 https://link.springer.com/chapter/10.1007/978-3-319-93445-7_...

frfl 6y ago

What affect does this have on dogs (if any)?

Blammar 6y ago
I'm thinking the same thing that happened with the Navy sonar and the beached beaked whales... (see https://royalsocietypublishing.org/doi/pdf/10.1098/rspb.2018.2533 https://royalsocietypublishing.org/doi/pdf/10.1098/rspb.2018...)

deleted 6y ago

[deleted]

reilly3000 6y ago

Man I've been harping on haptic holograms being the next UI mode since 2014, and this takes us half-way there. Congrats for getting some products to market in this space! I feel like there are a couple more years of VR stagnation, but by 2023-5 when 3mn SOCs are pretty common and affordable I think it will all hit at once and haptics will be a huge mode of interaction.

mywacaday 6y ago
Why does 3nm make a big difference?
- reilly3000 6y ago
  Incredibly low power draw in the low end, which allows for more mobility and diverse sources of energy. Like if you could wear meaningful compute that is powered with body kinetic energy or some flexible solar thing, I think that unlocks computing to happen in many places with many physical and virtual things working in concert. ESP-32 on BLE sleep can sip tiny amounts of energy, but cannot do any meaningful compute, and a low top end speed. I want a chip that can burst to 5GHZ and rest at 0.05mha. That can make generating or streaming a massive bitrate from a wearable. Then you can do sick-ass offline-first edge compute and be able to drive one of these haptics devices, projectors, VR, whatever with a tiny machine or local cluster. Wearable k3s if you will. If you could drive any display from your person, with local and remote indexes and objects of everything you want or need, interfaces to people and things, replay of everything, totally offline as you want or need. If it was fast enough, and didn’t require a whole backpack to drive and power it, then I think a computer like that would feel like a superpower.
  - meekrohprocess 6y ago
    Hang on, is that actually true though? I was under the impression that shrinking process nodes represented a trade-off in terms of power consumption on the low end. There must be reasons why nobody makes a 14nm microcontroller, right? Shrinking the process node means that you get less power consumption per transistor flip, but it can also increase the amount of static leakage current, which hurts designs that aim for energy budgets under say, 100uA. I could be wrong, and I think that leakage can be mitigated by the lower operating voltages on smaller process nodes, but I don't believe it is as simple as "smaller process = more power efficient". If you're talking about GHz-scale application processors it holds true, but getting that sort of chip to idle at 0.05mA might be hard. RAM can also consume a lot of power, if you have gigabytes of it. So until we have cheap high-density NVRAM, you might need a sort of 'hibernate' mode to get really low power consumption. And if you did that, you'd need to burn a bunch of energy to wake up and go back to sleep...busy, busy, busy.
    - reilly3000 6y ago
      I really am out of my depth to speak to specifics of 3nm or how the hell physics even works at that scale. I can just assert that mass adoption of ubiquitous and tactile computing has dependency on speed, locality, and energy consumption. That said, I found this really interesting: https://www.pcgamesn.com/samsung-3nm-production-performance https://www.pcgamesn.com/samsung-3nm-production-performance They seem to have shipped a novel gate design rather than just shipping a smaller-yet FitFET. You're spot on that CPU power consumption is a just a part of the equation!

drfuzzyness 6y ago

I really wish it were easier for artists and small operations to get a hold of these. Ultraleap acquired LeapMotion a while back[1] and made it against their development TOS[2] to give public demonstrations which killed most of the energy I saw in the interactive arts space. A year ago, I looked into using Ultrahaptics for interactive exhibits and the yearly licensing they quoted was really high for making a single exhibit so my company ended up dropping that pitch. I understand they want to keep their tech close to their chest, prevent competitors, and focus on large distribution deals, but it feels like they're holding it back from the people who might make the most interesting examples. [1]: https://www.theverge.com/2019/5/30/18645604/leap-motion-vr-hand-tracking-ultrahaptics-acquisition-rumor https://www.theverge.com/2019/5/30/18645604/leap-motion-vr-h... [2]: https://www.ultraleap.com/licensing/#licensingforhaptics https://www.ultraleap.com/licensing/#licensingforhaptics

PragmaticPulp 6y ago
You can buy the hardware from retail distributors now. $2500: https://www.mouser.com/ProductDetail/Ultraleap/USX/?qs=wnTfsH77Xs6Oy1gMSijNgA%3D%3D https://www.mouser.com/ProductDetail/Ultraleap/USX/?qs=wnTfs...

ArtDev 6y ago

I would love to be able to manage windows and tabs with this. So sci fi!

UShouldBWorking 6y ago
I have one and it is not even close to useful. I had similar ideas but it is light years away from being as good as a keyboard shortcut. I imagined waving my hands around like tom cruise in minority report but it jumps around way too much and is horribly imprecise.

jtagen 6y ago

I have one of these sitting in my closet, the pre-acquisition development module I got off ebay for a few students to work with. The sensation is very strong, but the novelty wore off quickly.

iamleppert 6y ago
How much do you want for it?
- jtagen 6y ago
  Think I paid $400, still in "like new" condition, everything works. Make an offer. jtagen at gmail.

modeless 6y ago

I'm curious if the people excited about this have tried it. The sensations are IMO not compelling. They definitely don't transform Minority Report-style air gestures into a viable mode of interaction. I think the cooler trick that ultrasonic phased arrays can perform is levitating 3D patterns of tiny styrofoam spheres. For those wanting to experiment with hand controlled UI, an Oculus Quest is probably the best entry point. Development is very accessible; you can access the hand tracking in the browser from JavaScript and go to town. And if you need more than hand tracking can provide, you get controllers too.

spuz 6y ago
Have you tried it?
- modeless 6y ago
  Yes, I have owned multiple Leap Motion devices before their acquisition and used multiple demonstrations of ultrasonic phased arrays at SIGGRAPH and GDC, including Ultraleap specifically. It's true that my experience is somewhat limited as I haven't owned an ultrasonic phased array to play around with it for longer periods of time.
  - zaroth 6y ago
    I think this is not an experience which can be properly described as “somewhat limited”.
rdw 6y ago
I got the chance to try something like this a few years ago at GDC. It was barely even detectable. But I still hope that the technology gets researched and built out as far as it can go, because _something_ is necessary for virtual interfaces to work.
- m463 6y ago
  I wonder if you get more sensitive if you spend time practicing. I've tried "reading" braille - feeling it - and it seems next to impossible to pick out the dots. But somehow blind people can read it, quickly.

cromwellian 6y ago

I feel like this is a dead end. Every year, haptic force-feedback gloves are getting better and better. I don't think tactile feedback without force feedback is the future.

ipsum2 6y ago
> haptic force-feedback gloves are getting better and better Any examples of commercial products that you think are compelling? Also, gloves are limiting - you need to put it on, figure out correct sizing, etc.
- purple-again 6y ago
  This is my current favorite approach. https://www.google.com/amp/s/www.euronews.com/living/amp/2020/04/30/revolutionary-new-device-simulates-the-feeling-of-touch-in-virtual-reality https://www.google.com/amp/s/www.euronews.com/living/amp/202...
  - kroltan 6y ago
    Non-AMP link: https://www.euronews.com/living/2020/04/30/revolutionary-new-device-simulates-the-feeling-of-touch-in-virtual-reality https://www.euronews.com/living/2020/04/30/revolutionary-new...

HammadB 6y ago

The only mainstream UX for haptics I see making sense is direct neural input of some kind. Ultrasonic based approaches are cool, but probably limited to art installations, museums and the like

deleted 6y ago
[deleted]

deepnotderp 6y ago

IMHO phased arrays like this are too bulky and expensive, what you need for this to go mainstream is a Spatial Ultrasound Modulator like this https://www.nature.com/articles/s41467-020-18347-2 https://www.nature.com/articles/s41467-020-18347-2 There are also MEMS piston based devices that could also work

29athrowaway 6y ago

Does it make dogs insane?

runawaybottle 6y ago

I always thought the thing we’d need to simulate in a simulation is physical touch. I don’t know of any chemicals that can set off a pressure sensation. So if we are in a simulation, there’s got to be some kind of tech that creates physical touch sensation as we interact with the virtual world.

waiseristy 6y ago

So, what are the chances we'll see a ShowHN post about someone turning this into a sex toy?

codezero 6y ago

I got burned by Leap Mothion I won’t touch a thing they create. I think my most fundamental complaint was when Leap Motion launched instead of being prescriptive about how it should work they deferred all drivers to app specific functionality when they could have at least had a reference generic driver that had consistent behavior across apps that could then be modified. The device was a hog on system resources and worked only in super niche areas and I think I am mostly disappointed they didn’t go another route with a more generic input device. I’ve got no indication this device takes a different approach but tbh I don’t have the energy to do the work to check.

SMFloris 6y ago
I actually reverse engineered most of the code for the drivers for LeapMotion for my undergraduate project. Turns out it was illegal as per their TOS so my professor had me quit that project and take it offline. The biggest problem I had encountered back then is that there was a packet transmitted with a certain signature in order to turn on the cameras that I could not reverse for the life of me (had something to do with the app version). Other than that: basically two IR cameras that stream in an interleaved pixel format (2 x 640) x 480 iirc. It was a fun device to use and hack, but the first time I had ran into the brick-wall that is an enclosed format.
- duncansw 6y ago
  Head of the Leap tracking team here - we actually unlocked the UVC drivers for our cameras a while back! https://github.com/leapmotion/leapuvc https://github.com/leapmotion/leapuvc > LeapUVC gives you access to the Leap Motion Controller image data through the industry standard UVC (Universal Video Class) interface. This gives you low level controls such as LED brightness, gamma, exposure, gain, resolution, and more.
  - SMFloris 6y ago
    Oh wow! That is great news! Great job, will dig out my leap motion once I get back and try it out!
6y ago

deleted 6y ago

[deleted]

Garlef 6y ago

I guess this could be enhanced to also serve as an input device. Combined with either microphones reacting to the reflections of the ultrasound or a high res theremin perhaps?

MertsA 6y ago
You'd be much better off with existing hand tracking or EMG systems. https://backyardbrains.com/experiments/RobotHand https://backyardbrains.com/experiments/RobotHand

nyanpasu64 6y ago

I tried putting my finger into a humidifier's ultrasound port once, exposing it to high levels of vibration. After a second or so, it caused a sharp pain in my finger. I wonder about the safety of this technology at various power levels, to skin as well as ears.

bambax 6y ago

I'm currently toying with making an "air" MIDI controller that uses ultrasonic distance sensors (HC SR04). Those sensors work around 40kHz, well beyond normal human hearing (human range is 20Hz to 20kHz and narrows with age) and I certainly don't hear anything. Yet my children, who are between 10-15, swear they can hear them (and I have at least verified they can tell by ear when they are on or off). I wonder how an array of 256 of these would sound to them? (From the product image: 16x16=256).

deleted 6y ago
[deleted]
codezero 6y ago
I suspect they could be hearing echos that fall into a lower frequency outside of your hearing band.
- bambax 6y ago
  Yes it's impossible they are hearing anything at 40kHz. Either the generated sounds are not really 40kHz, or there is something vibrating or resonating that produces something they can hear. But the question about running 256 of those simultaneously remains.
  - codezero 6y ago
    Absolutely!

MildlySerious 6y ago

I'm still disappointed in the Leap Motion, although mostly because of the drivers. They are so locked down that it's basically impossible to play with the hardware without going through their SDK, which, last time I checked, did not work with ARM devices and I never had any luck getting it to work on Ubuntu.

rad_gruchalski 6y ago

I am looking forward to the day when car manufacturers shove this stuff into cars to introduce tactile feedback for touch screens to solve the urging problem of getting rid of physical controls.