4 ms·

Step 1: Make an amazing game with gestures Step 2: ??? Step 3: Profit! Joke aside, such game mechanics, albeit with cursor gesturing rather than hand gesturi

by GuiA 6y ago

Step 1: Make an amazing game with gestures Step 2: ??? Step 3: Profit! Joke aside, such game mechanics, albeit with cursor gesturing rather than hand gesturing, have been tried many times before. See Arx Fatalis, Black & White, etc. Jonathan Blow has a pre-Braid prototype about that as well that he shows in some detail in a talk he gave. The reality is that sure, it sounds cool when you describe it like that, but there are many design issues to contend with: - how does the player remember all these obscure symbols ? - how do you deal with the frustration when a symbol isn't recognized? When it's confused by the system with another symbol? Etc. On top of that, there is the fact that drawing squiggles to cast spells is not a game mechanic that is inherently extremely fun. Why are people going to the bother of doing that vs pressing a button? It's all a lot of work to make it make sense, and there is little reason to believe that gesturing with your hands instead of your mouse would solve some of the more fundamental problems.

keenmaster 6y ago
The limitations you describe are rooted in a pre-VR world. VR is much more conducive to gestures. In Half Life Alyx, Valve could have technically allowed you to lob a grenade by just pressing a button "because it's easier." But they didn't, and opted for a gesture-based input. They could have allowed you to turn a wheel by mashing a button, but they didn't, and you actually have to turn the wheel in circles. These decisions would have been insane with a keyboard and mouse as inputs. They're natural for a VR game. Without a keyboard, it's cumbersome to switch between a bunch of spells. Gestures are both more immersive and functionally better. Maybe most people wouldn't memorize hundreds of spells, but a couple dozen are certainly feasible. There could be a spell "language," where the initial part of the gesture could correspond to the category of spell. Fireball, fire wave, etc...can start with the same basic gesture. All wave spells can end with the same gesture. Hundreds of spells can be constructed from permutations of 20-30 foundational gestures, kind of like how a core group of trilateral roots in Arabic and Hebrew can construct a much larger set of words (sometimes without previous knowledge of a specific word/permutation) https://en.wikipedia.org/wiki/Semitic_root https://en.wikipedia.org/wiki/Semitic_root
- GuiA 6y ago
  The examples you mention work well because there is a real world affordance (the wheel, the act of throwing) that the gesture maps to. I agree that rotating a wheel gesturally makes more sense as an interaction in VR than the same action does with a mouse pointer. The grenade example is a good one, because throwing/flicking doesn't require much feedback, unlike say rotation or fine grained manipulation. Of course the interaction would feel nicer if you could feel the weight of what you're throwing - the physics of throwing objects always feels kind of off in VR games - but it still works well enough. Note that many games did this with the Wiimote, and there is nothing about that interaction that is singular to VR. However, the moment you lose that affordance based in a real world object, and ask your user/player to make gestures in mid air based on recollection, things really do start to fall apart. There is a host of HCI literature/projects around this - e.g. check out papers/demos from the Kinect era. An RPG game like the one you describe could be made with Kinect, and I vaguely recall a few in development - but Kinect demonstrated well that the only kind of gestural interaction that works in long term sessions is dancing. Btw, this is why if you go to e.g. the VR center in Tokyo, almost every booth has physical props for you to hold/grip/sit on while you're fishing/golfing/shooting a gun/riding a bike in VR.
  - keenmaster 6y ago
    I suspect that gesture input adoption, categorically, has emergent properties. Many different things have to "go right" for the input category to take off in a way that is hard to grok. The use of gestures has to appeal to hardcore gamers (in this case). The motion tracking needs to be sufficiently advanced. There can't be a swifter, dopamine enhancing alternative (like mashing buttons on a keyboard). A critical mass of developers need to make games with gestures as a core mechanic. Etc... I offer Pavlov VR as an example. Pavlov is like Counterstrike, except many of the mechanics are gesture based. - You grab your gun off your back - You actually need to look through the scope for zoom, physically bringing your head and hands into alignment - You have to reach over your left shoulder to activate the walkie-talkie for remote communication - Reloading is a multi-step process and varies from gun to gun You might say that I'm proving your point about gestures mapping to real-life, but that's not why I bring up Pavlov. I'm using it to illustrate the aforementioned emergent properties, because all of those gestures add up to something unexpected: even with subpar graphics and a low budget, it is fun. Not only that, but it is fun in an unexpected, novel way for FPS games. Suddenly, your kill count, headshots, etc...don't matter as much. The immersion as well as the social element of the game are huge. You are so zoomed in, so in the moment, when you are in a firefight, poking out your head out to snipe the sniper before rushing back below the barrier, in a way that you wouldn't be in a 2D game. Microsoft could technically have made a Pavlov Kinect, but who would play it? Almost no one. People would complain that traditional controls make much more sense, and they would have a point. It's simply awkward to squirm around in front of a 2D screen. There's a disconnect between the input modality (3D gestures) and the display modality (2D screen) when using the Kinect. In VR, both the input and the display are in 3D harmony.
Swizec 6y ago
> drawing squiggles to cast spells is not a game mechanic that is inherently extremely fun Drawing squiggles isn't inherently fun. But dancing your spells now that's hecking fun. Human culture already uses lots of weird hand symbols to cast all sorts of spells. You can say "hello" with a hand gesture. You can cast teh "I'm sorry spell". There's a great "I need your attention" spell, another for "I can't understand/hear" ... and let's not forget the wonderful repertoire of spells for "You're an idiot" Hand and body gestures are a fantastic user interface. Computer recognition might not be quite sophisticated enough yet.
nico_h 6y ago
I think a starting point would be Avatar , the cartoon. The spells were based on mostly full body martial arts, but each element had a distinct flavor.
- GuiA 6y ago
  That’s actually a very neat line of thinking! If you could model it in terms of gait (slow and heavy vs light and limber) rather than specific gestures, it could make for a compelling system.
nico_h 6y ago
I raise Beat Saber as a counter example: it would be less tiring to press buttons than to flail your arms and wrists, but somehow it would not be as fun.
- GuiA 6y ago
  Beat Saber is a great example - it asks you to do something tiring, but it’s super fun and you have immediate feedback so the effort is worth it. Games can ask players to do almost anything as long as they’re having fun in return - but that means that the more outlandish things you ask them to do, the more fun they must have.
  - keenmaster 6y ago
    >as long as they’re having fun in return This is true, but not in a a way that maps directly to the gesture. For example, reloading in Pavlov is not fun in isolation. It’s a relatively tedious process that mirrors reloading in real life. It is rewarding to successfully reload while you’re taking fire and then proceed to kill your enemy. As a result, reloading gesture > reloading button, even though you need to memorize a different reloading action for each gun (it becomes muscle memory).

by GuiA 6y ago

keenmaster 6y ago

The limitations you describe are rooted in a pre-VR world. VR is much more conducive to gestures. In Half Life Alyx, Valve could have technically allowed you to lob a grenade by just pressing a button "because it's easier." But they didn't, and opted for a gesture-based input. They could have allowed you to turn a wheel by mashing a button, but they didn't, and you actually have to turn the wheel in circles. These decisions would have been insane with a keyboard and mouse as inputs. They're natural for a VR game. Without a keyboard, it's cumbersome to switch between a bunch of spells. Gestures are both more immersive and functionally better. Maybe most people wouldn't memorize hundreds of spells, but a couple dozen are certainly feasible. There could be a spell "language," where the initial part of the gesture could correspond to the category of spell. Fireball, fire wave, etc...can start with the same basic gesture. All wave spells can end with the same gesture. Hundreds of spells can be constructed from permutations of 20-30 foundational gestures, kind of like how a core group of trilateral roots in Arabic and Hebrew can construct a much larger set of words (sometimes without previous knowledge of a specific word/permutation) https://en.wikipedia.org/wiki/Semitic_root https://en.wikipedia.org/wiki/Semitic_root

GuiA 6y ago
The examples you mention work well because there is a real world affordance (the wheel, the act of throwing) that the gesture maps to. I agree that rotating a wheel gesturally makes more sense as an interaction in VR than the same action does with a mouse pointer. The grenade example is a good one, because throwing/flicking doesn't require much feedback, unlike say rotation or fine grained manipulation. Of course the interaction would feel nicer if you could feel the weight of what you're throwing - the physics of throwing objects always feels kind of off in VR games - but it still works well enough. Note that many games did this with the Wiimote, and there is nothing about that interaction that is singular to VR. However, the moment you lose that affordance based in a real world object, and ask your user/player to make gestures in mid air based on recollection, things really do start to fall apart. There is a host of HCI literature/projects around this - e.g. check out papers/demos from the Kinect era. An RPG game like the one you describe could be made with Kinect, and I vaguely recall a few in development - but Kinect demonstrated well that the only kind of gestural interaction that works in long term sessions is dancing. Btw, this is why if you go to e.g. the VR center in Tokyo, almost every booth has physical props for you to hold/grip/sit on while you're fishing/golfing/shooting a gun/riding a bike in VR.
- keenmaster 6y ago
  I suspect that gesture input adoption, categorically, has emergent properties. Many different things have to "go right" for the input category to take off in a way that is hard to grok. The use of gestures has to appeal to hardcore gamers (in this case). The motion tracking needs to be sufficiently advanced. There can't be a swifter, dopamine enhancing alternative (like mashing buttons on a keyboard). A critical mass of developers need to make games with gestures as a core mechanic. Etc... I offer Pavlov VR as an example. Pavlov is like Counterstrike, except many of the mechanics are gesture based. - You grab your gun off your back - You actually need to look through the scope for zoom, physically bringing your head and hands into alignment - You have to reach over your left shoulder to activate the walkie-talkie for remote communication - Reloading is a multi-step process and varies from gun to gun You might say that I'm proving your point about gestures mapping to real-life, but that's not why I bring up Pavlov. I'm using it to illustrate the aforementioned emergent properties, because all of those gestures add up to something unexpected: even with subpar graphics and a low budget, it is fun. Not only that, but it is fun in an unexpected, novel way for FPS games. Suddenly, your kill count, headshots, etc...don't matter as much. The immersion as well as the social element of the game are huge. You are so zoomed in, so in the moment, when you are in a firefight, poking out your head out to snipe the sniper before rushing back below the barrier, in a way that you wouldn't be in a 2D game. Microsoft could technically have made a Pavlov Kinect, but who would play it? Almost no one. People would complain that traditional controls make much more sense, and they would have a point. It's simply awkward to squirm around in front of a 2D screen. There's a disconnect between the input modality (3D gestures) and the display modality (2D screen) when using the Kinect. In VR, both the input and the display are in 3D harmony.

Swizec 6y ago

> drawing squiggles to cast spells is not a game mechanic that is inherently extremely fun Drawing squiggles isn't inherently fun. But dancing your spells now that's hecking fun. Human culture already uses lots of weird hand symbols to cast all sorts of spells. You can say "hello" with a hand gesture. You can cast teh "I'm sorry spell". There's a great "I need your attention" spell, another for "I can't understand/hear" ... and let's not forget the wonderful repertoire of spells for "You're an idiot" Hand and body gestures are a fantastic user interface. Computer recognition might not be quite sophisticated enough yet.

nico_h 6y ago

I think a starting point would be Avatar , the cartoon. The spells were based on mostly full body martial arts, but each element had a distinct flavor.

GuiA 6y ago
That’s actually a very neat line of thinking! If you could model it in terms of gait (slow and heavy vs light and limber) rather than specific gestures, it could make for a compelling system.

nico_h 6y ago

I raise Beat Saber as a counter example: it would be less tiring to press buttons than to flail your arms and wrists, but somehow it would not be as fun.

GuiA 6y ago
Beat Saber is a great example - it asks you to do something tiring, but it’s super fun and you have immediate feedback so the effort is worth it. Games can ask players to do almost anything as long as they’re having fun in return - but that means that the more outlandish things you ask them to do, the more fun they must have.
- keenmaster 6y ago
  >as long as they’re having fun in return This is true, but not in a a way that maps directly to the gesture. For example, reloading in Pavlov is not fun in isolation. It’s a relatively tedious process that mirrors reloading in real life. It is rewarding to successfully reload while you’re taking fire and then proceed to kill your enemy. As a result, reloading gesture > reloading button, even though you need to memorize a different reloading action for each gun (it becomes muscle memory).