Gesture Mapper: Hands

Step 1: Enable hand tracking

Step 2: Record Gestures

Position your left hand in the shape you’d like to map, then press Record Landmarks. As it’s recording, move your hand around a little to capture minor variations.

At the moment, this will map the hand as the webcam sees it: so pointing to the left 👈 would be considered different from pointing right 👉. If you would like both to be considered, then check the horizontal mirror button in Step 3.

 

Confidence:

Step 3: Configure

Gesture Name (no spaces):


Emphasize Fingers

Confidence:

Mirroring

Step 4: Copy the gesture description below

Step 5: Using the gesture

The Gesture Description above uses the handsfree.useGesture() method to load your gesture into your app. Like the handsfree.use(), which adds a .enable() and .disable() to your callbacks, gestures loaded with .useGesture() also receive a .enable() and .disable() method. Like plugins, gestures are also accessed by name: handsfree.gesture[gestureName].disable()

Checking outside of plugins

You can check which gesture is currently active with: handsfree.model.hands.getGesture(). This will return an array with a gesture object for each hand, that looks something like:

[
  // Left hand
  {name: 'myGesture', confidence: 10, pose: []},

  // Right hand
  {name: '', confidence: 0, pose: []},
]

Only gestures that have reached their required confidence will be active. If a gesture has a required confidence of 7.5 (default), but it got a 6, then no gesture will be active even if it’s the only one. Here’s an example of how you might react to a gesture:

const gesture = handsfree.model.hands.getGesture()
console.log(gesture)

// Check the left hand
if (gesture[0]?.name === 'myGesture') {
  // Do things with the gesture
}

// Check the right hand
if (gesture[1]?.name === 'myGesture') {
  // Do things with the gesture
}

Checking inside of plugins

As a convenience, the gesture array is added to the .gesture property of the hand model inside of plugins:

/**
 * Check from inside a plugin
 */
handsfree.use('myPlugin', ({hands}) => {
  if (!hands) return
  
  console.log(hands.gesture)
  if (hands.gesture[0]?.name === 'myGesture') {
    // Do things with the gesture
  }
})

 

Examples & Use Cases

Menu