Plugin: pinchers

Try it!

Pinch your fingers to see the different circles light up.

Hand Index [0] Middle [1] Ring [2] Pinky [3]
Left
Right

A collection of events, properties, and helper styles for finger pinching:

  • 👌 Pinch your thumb with any finger to set that fingers “click” state
  • Unpinched fingers are black, pinched fingers are red

Properties

Pinch States with .pinchState

This plugin adds handsfree.data.hands.pinchState to the Hands Model. It is a 2D array with the following:

handsfree.data.hands.pinchState = [
  // Left hand
  // index, middle, ring, pinky
  ['', '', '', ''],
  // Right hand
  // index, middle, ring, pinky
  ['', '', '', '']
]

Each index can be of one of the following states:

State Note
start When the pinch first starts
held Every frame the pinch is held
released When the pinch is released
const handsfree = new Handsfree({hands: true})
handsfree.use('logger', ({hands}) => {
  console.log(hands.pinchState)
})

Original Pinch Locations with .origPos

In addition the the .pinchState, you also have access to the original pixel {x, y} that the pinch occurred within the webcam through .origPinch. This is very useful for determining how far a pinch was “dragged”. Like with .pinchStatehandsfree.data.hands.origPinch contains one value per finger per hand:

// Log the original point of pinch
handsfree.on('finger-pinched-0-1', () => {
  // Display the x and y of the left pointer finger
  console.log(
    handsfree.data.hands.origPinch[0][0].x,
    handsfree.data.hands.origPinch[0][0].y
  )
})

Current Pinch Locations with .curPinch

Like .origPinch.curPinch lists the current pixel {x, y} that the pinch is happening at. This is useful for calculating the distance since the .origPinch:

// Log the original point of pinch
handsfree.on('finger-pinched-1-3', () => {
  // Display the x and y of the right pinky
  console.log(
    handsfree.data.hands.curPinch[1][3].x,
    handsfree.data.hands.curPinch[1][3].y
  )
})

Events

Currently this plugin emits an event for every individual finger, which you can listen to. There are a total of 8 possible events, where h represents the hand (0 = left, 1 = right) and f represents the index:

/* SPECIFIC HAND */
/* Any event */
handsfree-finger-pinched-h-f
/* start event */
handsfree-finger-pinched-start-h-f
/* held event */
handsfree-finger-pinched-held-h-f
/* released event */
handsfree-finger-pinched-released-h-f

/* ANY HAND */
/* Any event */
handsfree-finger-pinched-f
/* start event */
handsfree-finger-pinched-start-f
/* held event */
handsfree-finger-pinched-held-f
/* released event */
handsfree-finger-pinched-released-f

Here are a few examples for listening to these events:

// ## SPECIFIC HAND
// Listen to any event from left hand (0), index finger (0)
document.addEventListener('handsfree-finger-pinched-0-0')
// Listen to any event right hand (1), pinky finger (3)
handsfree.on('finger-pinched-1-3')

// Listen to a specific event from left hand (0), middle finger (1)
handsfree.on('finger-pinched-start-0-1')
// Listen to a specific event from right hand (1), ring finger (2)
document.addEventListener('handsfree-finger-pinched-1-2')

// ## ANY HAND
// Listen to any event from with any index finger (0)
document.addEventListener('handsfree-finger-pinched-0')
// Listen to any event with any pinky finger (3)
handsfree.on('finger-pinched-3')

// Listen to a specific event with any middle finger (1)
handsfree.on('finger-pinched-start-1')
// Listen to a specific event with any ring finger (2)
document.addEventListener('handsfree-finger-pinched-2')

Classes

This plugin comes with many helper classes to help you style your app based on the pinching fingers, they look like this:

/* ## SPECIFIC HAND */
/* Left hand (0), index finger (0) */
.handsfree-show-when-finger-pinched-0-0
.handsfree-hide-when-finger-pinched-0-0

/* Right hand (1), pinky finger (3) */
.handsfree-show-when-finger-pinched-1-3
.handsfree-hide-when-finger-pinched-1-3

/* ## ANY HAND */
/* Any index finger (0) */
.handsfree-show-when-finger-pinched-0
.handsfree-hide-when-finger-pinched-0

/* Any pinky finger (3) */
.handsfree-show-when-finger-pinched-3
.handsfree-hide-when-finger-pinched-3

Simply apply these classes to the elements you’d like to show/hide. If you’d like some more styling then you can take advantage of the body classes that get added:

/* Left (0) middle finger (1) */
body.handsfree-finger-pinched-0-1

/* Right (1) ring finger (2) */
body.handsfree-finger-pinched-1-2

/* ANY middle finger (1) */
body.handsfree-finger-pinched-1

/* ANY ring finger (2) */
body.handsfree-finger-pinched-2

Plugin Code

How to include into your project

This plugin is automatically included with Handsfree.js, so all you need to do is either enable it directly with handsfree.plugin.pinchers.enable() or enable it through it’s tag: handsfree.enablePlugins('browser')

handsfree.use('pinchers', {
  models: 'hands',
  enabled: true,
  tags: ['core'],

  // Index of fingertips
  fingertipIndex: [8, 12, 16, 20],

  // Number of frames the current element is the same as the last
  // [left, right]
  // [index, middle, ring, pinky]
  numFramesFocused: [[0, 0, 0, 0,], [0, 0, 0, 0]],

  // Whether the fingers are touching
  thresholdMet: [[0, 0, 0, 0,], [0, 0, 0, 0]],
  framesSinceLastGrab: [[0, 0, 0, 0,], [0, 0, 0, 0]],

  // The original grab point for each finger
  origPinch: [
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}],
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}]
  ],
  curPinch: [
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}],
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}]
  ],

  // Just downel
  pinchDowned: [
    [0, 0, 0, 0],
    [0, 0, 0, 0]
  ],
  pinchDown: [
    [false, false, false, false],
    [false, false, false, false]
  ],
  pinchUp: [
    [false, false, false, false],
    [false, false, false, false]
  ],

  // The tweened scrollTop, used to smoothen out scroll
  // [[leftHand], [rightHand]]
  tween: [
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}],
    [{x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}, {x: 0, y: 0}]
  ],

  // Number of frames that has passed since the last grab
  numFramesFocused: [[0, 0, 0, 0,], [0, 0, 0, 0]],

  // Number of frames mouse has been downed
  mouseDowned: 0,
  // Is the mouse up?
  mouseUp: false,
  // Whether one of the morph confidences have been met
  mouseThresholdMet: false,

  config: {
    // Number of frames over the same element before activating that element
    framesToFocus: 10,

    // Number of pixels the middle and thumb tips must be near each other to drag
    threshold: 50,

    // Number of frames where a hold is not registered before releasing a drag
    numThresholdErrorFrames: 5,

    maxMouseDownedFrames: 1
  },

  onUse () {
    this.$target = window
  },

  /**
   * Scroll the page when the cursor goes above/below the threshold
   */
  onFrame ({hands}) {
    if (!hands.multiHandLandmarks) return

    const height = this.handsfree.debug.$canvas.hands.height
    const leftVisible = hands.multiHandedness.some(hand => hand.label === 'Right')
    const rightVisible = hands.multiHandedness.some(hand => hand.label === 'Left')
    
    // Detect if the threshold for clicking is met with specific morphs
    for (let n = 0; n < hands.multiHandLandmarks.length; n++) {
      // Set the hand index
      let hand = hands.multiHandedness[n].label === 'Right' ? 0 : 1
      
      for (let finger = 0; finger < 4; finger++) {
        // Check if fingers are touching
        const a = hands.multiHandLandmarks[n][4].x - hands.multiHandLandmarks[n][this.fingertipIndex[finger]].x
        const b = hands.multiHandLandmarks[n][4].y - hands.multiHandLandmarks[n][this.fingertipIndex[finger]].y
        const c = Math.sqrt(a*a + b*b) * height
        const thresholdMet = this.thresholdMet[hand][finger] = c < this.config.threshold

        if (thresholdMet) {
          // Set the current pinch
          this.curPinch[hand][finger] = hands.multiHandLandmarks[n][4]
          
          // Store the original pinch
          if (this.framesSinceLastGrab[hand][finger] > this.config.numThresholdErrorFrames) {
            this.origPinch[hand][finger] = hands.multiHandLandmarks[n][4]
            this.handsfree.TweenMax.killTweensOf(this.tween[hand][finger])
          }
          this.framesSinceLastGrab[hand][finger] = 0
        }
        ++this.framesSinceLastGrab[hand][finger]
      }
    }

    // Update the hands object
    hands.origPinch = this.origPinch
    hands.curPinch = this.curPinch
    this.handsfree.data.hands = this.getPinchStates(hands, leftVisible, rightVisible)
  },

  /**
   * Check if we are "mouse clicking"
   */
  getPinchStates (hands, leftVisible, rightVisible) {
    const visible = [leftVisible, rightVisible]

    // Make sure states are available
    hands.pinchState = [
      ['', '', '', ''],
      ['', '', '', '']
    ]
    
    // Loop through every hand and finger
    for (let hand = 0; hand < 2; hand++) {
      for (let finger = 0; finger < 4; finger++) {
        // Click
        if (visible[hand] && this.thresholdMet[hand][finger]) {
          this.pinchDowned[hand][finger]++
          document.body.classList.add(`handsfree-finger-pinched-${hand}-${finger}`, `handsfree-finger-pinched-${finger}`)
        } else {
          this.pinchUp[hand][finger] = this.pinchDowned[hand][finger]
          this.pinchDowned[hand][finger] = 0
          document.body.classList.remove(`handsfree-finger-pinched-${hand}-${finger}`, `handsfree-finger-pinched-${finger}`)
        }
        
        // Set the state
        if (this.pinchDowned[hand][finger] > 0 && this.pinchDowned[hand][finger] <= this.config.maxMouseDownedFrames) {
          hands.pinchState[hand][finger] = 'start'
        } else if (this.pinchDowned[hand][finger] > this.config.maxMouseDownedFrames) {
          hands.pinchState[hand][finger] = 'held'
        } else if (this.pinchUp[hand][finger]) {
          hands.pinchState[hand][finger] = 'released'
        } else {
          hands.pinchState[hand][finger] = ''
        }

        // Emit an event
        if (hands.pinchState[hand][finger]) {
          // Specific hand
          this.handsfree.emit(`finger-pinched-${hand}-${finger}`, {
            event: hands.pinchState[hand][finger],
            origPinch: hands.origPinch[hand][finger],
            curPinch: hands.curPinch[hand][finger]
          })
          this.handsfree.emit(`finger-pinched-${hands.pinchState[hand][finger]}-${hand}-${finger}`, {
            event: hands.pinchState[hand][finger],
            origPinch: hands.origPinch[hand][finger],
            curPinch: hands.curPinch[hand][finger]
          })
          // Any hand
          this.handsfree.emit(`finger-pinched-${finger}`, {
            event: hands.pinchState[hand][finger],
            origPinch: hands.origPinch[hand][finger],
            curPinch: hands.curPinch[hand][finger]
          })
          this.handsfree.emit(`finger-pinched-${hands.pinchState[hand][finger]}-${finger}`, {
            event: hands.pinchState[hand][finger],
            origPinch: hands.origPinch[hand][finger],
            curPinch: hands.curPinch[hand][finger]
          })
        }
      }
    }

    return hands
  }
})

Examples and Use Cases

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Menu