Add face, hand, and pose tracking to your projects in a snap ✨👌

// Enable Mediapipe's "Hands" model
const handsfree = new Handsfree({hands: true})
// Enable plugins tagged with "browser"
handsfree.enablePlugins('browser')
// Start tracking
handsfree.start()
Powered by

Installing

Some models are over 10Mb+ and may take a few seconds to load initially, but subsequent uses of the same version will be cached and loaded much more quickly.

<head>
  <!-- Include Handsfree.js -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@8.4.4/build/lib/assets/handsfree.css" />
  <script src="https://unpkg.com/handsfree@8.4.4/build/lib/handsfree.js"></script>
</head>
 
<body>
  <!-- Instantiate and start it after <body> -->
  <script>
    const handsfree = new Handsfree({hands: true})
    handsfree.enablePlugins('browser')
    handsfree.start()
  </script>
</body>

Handsfree.js is a pluggable library and comes with plugins to help you browse handsfree

Try it! Click the button below to read the rest of this page with hand tracking:

handsfree.update({hands: true})
handsfree.enablePlugins('browser')
handsfree.start()
  • Point your palm towards the webcam to move the red pointers 🖐
  • Pinch your index finger and thumb to “grab” the page 👌
  • Move your pinched hand to scroll the page
Hand Index [0] Middle [1] Ring [2] Pinky [3]
Left
Right

 

The circles above will turn red as you pinch the corresponding fingers. This feature is part of the Pincher plugin. The Pinchers plugin is tagged with “browser” and so is automatically enabled when you run handsfree.enablePlugins('browser')

Handsfree.js comes with a set of common helper styles and events to help you react to states quickly

You can use helper classes like .handsfree-show-when-loading and listen to events like handsfree-loading to help you quickly setup loading states for your app. For example, here’s a common “Go Handsfree” button pattern that is often used:

<span class="button">
  <button class="handsfree-show-when-stopped handsfree-hide-when-loading" onclick="handsfree.start()">Go Handsfree</button>
  <button class="handsfree-show-when-loading" disabled>...loading...</button>
  <button class="handsfree-show-when-started handsfree-hide-when-loading" onclick="handsfree.stop()">Stop Handsfree</button>
</span>

Handsfree.js comes with 5 models to mix and match, and comes with an API to quickly add even more

A person holding their hands up towards webcam, with a green hand skeleton overlaid on top of the hands in 3D

TensorFlow Handpose

handsfree.update({handpose: true})

This model includes a fingertip raycaster, center of palm object, and a minimal THREE environment which doubles as a basic debugger for your project.

A person making various face gestures, with a wireframe mesh laid on top of the face

MediaPipe FaceMesh

handsfree.update({facemesh: true})
  • 468 2D face landmarks
  • Track up to 4 faces at once
  • Extra helpers and plugins coming soon

This model doesn’t come with any bonuses or plugins yet but they’ll come soon. The API will remain exactly the same, so feel free to started with this model today!

A person flailing their arms and legs, with a skeleton overlaid above their arms, torso, legs, and face

MediaPipe Pose

handsfree.update({pose: true})
  • Full body mode with 33 2D pose landmarks
  • Upper body mode with 25 2D upper pose landmarks
  • Extra helpers and plugins coming soon

This model doesn’t come with any bonuses or plugins yet but they’ll come soon. The API will remain exactly the same, so feel free to started with this model today!

A split screen view: on the right is a person moving their head and making face gestures, and on the right is a pointer moving and clicking things in tandem

Jeeliz Weboji

handsfree.update({weboji: true})
  • 6DOF head pose estimations
  • 11 face morphs and 16 helper states
  • Comes with “Face Pointer” based plugins

This model uses morphs to help estimate various face states simultaneously and includes assistive tech plugins for browsing pages with face gestures.

Combine face, pose, and hand tracking with other libraries such as p5.js, three.js, and A-Frame

Try it! Use the same pinch gesture 👌 to paint on the p5 sketch (the white area).

  • Use different fingers on your right hand to try different colors
  • Pinch your left pinky and thumb together to clear the drawing
  • Pinch outside the sketch to continue scrolling

Use head tracking to view, test, and develop WebXR applications hands-free

Try it! Click the button below to switch to Face Tracking and view the the A-Frame by slowly turning your head

  • Turning your head 15deg in real life turns it 3x in the A-Frame
  • Move closer and further away to move 2x in the A-Frame

Use Handsfree.js with any framework or platform and use them with hand, pose, and face tracking

Explore existing websites in new ways

Explore assistive technologies

Bridge the 2D and Extended Reality web

Explore hands-free art and music

Explore hands-free apps and games

Interact with the physical world

Try the Handsfree.js Pixelfelt Bookmarklet or the Handsfree Browser Extension and add hand, pose, and face tracking on supported sites

The Handsfree Pixelfelt Bookmarklet loads an integrated code environment for Handsfree.js

The Handsfree.js Browser Extension Starter Kit helps you explore any website, including WebXR ones

The Handsfree.js repo can be loaded as an unpacked Chrome Extension, see how

How it works

  1. Drag the bookmark to your bookmarks bar
  2. Click the bookmark to load Pixelfelt into a site
  3. Control pointers, click on things, and scroll pages with face or hand gestures
  4. Use the code editor to make a website usable hands-free, live as everything is running
  5. Drag the toggleable Picture-in-Picture around as I stream

Explore some of the sites converted to be used hands-free with the Handsfree.js Pixelfelt Bookmarklet

Quickstart Workflow

The following workflow demonstrates how to use all features of Handsfree.js. Check out the Guides and References to dive deeper, explore some examples, and feel free to post on Discord if you get stuck!

// Let's enable face tracking with the default Face Pointer
const handsfree = new Handsfree({weboji: true})
handsfree.enablePlugins('browser')

// Now let's start things up
handsfree.start()

// Let's create a plugin called "logger"
// - Plugins run on every frame and is how you "plug in" to the main loop
// - "this" context is the plugin itself. In this case, handsfree.plugin.logger
handsfree.use('logger', data => {
  console.log(data.weboji.morphs, data.weboji.rotation, data.weboji.pointer, data, this)
})

// Let's switch to hand tracking now. To demonstrate that you can do this live,
// let's create a plugin that switches to hand tracking when both eyebrows go up
handsfree.use('handTrackingSwitcher', ({weboji}) => {
  if (weboji.state.browsUp) {
    // Disable this plugin
    // Same as handsfree.plugin.handTrackingSwitcher.disable()
    this.disable()

    // Turn off face tracking and enable hand tracking
    handsfree.update({
      weboji: false,
      hands: true
    })
  }
})

// You can enable and disable any combination of models and plugins
handsfree.update({
  // Disable weboji which is currently running
  weboji: false,
  // Start the pose model
  pose: true,

  // This is also how you configure (or pre-configure) a bunch of plugins at once
  plugin: {
    fingerPointer: {enabled: false},
    faceScroll: {
      vertScroll: {
        scrollSpeed: 0.01
      }
    }
  }
})

// Disable all plugins
handsfree.disablePlugins()
// Enable only the plugins for making music (not actually implemented yet)
handsfree.enablePlugins('music')

// Overwrite our logger to display the original model APIs
handsfree.plugin.logger.onFrame = (data) => {
  console.log(handsfree.model.pose?.api, handsfree.model.weboji?.api, handsfree.model.pose?.api)
}

Going further

This site is still a work in progress and new examples, tools, and tutorials will be published in the coming weeks. For now, check out the following resources:

Get Involved!

Handsfree.js has been in development for over 3 years, and it’s time to open up contributions to the wider tech community! Here are some ways you can get involved:

Project Notes

  • I’m still just getting this site started so it might be buggy
    • The old Vaporwave docs are still available at Handsfree.js.org
    • I try to go extra with everything so some demos may appear slow
    • This site is still not accessible yet but my goal is to eventually have it get recognized for it’s magical accessibility
    • I am very behind on documentation 😅
  • I’ve been working on Handsfree.js for 3 years, beginning with SeeClarke.js, with the goal of creating a creative assistive tech platform that is so good that I myself use it full time
    • Handsfree.js is one component of the whole thing
      • Midiblocks is a “Face Coding” platform built on Blockly, which is itself accessible with screen readers!
        • Goal is to encapsulate Handsfree.js into blocks that people can use to play games etc
  • I’ll have a roadmap and more information up soon, follow along on my #100DaysofCode
Menu