Precise Finger Tracking

 Fascinating and considerable piece.  Had experimented with a much simpler proposed method of interacting with packaging materials. 

AuraRing: Precise Electromagnetic Finger Tracking   By Farshid Salemi Parizi, Eric Whitmire, Shwetak Patel

Communications of the ACM, October 2022, Vol. 65 No. 10, Pages 85-92  10.1145/3556639

Wearable computing platforms, such as smartwatches and head-mounted mixed reality displays, demand new input devices for high-fidelity interaction. We present AuraRing, a wearable magnetic tracking system designed for tracking fine-grained finger movement. The hardware consists of a ring with an embedded electromagnetic transmitter coil and a wristband with multiple sensor coils. By measuring the magnetic fields at different points around the wrist, AuraRing estimates the five degree-of-freedom pose of the ring. AuraRing is trained only on simulated data and requires no runtime supervised training, ensuring user and session independence. It has a dynamic accuracy of 4.4 mm, as measured through a user evaluation with optical ground truth. The ring is completely self-contained and consumes just 2.3 mW of power.

1. Introduction

New computing platforms are becoming increasingly coupled to the user. Wearable devices such as smartwatches, smart rings, and head-mounted mixed reality devices are offering unprecedented access to computing and information on the go. As computing platforms evolve from devices we carry with us to devices we wear on us, there is a renewed demand for input techniques that are decoupled from the display. These input devices must be continuously available and subtle to support various contexts of use while enabling robust and expressive interaction.

Many of today's most effective input devices rely on continuous tracking of the fingers or handheld objects; smartphones and smartwatches track 2D finger position, a mouse tracks 2D hand position, and augmented reality devices like the Microsoft HoloLens 2 track the 3D pose of the finger. These proven input techniques stand in contrast to existing lines of research around sensor-based efforts to track finger motion, which often focus on discrete, gesture-based interactions.5,17,20 Although gesture recognition is useful, it is just one part of the interaction language needed for mobile wearable computing. For common tasks such as object manipulation, drawing, sliding, swipe-based text input, or body-pose reconstruction, an input device that supports absolute, continuous tracking with millimeter-level precision will be most appropriate. Moreover, building a gesture recognizer on top of a tracking system, instead of direct classification from a sensor stream, can enable input devices that are more robust and extensible (for example, supporting different users and contexts out of the box).

In this work, we present AuraRing,14 a precise, millimeter-level ring tracking system. AuraRing consists of two components: a wireless, self-contained ring and a wristband that tracks the absolute position and orientation of the ring with respect to the wristband in real time. The low-power, battery-operated ring generates an oscillating magnetic field around the hand. As the user moves their finger and wrist, the relative position and orientation of this field changes with respect to the wristband. Sensors embedded in the wristband measure these fields at different locations and use these measurements to estimate the pose of the ring with respect to the wristband. AuraRing is effectively a 5 degree-of-freedom (DoF) tracking system—that is, it tracks three positional components (x, y, z) and two rotational components (yaw and pitch) of pose. When the ring is worn, this allows AuraRing to capture flexion/extension and abduction/adduction of both the wrist and finger metacarpal joint.

Compared with prior work, our approach leverages common device form factors—a wristband and a ring—and does not require affixing magnets,3 iron-core coils,4 or magnetometers3 to the fingertips. AuraRing leverages the insight that an electromagnetic coil is better placed around the finger than on it. Our approach also delivers significant improvements in power, range, portability, and tracking precision. Compared to a typical commercial EM tracking system,15 AuraRing uses approximately 2000× less power in a much smaller form factor.

We present two tracking solutions that leverage this model—a optimization-based approach and a neural network approach trained only on simulated data. We demonstrate that at runtime, AuraRing can track finger motion across different users despite small changes in how the device is positioned on the wrist and finger.

AuraRing offers a flexible solution for tracking the finger in either direct or indirect pointing tasks. We envision that AuraRing can be used either as a standalone input device or in tandem with a wrist-tracking solution. For standalone scenarios, AuraRing offers a rich input source for smartwatches or smart glasses. With AuraRing, a user can provide input using their finger and wrist while keeping their arm motionless at their side or resting on a table. This capability is particularly useful in public settings, when large-hand motions would be distracting and socially unacceptable.  ... '