Spray on Smart Skin

Examined a number of related methods, without success. 

Spray-on Smart Skin Reads Typing and Hand Gestures No cameras, keyboards, sensor gloves, or VR motion trackers required      PRACHI PATEL  Spectrum IEEE

KYUN KYU “RICHARD” KIM/  STANFORD UNIVERSITY

A new AI learning scheme combined with a spray-on smart skin can decipher the movements of human hands to recognize typing, sign language, and even the shape of simple familiar objects. The technology quickly recognizes and interprets hand motion with limited data and minimal training and should work for all users, its developers say.

Besides finding use in gaming and virtual reality, the new hand-task-cognition technology could allow people to communicate with others and with machines using gestures. Other applications the technologists envision include surgeons remotely controlling medical devices, as well as a new modality for robots and prosthetics to achieve object and motion recognition.

The gesture-recognition technology developed so far has relied on bulky wrist bands that measure electrical signals produced by muscles or on wearable gloves with strain sensors at each joint. Other approaches have involved cameras that track human motion and interpret it using machine learning. Those motion-capturing camera systems require images taken from multiple surrounding angles, which means multiple cameras are needed for a single gesture-recognition system. These multicam systems also suffer from the inherent limitations of vision-based sensors, says Sungho Jo, a professor in the school of computing at the Korea Advanced Institute of Science and Technology. Such limitations include regions of a workspace that are not covered by multiple cameras, as well as errors that inevitably occur when a hand or other object is occluded from view.

The software used so far has also been cumbersome. Researchers have typically relied on machine-learning models based on supervised learning algorithms that are computationally intense. They require that a large amount of data be collected for each new user and task, all of which require humans to label.  ... '