Augmenting touchless gesture recognition with haptic feedback

By Vicky Messer

Ever since the development of the trackball in the 1940s and the consumer release of the computer mouse in the 1980s, engineers have sought to push the limitations of computer-human interaction in an attempt to achieve a more intuitive and natural user experience. An obvious avenue to explore has been the gestures used in face-to-face communications, and the market has come a long way especially when considering developments in sophisticated audio and visual capabilities. The importance of the human sense of touch has also been championed with haptic feedback added to many consumer devices. Traditionally, however, the two markets have been distinct entities.

According to market analyst firm GIA, the global market for gesture recognition[1] will exceed $12.7 billion in 2020, driven by expanding applications in gaming, healthcare, automation, consumer electronics and automotive sectors. Consumer electronics (including gaming) currently accounts for approximately 99% of the gesture recognition market. A second report[2] predicts the market for both gesture recognition and touchless sensing technologies will have a CAGR of 28% over the next few years and is predicted to be worth $24 billion worldwide by 2020.

Touching and Touchless

Controlling devices with human gestures is, of course, now commonplace: the ability to swipe, pinch, push virtual buttons and type on touchscreens is provided by almost every type of electronic equipment. However, almost all require the physical touching of controls or surfaces. Clearly, this has limitations such as wear and tear of electrical switches and vulnerability to contamination – the classic example being spilt coffee on a computer keyboard. Some less obvious examples include the possibility of cross-infection from medical equipment controls or public toilets, or possible injury from hot or cold surfaces or sharp or electrically charged objects.

Controlling equipment via human gestures without touching has become a growing area of interest to major technology companies over the past few years. The enormous global gaming market, in particular, has driven demand for the detection of body movements at a distance. However, a fundamental weakness of touchless controls is that they do not offer the desired level of precision and tactile feedback necessary to provide the best user experience. Users will prefer to feel a control or different textures and surfaces, and ideally also have audible confirmation. Effective simulation of tactile feedback – haptics – is, therefore, being seen as the key to unlocking the potential of touchless gesture recognition.

Tactile Feedback

The accuracy and usefulness of human gesture recognition is advancing rapidly and will continue to do so for decades to come. The issues associated with image-based systems are a focus of development efforts, including adaptability to varying lighting conditions and backgrounds and managing unwanted objects in the field of view. However, only seeing and hearing limits human perception and the ability to effectively control computers, which can be a significant problem particularly in safety critical applications. Simply recognising human gestures does not necessarily provide the tactile feedback experienced from physical controls such as switches and dials.

For example, even with the use of predictive text, there is the need for the user to check the screen to ensure the correct key has been pressed when using the virtual keyboard of a tablet, phone or touchscreen. Adding the third sense of touch provides feedback that a key, button or switch has been activated correctly and substantially enhances perception and control. Tactile feedback is even more important in some other applications such as vibrating the steering wheel to simulate the effect of driving over a rumble strip, which warns a distracted driver that their vehicle is drifting from its road lane. In some scenarios, tactile feedback will be an absolute necessity for the public acceptance of touchless gesture recognition.

Haptics

Haptic feedback is the sensation of touch and is already well known in computer gaming to provide a more immersive environment, and especially in various applications such as aircraft control where user attention is critical. Even flat-panel touchscreens can use programmed interference patterns between several tiny vibrators around the edge of the glass display to produce haptic feedback at the fingertips and simulate different surface textures.

Many new consumer products now feature haptic feedback – for example, major smart phone makers are now implementing vibrational haptic feedback in some of their latest models – generating lots of interest for numerous other applications worldwide. Some standard APIs (Application Programming Interfaces) are now available for research in the field of haptics, and several of these are being used to produce tactile sensations in simulations and training for remote and keyhole surgery such as laparoscopy and other medical applications.

Although tactile feedback provides a much more complete user experience – it closes the loop to provide helpful non-visual cues – almost all these approaches are limited by the need to be in physical contact with the haptic system. Therefore, they cannot be used in applications based on touchless gesture recognition at a distance.

Evaluation Kit

Providing touchless haptic feedback has been impossible until very recently. Ultrahaptics has developed a range of focused ultrasound techniques to augment recognition and control systems for hand gestures with touchless or mid-air haptic feedback. The company’s haptics evaluation kit – the only solution for touchless gesture recognition on the market today – includes a ‘leap motion’ camera and gesture recognition system. The kit allows developers to experiment and generate novel haptic sensations in touchless gesture control interfaces for a range of devices including computers, consumer electronics, domestic appliances and automotive dashboards.

Recognizing human gestures and using them to control computers and equipment is well proven given the existing market of billions of smart consumer device and other electronic equipment. The advantages of not touching equipment are valuable, if not vital, in many use cases. Touchless gesture recognition has therefore become a well-established market sector and an area of academic and intense commercial research by major global corporations.

There is little doubt that many challenges remain in perfecting the user experience, including maximizing the range and field of view, and particularly in the accuracy of detection – not only of position and movement – but perhaps more importantly in the correct understanding of a user’s intention. In addition to incremental improvements from enhanced software algorithms, plus more accurate lower-cost and lower-power camera and sensor hardware, tactile feedback adds another new dimension to the accuracy of touchless gesture recognition in interpreting a user’s commands and confirming the correct action.

 

[1] Global Industry Analysts Inc. – Gesture Recognition, June 2015

[2] Markets and Markets – Gesture Recognition & Touchless Sensing Market, June 2015

Vicky Messer is the Technical Marketing Manager at Ultrahaptics