Situational awareness for humanoid robots is more complex than that needed by standard industrial robots. Humanoids require a semantic understanding of the environment, plus the ability to recognize objects and predict future states.
That takes a multi-layered, real-time awareness of their surroundings to operate safely and effectively alongside humans, including walking and grasping objects. Data from multiple sensor modalities like cameras, mm-wave radar, LiDAR, IMUs, encoders, force sensors, and more must be fused and integrated into a single, cohesive 3D model.
Sensor fusion relies on a variety of software tools to integrate multimodal sensor data into a single unified spatial model. That provides humanoids with the situational awareness required to complete complex human-like tasks like navigation, walking, and grasping objects in real-time.
It also supports safe and predictable interaction with nearby people and other robots. From the bottoms of feet to fingertips and shoulders, data from tactile force, torque, and load sensors help track and maintain dynamic physical interactions (Figure 1).
Being aware
Situational awareness is not an end; it’s the first part of a continuum of perception and participation by humanoids in their surroundings. It relies on a three-layer architecture and enables humanoids to independently perform useful tasks.
The process begins with perception, the humanoid’s senses, and the sensors. Masses of heterogeneous data about the environment are consumed from 3D LiDAR, cameras, tactile force, microphones, IMUs, and more. That data is fused to construct a 3D digital map, and as the basis for understanding and interacting with people.
That information is combined with the requirements of specific tasks like walking or grasping objects in the reasoning phase using a variety of AI/ML tools. The reasoning phase develops detailed task plans to control humanoid motion and interactions with its surroundings. The results of reasoning are translated into task execution using physical AI (PAI) tools.
Circular process
Situational awareness is a circular process. As tasks are executed, physical states of elements like feet, legs, arms, hands, and so on (proprioception) are combined with external environmental data (exteroception) to enable humanoids to adapt in real time to unexpected developments.
That adaptability is required to ensure that the results match the expectations generated by the reasoning and task planning step. That demands continuous and complex real-time sensor fusion.
Three levels of fusion
It takes multiple levels to effectively fuse data from multiple sensor modalities (Figure 2). Fusion starts with raw data. This data-level fusion uses techniques like Kalman and complementary filters, weighted averages, and other tools to diminish noise and improve data quality. Features are also extracted at this level.
Feature fusion is the second level and combines the features into a unified feature vector structured to provide a more comprehensive picture of the overall sensor data. Commonly employed tools include factor analysis and principal component analysis (PCA), along with multidimensional scaling (MDS) or other dimensionality reduction methods to increase the efficiency and effectiveness of the classification process.
Decision-level fusion is especially important when accuracy is crsitical but there is limited training data or high uncertainty. Methods like majority voting and Bayesian networks are used to perform a combined analysis on predictions or classifications from several models to arrive at an improved result.
Morphological computations
The use of hands, in place of less capable end-effectors, is a key for deployment of advanced humanoids. Wrist and finger touch (tactile force) sensors can be fused with visual information from RGB cameras to identify and safely grasp and manipulate even fragile objects. Development of simplified methods is being pursued.
For example, a prototype was fabricated using multi-material 3D printing. Compliant materials were used for the sensors and joints, but a rigid material was used for the “bones” to provide support.
The use of compliant materials means the hand conforms to the shape of an object without the need for positional feedback. The use of soft sensors further ensures more delicate contact with objects (Figure 3).
The tactile data acquired from grasping objects is analyzed using morphological computation, where the system’s physical form and dynamics perform tasks without the need for energy-intensive external processing. In this prototype system, a multiple-grasp implementation was used to enhance object identification.
Summary
Situational awareness is needed to support the effective and safe operation of humanoids. It requires the fusion of data from diverse sensor modalities, AI/ML analysis of the resulting data, and translation into execution of specific tasks. It’s a circular real-time discipline that’s still being refined.
References
Beyond Wheels: Designing and Building a Walking/Bipedal Robot, KEYi Technology
Energy-Aware Sensor Fusion Architecture for Autonomous Channel Robot Navigation in Constrained Environments, MDPI sensors
Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition, MDPI sensors
Humanoid Factors: Design Principles for AI Humanoids in Human Worlds, arXiv
Humanoid Robot Reference Guide, STMicroelectronics
Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System, MDPI sensors
Recent Advancements in Humanoid Robot Heads: Mechanics, Perception, and Computational Systems, MDPI biomimetics
Safety meets speeds: how next-gen sensors free robots from cages, imec
Soft Robotic Hand with Tactile Sensing, Oxford Robotics Institute
The Humanoid Robotics Nervous System: Perception, Balance, and Real-Time Control, Panasonic
What Is Morphological Computation? On How the Body Contributes to Cognition and Control, MIT
Related EEWorld Online content
What kinds of PAI dev kits are available for humanoid robotics?
How do humanoid robots and physical artificial intelligence fit into industry 5.0?
Does Industry 5.0 exist and what does it mean?
How is power limiting the adoption of physical artificial intelligence in humanoid robotics?
What is physical artificial intelligence and why is it important?