Discovery of Differences in the Brain's Perception of "Naturalness" in Human Jumping Motion Based on Vertical Projectile Motion Model
Towards improved realism of character movements in metaverse space and visual media

News Highlights:

Outline of the Findings

Background

NTT Communication Science Laboratories is conducting research to establish design principles for media content that feels more immersive by understanding of the brain mechanisms involved in experiencing various impressions of media content. In movies and video games, full-body movements of CG characters may appear sometimes unnatural, while exaggerated actions can still feel realistic. Understanding how the brain processes human motion is important for improving the visual experiences in media content.
　For example, in works of fiction, jumps that appear overly floating or unnaturally low tend to be perceived as unnatural. Jumping is a fundamental movement for humans since it is heavily influenced by gravity, an important law of physics for terrestrial animals. This makes it an ideal subject for studying the brain mechanisms involved in judging the naturalness of body movements. A leading hypothesis in perceptual psychology research has been that humans estimate the motion of objects under gravity using an approximate physical model. Since this model is approximate, it can sometimes lead to perceiving object motion as natural even, in conditions that differ from real-world physics¹. In fact, it has been known that when a ball bounces under weaker gravity than Earth, it is often perceived as natural. If human jumping movements were judged using the same approximate physics model, one would expect that movements appearing to occur under weak gravity as more natural. However, previous research in this context has only dealt with the motion of a simple object such as a ball, and complex movement like human jumping—where multiple body parts coordinate²-unexplored.

Research findings

This study conducted a psychophysical experiments³ in which participants rated the naturalness of various modified jumping movement, and used computational modeling to analyze how gravity-based predictions underly the perception. The result showed that even when jump movements were altered from actual human motion, as long as the trajectory followed Earth's gravity, participants still rated them as natural (Figure 2). In other words, this result suggests that the gravity model inherent in the brain is also used for impression judgment of body motion.

(1) Experiments on human's impressions for various modified jumps

Using actual human jump data, the researchers created new jump videos in which the peak height and the duration of the jump were modulated independently by between 0.5x to 2x (Figure 1). For example, if both the height and time length are reduced to 0.5x, the jump appeared shorter and faster, while if both were increased to 2x, the jump appeared taller and slower. In an online experiment, participants observed these videos and evaluated the "naturalness" of the jumps.

Figure 1 Example of Jump Video with Independently Modified Height and Duration Used in the Experiment

The participants' naturalness impressions of these videos are shown in Figure 2. Interestingly, it was found that not only the original jump video (with both height and duration 1x) but also some jump videos in which height and duration were modified to differ from reality, were still evaluated as natural. These modified jump videos follow the trajectory predicted by laws of gravity on Earth (blue line in Figure 2). In other words, even if a jump's height or duration appears unlikely based on a person's physical size, as long as the jump trajectory adheres accurately to the laws of gravity on Earth, in tends to be perceived as natural. On the other hand, videos that followed the gravitational laws previously reported as natural for simple bouncing motion tended to be rated as less natural (yellow line in Figure 2).

Figure 2 Results of Participants' Evaluation of the Perceived Naturalness of the Jump Videos
Note. Each grid cell corresponds to one video, and the whiter the cell, the more natural the video was perceived to be. These results indicate that jump movements that follow the laws of gravity of Earth (jump motions near the blue line) are perceived as more natural.

(2) A computational model for determining whether a jump trajectory follows the laws of gravity on Earth

To explain the results of human perception of natural movement, we devised a model⁴ to calculate whether the jump trajectory obeys the law of gravity on Earth. The model makes three assumptions about the processing in the observer's brain. (1) First, the observer calculates the initial speed of the jump from the video. (2) Next, the observer predicts how the initial speed of the jump will influence the speed of the jump according to the laws of gravity as they apply on Earth. (3) Finally, the observer compares how the predicted jump speed changes with the actual speed changes in the video to measure the discrepancy between them. The smaller the error, the more natural the jump appears.
　Figure 3 illustrates the relationship between the calculated errors of the speed change for each video and participants' evaluations of naturalness. As shown in the figure, jumps with smaller calculation errors tend to be perceived as more natural. The correlation coefficient between the model's result and the participant's perception of naturalness was -0.89, demonstrating a strong relationship between the accuracy of the model's predictions and how natural the jumps appeared to observers. Furthermore, when we varied the gravitational acceleration used in the model, testing values from lower to higher than Earth's actual gravity, we found that the strongest correlation occurred when the model used a gravitational acceleration value nearly identical to Earth's. This suggests that, unlike the perception of simple object motion—where an approximate physics model is often used—human perception of body movement relies on an accurate internal model of Earth's gravitational laws to judge naturalness.

Figure 3 Correlation Between Computational Model Predictions and Perceived Naturalness of Jumping Motion
Note. (Left) There is a strong correlation between the error calculated by the computational model (which predicts jumps based on Earth's gravitational laws) and participants' evaluations of the perceived naturalness of the jumps. (Right) When the model uses Earth's gravitational acceleration, the correlation with participants' naturalness ratings is at its highest.

Outlook

These findings suggest that humans possess distinct models of naturalness for different types of external objects in motion. Moving forward, we will investigate the impression of the movements from animals and other entities that fall between simple objects and human motion. This will help uncover the mechanisms behind how different perceptual models apply to various types of movement. These insights not only significantly advance our understanding of how humans judge the naturalness of body movements but also contribute to identifying key elements necessary for enhancing the realism of character motion in the metaverse and visual media.

[Paper information]

Journal: Proceedings of the Royal Society B: Biological Sciences
Subject: Computational account for the naturalness perception of others' jumping motion based on a vertical projectile motion model
Authors: Takumi Yokosaka, Yusuke Ujitoko and Takahiro Kawabe
DOI: https://doi.org/10.1098/rspb.2024.1490
URL: https://royalsocietypublishing.org/doi/10.1098/rspb.2024.1490

^1.Since this model is approximate, it may incorrectly measure object motion
The appropriate model used here can be described as a model in which the laws of physical motion in the real world are overly generalized [Vicovaro, 2023]. This model has advantages such as being able to explain many physical phenomena encountered in daily life while maintaining a low computational load for the human brain. However, due to its simplifications, there are cases where it fails to explain object motion.

^2.Complex movements involving coordinated motion of multiple body parts
Motion capture data of the major body joints, as shown in Figure 1. This study used the Carnegie Mellon University Motion Capture Database. (http://mocap.cs.cmu.edu/)

^3.Psychophysical experiment
The experiment design measured the participants' responses change when presented with controlled stimuli (in this study, videos in which jump height and jump duration were manipulated). The goal of this study is to investigate the underlying human information processing mechanisms that exist between these variables, specifically examining how participants rate the perceived naturalness of jumps.

^4.Model for calculating whether a jump trajectory follows the laws of gravity on earth
This model can be expressed using the following equation.

Here, v represents the observed velocity series of a jump, and v_predict is the velocity series predicted by the model based on Earth's gravitational acceleration. If the overall (i.e., over N period) discrepancy between v and v_predict is small, the RMS velocity error (RMS_vel) is also small. A smaller RMS_vel value indicates that the observed jumping motion closely follows the velocity changes expected under Earth's gravitational acceleration.

About NTT

NTT contributes to a sustainable society through the power of innovation. We are a leading global technology company providing services to consumers and businesses as a mobile operator, infrastructure, networks, applications, and consulting provider. Our offerings include digital business consulting, managed application services, workplace and cloud solutions, data center and edge computing, all supported by our deep global industry expertise. We are over $93B in revenue and 330,000 employees, with $3.6B in annual R&D investments. Our operations span across 80+ countries and regions, allowing us to serve clients in over 190 of them. We serve over 75% of Fortune Global 100 companies, thousands of other enterprise and government clients and millions of consumers.