Predicting what happens next with egocentric action prediction

Editorial / February 12, 2025

In the world of wordplay, a phrase can have different meanings. However, an innocuous word like "egocentric" could hold disparate inferences when applied to English, Psychoanalysis and Computer Science.

Picture this. In English, the word "egocentric" means thinking only of oneself, without regard for the feelings or desires of others. In the parlance of psychoanalysis, Sigmund Freud referred to the word as the drive by a person to amplify the vision of one's self and self-importance. 

However, scientists are attempting to couple the human brain with algorithms to predict future actions, calling this egocentric action prediction.

To achieve this goal, Dr. Snehasis Mukherjee, Associate Professor, Department of Computer Science and Engineering, has broken down the adjective of egocentrism into a noun and a verb. 

Using the Reduced Verb Space Generator (RVSGen) algorithm, Dr. Mukherjee feeds in data received from a person wearing a camera (noun) to predict what they will do next (verb).

In a world where wearable cameras are becoming increasingly common, predicting human actions from a first-person perspective is a fascinating challenge. Egocentric action prediction is an emerging field in computer vision that seeks to anticipate what a person will do next based on partial video footage. This research is critical for applications like assisting the elderly, enhancing robotics, and improving sports analytics.

Imagine the final ball of a match where six runs are needed to win or a decisive penalty kick to decide a tie-breaker, and one has no idea where the ball will be delivered. The research done by Dr. Mukherjee, an alumnus of the Indian Statistical Institute, can help the batter or goalkeeper know precisely what to expect.

"The main application area of egocentric action prediction is robot navigation. We can apply the system to detect and recognise the events around a moving robot. Further, the proposed system can be used as a tool for a visually challenged person. A camera is fixed on the person's head. It can alert the person if some unusual events are happening around them," says Dr. Mukherjee.

Working at the University's AI and ML Laboratory – where cutting-edge research takes place on Image and Video Content Analysis, Computer Vision, Neural Architecture Search, Medical Image Computing, and Probabilistic Machine Learning – Dr. Mukherjee says that this facility has helped students pursue research even after leaving the campus. 

"I started to work on Egocentric videos with my B.Tech students Manav Prabhakar and Kovvuri Sai Gopal Reddy under the Opportunities for Undergraduate Research (OUR) program. Prabhakar has completed his MS from the University of Michigan, Ann Arbor, and Sai Gopal is looking for an MS opportunity while working in a software company," says Dr. Mukherjee.

The research by Dr. Mukherjee paves the way for exciting advancements in AI-driven egocentric vision. Future enhancements could integrate additional sensory data, such as gaze tracking and audio cues, to further refine predictions.

Applications in autonomous robotics, assistive technology, and immersive gaming are just the beginning. Egocentric action prediction is a step toward a future where AI anticipates human needs, making technology more intuitive and responsive to our daily lives.