For the past three decades, NASA’s Mars rovers have been fundamental to our understanding of the Red Planet. However, the inherent communications delay due to the vast distance between Earth and Mars has always been a significant challenge, resulting in planning ahead for these communication episodes.
This talk will explore how we can overcome this obstacle by equipping a minimalist rover with the ability to perform complex tasks and make decisions independently, harnessing the power of the Semantic Kernel and LLaVA model. Rather than planning ahead, control can be transferred to Semantic Kernel agents powered by interactive planners capable of working with plugins that can read sensors and control systems.
Moreover, the integration with LLaVA (Large Language and Vision Assistant) models improves the rover’s understanding of its environment. LLaVA can reason over images, providing valuable information to interactive planners at every step. This feature enables the rover to make informed decisions based on the environment, thereby further increasing its autonomy. This breakthrough could significantly reduce the impact of the 15-minute communications delay using AI, bringing us closer to the Mars rover that NASA has wanted for the past 30 years.
