Learning to Perceive the World Through Control: Empowerment-Based Representation Learning

Lay Summary

Most real-world reinforcement learning problems involve very high-dimensional observations that contain many irrelevant details and distractors. For example, a robot navigating a room should focus on walls, doors, and obstacles—not the color of the floor or changing lighting conditions. In reinforcement learning, however, the agent must collect its own data through directed exploration, which becomes especially challenging when observations are noisy and cluttered, since it is difficult to tell which aspects of the environment are meaningful and under the agent's control versus which are simply noise.

In this work, we show that Empowerment—an objective that measures how much influence an agent has over its environment—provides a principled way to learn concise representations that capture only the aspects of the environment that matter for control while ignoring irrelevant distractors. In this work, we show that Empowerment—an objective that measures how much influence an agent has over its environment—provides a principled way to learn concise representations that capture only the aspects of the environment that matter for control while ignoring irrelevant distractors. We show this property both theoretically and emperically. This makes empowerment a great candidate for unsupervised RL representation learning because it prescribe how to collect data and how to learn representations at the same time.

Finally, we demonstrate that these representations improve performance on downstream tasks, particularly in environments with noisy or distracting observations.