I'm presenting this 3-5:30pm on Saturday, Hall 3 #396 🌞
come chat about designing rewards and intrinsic motivation for RL + meta RL!

1/3 Out now: new paper on people's perception of AI (robot) creativity! Core finding: we attribute more creativity to a creative act if people not only see the final artwork, but also its creation process & the robot making it. Video: vimeo.com/1073134853 Open-access paper: doi.org/10.1145/3711...

BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping Intrinsic motivation and reward shaping guide reinforcement learning (RL) agents by adding pseudo-rewards, which can lead to useful emergent behaviors. However, they can also encourage counterproducti...

Get all the details in our paper: arxiv.org/abs/2409.05358 🚀

This work was a joint effort with Michael Dennis and Stuart Russell at UC Berkeley!

5️⃣We demonstrate our framework in Mountain Car. We set the potential to the maximum displacement the agent learnt to reach so far, signaling the value of its training. Rewarding displacement directly (pink) led to reward-hacking but the BAMPF (green) preserved optimality✅

4️⃣We get a new typology for intrinsic motivation & reward shaping terms based on which BAMDP value component they signal! They hinder exploration if they align poorly with actual value, e.g., prediction error is high for watching a noisy TV but no valuable information is gained.

3️⃣To guide more efficient exploration, BAMPF potentials should encode BAMDP state value. To gain further insights, we decompose BAMDP value into the value of the information gathered🧠 and the value of the MDP state given prior knowledge only🌎.

2️⃣Harmful reward-hacking policies maximize modified rewards to the detriment of true rewards. We prove that converting IM and reward shaping terms to BAMDP potential-based shaping functions (BAMPFs) prevents hacking, and empirically validate this in both RL and meta-RL.

1️⃣We cast RL agents as policies in Bayes-Adaptive MDPs, which augment the MDP state with the history of all environment interactions. Optimal exploration maximizes BAMDP state value, and pseudo-rewards guide RL agents by rewarding them for going to more valuable BAMDP states.

🚨Our new #ICLR2025 paper presents a unified framework for intrinsic motivation and reward shaping: they signal the value of the RL agent’s state🤖=external state🌎+past experience🧠. Rewards based on potentials over the learning agent’s state provably avoid reward hacking!🧵