Enhancing Reward Acquisition in Reinforcement Learning II: Advancements in Reward Discovery
In the complex world of multi-agent reinforcement learning (MARL), traditional reward learning struggles, especially in scenarios like the "Letter Problem" where environments are dynamic and global rewards are sparse. A new approach, Counterfactual Reward Learning, is proving to be a promising solution.
Counterfactual Reward Learning, a refinement of reward learning, focuses on finding the reward function of a Reinforcement Learning problem. Its primary advantage lies in its ability to enhance credit assignment and stabilize learning in non-stationary and sparse reward environments.
In the "Letter Problem", Counterfactual Reward Learning constructs a counterfactual action advantage function. This function evaluates each agent's reward based on what their reward would be if they had acted differently, while other agents' actions remained fixed. This isolation of individual action impacts within the group context is key to addressing the challenges posed by the "Letter Problem".
The Counterfactual Group Relative Policy Advantage (CGRPA) technique, a specific application of Counterfactual Reward Learning, provides intrinsic credit signals that reflect each agent’s impact under evolving task demands. This aids in better attributing credit to individual contributions even as the environment and opponent difficulty change dynamically.
Moreover, CGRPA improves training stability and final performance by offering more reliable policy updates during curriculum learning. It also handles non-stationary environments and sparse rewards by enhancing intrinsic rewards through counterfactual evaluation, guiding more efficient and stable learning of policies despite changing opponent strategies or task conditions.
The value function in this setting is a function of a seen history with length n and a policy π, incorporating an expectation over the reward function based on the reward learning process. The environment has one state and one observation for every grid, modeling the uncertainty about the reward using the distribution given by the reward learning process.
Despite its advantages, Counterfactual Reward Learning is not without practical and technical limitations. The need to come up with a counterfactual policy and the need for a model of the environment are among the challenges that need to be addressed.
Initially, the agent is uncertain about which room to clean and until it reads the letter, the reward-function learning process gives equal probabilities to cleaning the kitchen and the living room. After reading the letter, the reward-function learning process gives 100% to the reward function corresponding to the instruction from the letter.
The counterfactual policy always ends with cleaning the kitchen, but an agent learning their policy using the counterfactual value function may end up cleaning either room depending on what is written in the letter. Computationally, all of this can be implemented like a typical Reinforcement Learning problem, but with multiple states that correspond to the belief about the reward function that the agent holds in that state.
Counterfactual Reward Learning is particularly useful when there is an idea of the strategy that will lead to learning the correct reward function, but not of how the final policy should look like. The "Letter Problem" is modelled within the POMDP framework as a 3x3 gridworld with the agent, envelope, kitchen, and living room represented as grids.
Under the right circumstances, Counterfactual Reward Learning can be a helpful tool for making the outcome of a Reward Learning process more aligned with human intentions. However, an unsafe version of the Letter Problem arises when the agent can manipulate the letter before reading it, leading the agent to always clean the kitchen even if the instructions were to clean the living room.
To get the probabilities of a reward function from the counterfactual learning process, you need to compute the P(h|s(1) = s, π) term, which requires either the ability to simulate histories based on a policy or the resources to run enough episodes to estimate this probability.
In conclusion, Counterfactual Reward Learning offers a promising solution to the challenge of reward ambiguity and instability in the "Letter Problem" and similar cooperative settings. By using counterfactual reasoning to evaluate how different actions affect outcomes, it leads to more effective and robust learning.
The Counterfactual Reward Learning, an enhancement of traditional reward learning, utilizes artificial intelligence to construct a counterfactual action advantage function, providing a means to evaluate each agent's reward based on what their reward would be if they had acted differently, while other agents' actions remained fixed. This technique, specifically the Counterfactual Group Relative Policy Advantage (CGRPA), aids in better attributing credit to individual contributions even as the environment and opponent difficulty change dynamically, thus improving training stability and final performance.
In the "Letter Problem," where global rewards are sparse and environments are dynamic, Counterfactual Reward Learning allows for more reliable policy updates during curriculum learning and handles non-stationary environments by enhancing intrinsic rewards through counterfactual evaluation, guiding more efficient and stable learning of policies despite changing opponent strategies or task conditions.
