Reinforcement Learning (RL) is an area of Machine Learning where we attempt to optimize the actions or decisions that an agent takes to navigate through an environment. Whether a decision is optimal is solely determined by the reward function, which is typically complex and challenging to design by hand: What would be a suitable reward function for a robot assisting humans in a household environment or for autonomous vehicles navigating through a busy urban environment? Moreover, even well-defined reward functions can lead to unexpected behaviours due to distributional shifts or over-optimization, raising practical and safety concerns.
Reinforcement Learning from Human Feedback (RLHF) is a variant of RL that aims to model a reward function either from observed demonstrations or human feedback, addressing not only the limitations of traditional reward function engineering but also benefitting agent alignment. The reward function is dynamically refined and adjusted to distributional shifts as the agent learns.
RLHF has been successful in many applications, such as robotics, games, etc. It has specially played a decisive role in the training of large language models (LLMs), like ChatGPT, where it helps in targeting the model's capabilities toward human preferences.
We will focus on approaches that learn a reward model from human feedback and then use this model to train a policy. Furthermore, we focus on approaches where human feedback is the only source of truth about the objective. This excludes approaches to reward shaping, feature engineering, and other forms of human guidance that are supplementary to a given objective. We also put an emphasis on providing feedback in an interactive, online manner. This excludes imitation learning, learning from demonstration, and pure inverse RL. Finally, we focus on works in which the human is included in the loop, but the agent is not blocked by the human’s feedback, and the human does not need to be present continuously. This distinguishes RLHF from more direct methods of incorporating a human into the RL loop, and we believe that this is key for practicality and efficiency.
The field of inverse RL aims to infer reward functions from human demonstrations. While this can partially resolve reward engineering challenges, it faces inherent difficulties:
Interactive feedback, by contrast:
Interactive feedback can also be used to complement demonstrations, in which case it can be used to shape and refine capabilities learned through initial training, like behavioural cloning, thereby preventing overfitting to demonstrated behaviour.
Reward engineering in RL presents significant challenges, as accurately specifying reward functions is notoriously difficult. These challenges can be mitigated by utilizing human feedback, which enables training agents for tasks that are hard to define manually and helps avoid safety issues arising from misaligned rewards. Safety issues related to a misalignment between the agent’s and the human objectives are studied as the AI alignment problem.
Excessive optimization for poorly specified rewards typically leads to unintended behaviours. Agents may exploit simulation flaws for higher rewards or engage in reward hacking, where the behaviour maximizes the specified reward but deviates from the intended objective. This is evident in cases where agents focus on intermediate rewards without achieving the actual goal, or prematurely exit a game to avoid negative rewards. The root of these issues is that the reward function does not properly reflect the actual learning task. It is crucial to prevent misaligned reward functions from leading to harmful outcomes, like a care robot causing injury or a self-driving car jeopardizing road safety.