What is Deep Reinforcement Learning?

Planning requires access to the model (T, R). However, in many cases, we don’t have this.

Reinforcement learning: learn optimal policy from (interaction) data.

Roughly:

• act using an ‘exploration policy’ $π$.
• collect the generated transitions: $\{ (s, a, s’, r) \}$.
• then improve the policy.

Two main approaches:

• model-based RL: learn a model, than plan.
• model-free RL: e.g., try to learn $q^*$ directly.

Until now we have seen model-based RL, the ‘classic’ approach:

• by counting!
• $N(s,a,s’)$ counts how often we saw $(s,a)→ s’$
• estimate $P(s’|s,a) = N(s,a,s’) / N(s,a)$
• (and similar for rewards)

Given model... use planning!

Current research is based on answering the question: how to combine this with deep learning?

Model free Reinforcement Learning

The combination of Reinforcement Learning and Deep Learning resulted in Model Free RL.