Sequential Decision-Making

Agents & environment interact over time in a feedback loop:

These problems are known as Temporal credit assignment problems, categorized by:

• Needs sequential action selection to be solved.
• Has state transitions triggered by agent actions.
  • In other words, the environment changes over time.
• Involves delayed rewards, defining task success/failure.

Some examples:

• Robot navigation.
• Car driving.
• Playing games, like chess, backgammon, etc.
• etc.

In a more explicit definition, in these problems we want to answer the question: which action at what time step was responsible for the rewards? E.g.,

• Was the “brake action” to blame when crashing into a tree? Or was it the “overtake action” 10 seconds before on the narrow road?
• Did you pass your exam due to studying hard the day before? Or due to efforts during the entire quarter?

<aside> 📝 Which of the following are Temporal Credit problems?

1. Medical treatment of a patient over time.
2. Inspection of manufactured parts on a survey belt.
3. Determining the best slot machine (our of 5 in a casino).
4. Controlling traffic lights at a busy intersection.

• Answer </aside>

Suppsoe you are at $S$ and what to go to $G$:

You want to avoid falling in the cliff. Moreover, you want to get to $G$ fast, but your movements are noisy.