5.2. Model-Based Learning

Model-Based Learning

In model-based learning, the agent builds empirical models of the transition and reward functions, such that
$\hat{T}(s,a,s’)=\frac{\#(s,a,s’)}{{\sum }_{s’}\#(s,a,s’)}\text{and}\hat{R}(s,a,s’),$
where $\hat{R}$ is estimated based on observed rewards. It does so as follows:

1. Exploration: the agent executes actions and collects samples $(s,a,s’,r)$.
2. Counting: the agent keeps track of how many times each sample is executed.
3. Normalization: the agent divides the count of each sample $(s,a,s’,r)$. by the total count of Q-states $(s,a)$ to convert frequencies into probabilities (as per the equation above).
4. Reward estimation: average observed rewards for each $(s,a,s’)$.