Q-Learning From Scratch

This is a sample post. Delete it once you’ve written your first real one.

The idea

Q-learning learns an action-value function $Q(s, a)$ that estimates the expected return of taking action $a$ in state $s$. The whole thing hangs on one update:

where $\alpha$ is the learning rate and $\gamma$ the discount factor.

The code

import numpy as np

def q_learning(env, episodes=500, alpha=0.1, gamma=0.99, eps=0.1):
    Q = np.zeros((env.n_states, env.n_actions))
    for _ in range(episodes):
        s, done = env.reset(), False
        while not done:
            a = env.action_space.sample() if np.random.rand() < eps else Q[s].argmax()
            s2, r, done = env.step(a)
            Q[s, a] += alpha * (r + gamma * Q[s2].max() - Q[s, a])
            s = s2
    return Q

Notes

• math: true in the front matter turns on KaTeX for this page.
• Code blocks get a copy button automatically.
• Set draft: false to publish.