[{"content":"This is a sample post. Delete it once you\u0026rsquo;ve written your first real one.\nNotes math: true in the front matter turns on KaTeX for this page. Code blocks get a copy button automatically. Set draft: false to publish. ","permalink":"https://dec0dedd.github.io/posts/rl_iteration/","summary":"\u003cp\u003eThis is a sample post. Delete it once you\u0026rsquo;ve written your first real one.\u003c/p\u003e\n\u003ch2 id=\"notes\"\u003eNotes\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003emath: true\u003c/code\u003e in the front matter turns on KaTeX for this page.\u003c/li\u003e\n\u003cli\u003eCode blocks get a copy button automatically.\u003c/li\u003e\n\u003cli\u003eSet \u003ccode\u003edraft: false\u003c/code\u003e to publish.\u003c/li\u003e\n\u003c/ul\u003e","title":"Intro to Value and Policy Iteration"},{"content":"This is a sample post. Delete it once you\u0026rsquo;ve written your first real one.\nThe idea Q-learning learns an action-value function $Q(s, a)$ that estimates the expected return of taking action $a$ in state $s$. The whole thing hangs on one update:\n$$ Q(s, a) \\leftarrow Q(s, a) + \\alpha \\left[ r + \\gamma \\max_{a'} Q(s', a') - Q(s, a) \\right] $$where $\\alpha$ is the learning rate and $\\gamma$ the discount factor.\nThe code import numpy as np def q_learning(env, episodes=500, alpha=0.1, gamma=0.99, eps=0.1): Q = np.zeros((env.n_states, env.n_actions)) for _ in range(episodes): s, done = env.reset(), False while not done: a = env.action_space.sample() if np.random.rand() \u0026lt; eps else Q[s].argmax() s2, r, done = env.step(a) Q[s, a] += alpha * (r + gamma * Q[s2].max() - Q[s, a]) s = s2 return Q Notes math: true in the front matter turns on KaTeX for this page. Code blocks get a copy button automatically. Set draft: false to publish. ","permalink":"https://dec0dedd.github.io/posts/q-learning-from-scratch/","summary":"Building tabular Q-learning from the ground up to actually understand the update rule.","title":"Q-Learning From Scratch"}]