引言 / Introduction
This chapter considers policy gradient methods — methods that learn a parameterized policy π(a|s,θ) that can select actions without consulting a value function. Updates approximate gradient ascent in performance J(θ). Methods learning both policy and value function are called actor-critic methods.
本章考虑策略梯度方法——学习参数化策略π(a|s,θ)的方法，可以不依赖价值函数来选择动作。更新近似性能J(θ)的梯度上升。同时学习策略和价值函数的方法称为actor-critic方法。
θt+1 = θt + α∇̂J(θt)
13.1 策略近似及其优势 / Policy Approximation and its Advantages
The policy π(a|s,θ) must be differentiable w.r.t. θ. For discrete action spaces, the most common parameterization uses soft-max in action preferences:
策略π(a|s,θ)必须对θ可微。对于离散动作空间，最常见的参数化使用动作偏好上的soft-max：
π(a|s,θ) = eh(s,a,θ) / Σb eh(s,b,θ)
where h(s,a,θ) can be linear (θTx(s,a)) or a deep ANN. Advantages over ε-greedy action-value methods:
其中h(s,a,θ)可以是线性的(θTx(s,a))或深度ANN。相比ε-greedy动作价值方法的优势：
- Approaches deterministic policy: preferences can go to ±∞, unlike ε-greedy which always has ε random probability
- Arbitrary probabilities: can learn specific stochastic policies (e.g., bluffing in Poker)
- Simpler function: on some problems policy is simpler to approximate than value function
- Prior knowledge injection: parameterization can encode desired policy structure
- 趋近确定性策略：偏好可趋向±∞，不像ε-greedy总有ε的随机概率
- 任意概率：可学习特定随机策略（如扑克中的虚张声势）
- 更简单的函数：某些问题上策略比值函数更容易近似
- 先验知识注入：参数化可编码期望的策略结构
例13.1：短走廊（Short Corridor）。三个状态看起来相同（函数近似下），动作在中间状态反转。ε-greedy被迫在两个策略间选择（值<-44），而策略方法可学习最优概率≈0.59（值≈-11.6）。
13.2 策略梯度定理 / The Policy Gradient Theorem
The policy gradient theorem provides an analytic expression for the gradient of performance w.r.t. the policy parameter that does not involve the derivative of the state distribution — a remarkable result!
策略梯度定理提供了性能对策略参数梯度的解析表达式，不涉及状态分布的导数——一个非凡的结果！
∇J(θ) ∝ Σs μ(s) Σa qπ(s,a) ∇π(a|s,θ)
For episodic case: J(θ) = vπθ(s0). The key insight: even though policy changes affect the state distribution (which is unknown), the gradient expression only involves the on-policy distribution μ(s) and can be estimated by sampling. For continuing case: J(θ) = r(π), and the relationship is an exact equality.
对于回合制：J(θ)=vπθ(s0)。关键洞察：尽管策略变化影响状态分布（未知），梯度表达式只涉及on-policy分布μ(s)且可通过采样估计。对于持续性：J(θ)=r(π)，关系为精确等式。
13.3 REINFORCE：蒙特卡罗策略梯度
From the policy gradient theorem, convert the sum over states to an expectation, then sample. The REINFORCE update (Williams, 1992):
从策略梯度定理，将状态求和转为期望，然后采样。REINFORCE更新(Williams, 1992)：
θt+1 = θt + α Gt · ∇π(At|St,θt) / π(At|St,θt)
= θt + α Gt · ∇ln π(At|St,θt)
The vector ∇ln π is called the eligibility vector. The update is proportional to return Gt and inversely proportional to action probability — favors high-return actions while correcting for frequency bias. Uses complete return → Monte Carlo method, episodic only.
向量∇lnπ称为资格向量。更新与回报Gt成正比，与动作概率成反比——偏好高回报动作同时修正频率偏差。使用完整回报→蒙特卡罗方法，仅限回合制。
REINFORCE (episodic) for π*
────────────────────────────────────────────
Input: differentiable π(a|s,θ)
Parameter: α > 0
Initialize θ ∈ ℝd'
Loop for each episode:
Generate episode S₀,A₀,R₁,...,ST-1,AT-1,RT following π
For t = 0, 1, ..., T-1:
G ← Σ γk-t-1 Rk (k=t+1 to T)
θ ← θ + α γt G ∇ln π(At|St, θ)
For soft-max with linear preferences h(s,a,θ)=θTx(s,a), the eligibility vector is: ∇ln π(a|s,θ) = x(s,a) − Σb π(b|s,θ)x(s,b).
对于线性偏好的soft-max h(s,a,θ)=θTx(s,a)，资格向量为：∇lnπ(a|s,θ)=x(s,a)−Σbπ(b|s,θ)x(s,b)。
13.4 带基线的REINFORCE / REINFORCE with Baseline
The policy gradient theorem generalizes to include a baseline b(s) — any function not varying with a. The baseline leaves the expected update unchanged but can dramatically reduce variance:
策略梯度定理泛化为包含基线b(s)——不随a变化的任意函数。基线不改变更新期望但可显著降低方差：
θt+1 = θt + α (Gt − b(St)) · ∇ln π(At|St, θt)
Natural choice: b(s) = v̂(s, w), a learned state-value function. The baseline helps differentiate actions in states where all actions have similar values. Two step sizes needed: αθ for policy, αw for value function.
自然选择：b(s)=v̂(s,w)，一个学习的状态价值函数。基线帮助在所有动作价值相似的状态中区分动作。需要两个步长：αθ用于策略，αw用于价值函数。
REINFORCE with Baseline (episodic)
────────────────────────────────────────────
Input: differentiable π(a|s,θ), v̂(s,w)
Parameters: αθ > 0, αw > 0
Initialize θ, w
Loop for each episode:
Generate episode following π(·|·,θ)
For t = 0, 1, ..., T-1:
G ← Σ γk-t-1 Rk
δ ← G − v̂(St, w)
w ← w + αw δ ∇v̂(St, w)
θ ← θ + αθ γt δ ∇ln π(At|St, θ)
13.5 Actor-Critic方法 / Actor-Critic Methods
Actor-critic methods use the value function for bootstrapping (not just as baseline). This introduces bias but reduces variance and enables online learning. One-step actor-critic replaces Gt with the one-step return:
Actor-critic方法将价值函数用于自举（不仅是基线）。这引入偏差但降低方差并实现在线学习。单步actor-critic用单步回报替代Gt：
θt+1 = θt + α [Rt+1 + γv̂(St+1,w) − v̂(St,w)] · ∇ln π(At|St,θt)
= θt + α δt · ∇ln π(At|St,θt)
One-step Actor-Critic (episodic)
────────────────────────────────────────────
Input: differentiable π(a|s,θ), v̂(s,w)
Parameters: αθ, αw > 0
Initialize θ, w
Loop for each episode:
Initialize S, I ← 1
Loop while S not terminal:
A ~ π(·|S, θ)
Take A, observe S', R
δ ← R + γv̂(S', w) − v̂(S, w)
w ← w + αw δ ∇v̂(S, w)
θ ← θ + αθ I δ ∇ln π(A|S, θ)
I ← γI, S ← S'
Actor-Critic with Eligibility Traces: use separate traces for actor (zθ) and critic (zw). The actor trace accumulates ∇ln π; the critic trace accumulates ∇v̂. Both decay with their own λ parameters.
带资格迹的Actor-Critic：为actor(zθ)和critic(zw)使用独立的迹。Actor迹累积∇lnπ；critic迹累积∇v̂。两者以各自的λ参数衰减。
13.6 持续性问题的策略梯度 / Policy Gradient for Continuing Problems
For continuing problems: J(θ) = r(π) = average reward rate. Values defined w.r.t. differential return:
对于持续性问题：J(θ)=r(π)=平均奖励率。价值关于差分回报定义：
Gt = (Rt+1 − r̄) + (Rt+2 − r̄) + (Rt+3 − r̄) + ···
The policy gradient theorem holds with equality (not just proportionality). TD error: δt = Rt+1 − R̄t+1 + v̂(St+1,w) − v̂(St,w). Average reward R̄ updated with step size αR̄.
策略梯度定理以等式成立（不仅是正比）。TD误差：δt=Rt+1−R̄t+1+v̂(St+1,w)−v̂(St,w)。平均奖励R̄用步长αR̄更新。
Actor-Critic (continuing, backward view)
────────────────────────────────────────────
Parameters: λw, λθ ∈ [0,1], αw, αθ, αR̄ > 0
Initialize R̄, w, θ, S
zw ← 0, zθ ← 0
Loop forever:
A ~ π(·|S, θ)
Take A, observe S', R
δ ← R − R̄ + v̂(S', w) − v̂(S, w)
R̄ ← R̄ + αR̄ δ
zw ← γλwzw + ∇v̂(S, w)
zθ ← γλθzθ + ∇ln π(A|S, θ)
w ← w + αw δ zw
θ ← θ + αθ δ zθ
S ← S'
13.7 连续动作的策略参数化 / Policy Parameterization for Continuous Actions
For continuous action spaces, learn statistics of the probability distribution. Using a Gaussian policy:
对于连续动作空间，学习概率分布的统计量。使用高斯策略：
π(a|s,θ) = 1/(σ(s,θ)√(2π)) · exp(−(a − μ(s,θ))² / (2σ(s,θ)²))
where μ(s,θ) = θμTxμ(s) and σ(s,θ) = exp(θσTxσ(s)). The eligibility vector has two parts:
其中μ(s,θ)=θμTxμ(s)，σ(s,θ)=exp(θσTxσ(s))。资格向量有两部分：
∇θμ ln π = (a − μ)/σ² · xμ(s)
∇θσ ln π = [(a − μ)²/σ² − 1] · xσ(s)
13.8 总结 / Summary
Policy gradient methods learn parameterized policies without consulting value functions for action selection. Key advantages:
策略梯度方法学习参数化策略，动作选择不依赖价值函数。关键优势：
- Learn specific action probabilities (stochastic optimal policies)
- Approach deterministic policies asymptotically
- Naturally handle continuous action spaces
- Policy gradient theorem: exact gradient formula without state distribution derivatives
- On some problems, policy is simpler to approximate than value function
- 学习特定动作概率（随机最优策略）
- 渐近趋近确定性策略
- 自然处理连续动作空间
- 策略梯度定理：不涉及状态分布导数的精确梯度公式
- 某些问题上策略比值函数更易近似
REINFORCE follows directly from the policy gradient theorem (MC). Adding a baseline reduces variance without bias. Using value function for bootstrapping (actor-critic) introduces bias but substantially reduces variance. The critic criticizes the actor's choices; the actor improves accordingly.
REINFORCE直接源于策略梯度定理(MC)。添加基线降低方差而不引入偏差。将价值函数用于自举(actor-critic)引入偏差但显著降低方差。Critic评价actor的选择；actor据此改进。
📋 核心概念速查 / Key Concepts Summary
| 概念 / Concept | 定义 / Definition |
|---|
| 策略梯度方法 | 通过近似性能对策略参数的梯度上升来更新参数化策略 |
| 策略梯度定理 | ∇J(θ)∝Σμ(s)Σaqπ(s,a)∇π(a|s,θ)，不涉及状态分布导数 |
| REINFORCE | 蒙特卡罗策略梯度：θ←θ+αGt∇lnπ(At|St,θ) |
| 资格向量 | ∇lnπ(a|s,θ)=∇π/π，策略参数化的唯一出处 |
| 基线 Baseline | 任意不随a变化的b(s)，降低方差不改变期望 |
| Actor-Critic | Actor=学习策略，Critic=学习价值函数（用于自举） |
| Soft-max偏好 | π(a|s,θ)=eh(s,a,θ)/Σeh(s,b,θ)，可趋近确定性 |
| 高斯策略 | 连续动作：π为高斯分布，参数μ(s,θ)和σ(s,θ)由函数近似器给出 |
| 差分回报 | 持续性问题：Gt=Σ(Rk−r(π))，替代折扣回报 |