285-0 - How to?

flowchart TD
    PG["Vanilla Policy Gradient<br>[on-policy]"] -->|"Add V(s)"| AC
    AC["Actor-Critic<br>[on-policy]<br>·MC: Σ(γ^t r_t) - V(s)<br>·单步 TD: r + γ*V(s+1) - V(s)<br>·GAE"]
    PG -->|"mean_net + logstd 计算概率密度"| cont[支持连续动作]
    AC -->|"1: remove V and actor<br>2: add Q(s, a); a = argmax<br>3: ε-greedy<br>4) replay buffer 采样"| Q-learning["Q-learning<br>[off-policy, discrete]<br>但这玩意儿不稳定"]
    Q-learning -->|"Q_target"| DQN
    DQN -->|"计算 label Q 时改用 Q_online 来选 action"| Double-DQN["Double-DQN<br>[off-policy, no actor]"]
    Double-DQN -->|"1: Add actor，loss = -Q(actor(s))<br>2: entropy loss 替代 ε<br>3: 悲观估计"| SAC["Soft Actor-Critic<br>（DDPG 则是离散的）"]
    SAC -->|"1: add V(s)，使用 expectile loss<br>2: use V(s) in critic_loss<br> and actor_loss"| IQL["Implicit Q-learning<br>[offline]"]
    AC -->|"1: IS clipped[0.8, 1.2]<br>2: Entropy loss<br>3: KL(π_old ‖ π_new)<br>4: replay buffer 需存概率"| PPO["PPO<br>[on-policy, epoch>1]"]
    PG -->|"rollout N times. advantage = score - mean(score)"| GR["GR-REINFORCE<br>on-policy，可以 kl loss"]
    GR -->|"PPO IS clipped"| GRPO

Utils

https://rail.eecs.berkeley.edu/deeprlcourse/

伪代码

以下的 loss 省略 mean().
critic 指的是 Q(s, a)，不是指 V(s).

PG: on-policy, 离散和连续动作均可

1
s, a, r, nxt_s, done = rollout()
2
actor_loss = -actor(s).log_prob(a) * reward_to_go(r)
3
#             |<---------------->|
4
#      Gradient increases prob of good actions.

AC (baseline): on-policy

好处：降低优势方差. 这里没有 Q_net 而是用 V_net.

1
s, a, r, nxt_s, done = rollout() # 通过 Gaussian-policy 支持连续动作
2
value_net_loss = mse(
3
    value_net(s),
4
    reward_to_go(r)
5
)
6

7
actor_loss = -actor(s).log_prob(a) * (reward_to_go(r) - value_net(s).detach())
8
#                                    |<------------------------------------->|
9
#                                                    advantage

Double-DQN: off-policy, discrete-only

没有 actor. 直接 argmax Q.
好处：通过 off-policy 大幅提升 sample efficiency.

1
s, a, r, nxt_s, done = replaybuffer.sample()
2
critic_loss = mse(
3
    online_critic(s, a),
4
    [no grad] r + gamma * target_critic(nxt_s)[argmax_a online_critic(nxt_s, all a) * (1 - done)])
5
)

其中离散 critic: s -> value[b, num_actions] 输出所有 action 的价值.

SAC: off-policy

重新引入 actor 输出 distribution. 好处：支持连续动作.

1
s, a, r, nxt_s, done = replaybuffer.sample()
2
critic_loss = mse(
3
    online_critic(s, a),
4
    [no_grad] r + gamma * target_critic(nxt_s, actor(nxt_s).sample()) * (1 - done)
5
    #                                          |<-------->|
6
    #                                       this is a distribution
7
)
8
actor_loss = -online_critic(s, actor(s).rsample()) - entropy(actor(s)) * temperature
9
#             |<--------->|
10
#             critic 传播梯度，但不在 actor optimizer 中所以不会被更新.

其中连续 critic: s, a -> value[b,]，输出给定 action 价值
连续 actor: s -> a[b, action_dim][dtype=Distribution]

TD3

off policy. 相比 SAC, 将 actor 改为确定性输出，并在计算目标 Q 值时给动作加截断噪声平滑估计；actor 延迟更新。

1
s, a, r, nxt_s, done = replaybuffer.sample()
2
with no_grad:
3
    nxt_a = clip(target_actor(nxt_s) + clipped_noise())
4
    y = r + gamma * (1 - done) * min(target_Q1(nxt_s, nxt_a), target_Q2(nxt_s, nxt_a))
5

6
critic_loss = mse(Q1(s, a), y) + mse(Q2(s, a), y)
7
# 每步更新 critic；每隔两步更新 actor 和所有 target
8
actor_loss = -Q1(s, actor(s)).mean()

Implicit Q-learning: offline

有 Q, V, actor.
动机：由于 SAC critic 和 actor 均依赖下一步模拟执行（即贝尔曼方程），即 critic(s, actor(s).rsample())，在仅有静态数据集的情况下，rsample() 可能产生远离静态数据集的数据，优势估计通常偏大. 这里训练 value_net 的目的就是更换 critic_loss 和 actor_loss，但是 value_net 只能使用静态数据集.
为什么需要高 expectile：AC 等算法中 V(s) 和 critic 是 conditioned on current actor 或者 best actor possible. 而 IQL 禁止用 actor 采样，因此 V(s) 应当 conditioned on somewhat good actor.

1
1. s, a, r, nxt_s, done = replaybuffer.sample()
2
diff = target_critic(s, a).detach() - value_net(s) # make this smaller, but with weight.
3
value_net_loss = where(diff > 0, 0.7, 0.3) * diff**2
4
# 解释：如果 where(diff>0,0.5,0.5) 则 value_net 收敛到所有动作 Q 值的平均值; 如果是 0.7 则收敛到 Q 的近似最大值.
5
# e.g. 考虑同一个状态 s 的多个 a 价值分别是 1, 2, 9，expectile loss 更倾向于拟合 9.
6
# GPT 说道：expectile 关心的是上下两边的平方误差力矩怎么平衡.
7

8
2. critic_loss = mse(
9
    online_critic(s, a),
10
    [no_grad] r + gamma * value_net(nxt_s) * (1 - done)
11
)
12

13
3. actor_loss = -exp(beta * (target_critic(s, a) - value_net(s)).detach()) * actor(s).log_prob(a)
14
#             |<---------- advantage weight, no grad ---------------->|
15
#                     截断到 100. 动作优势大就提升数据集中该动作的概率.

actor loss 形式似乎启发式，并且回到了 PG-like.

FQL: offline

1
1. 纯模仿学习 flow_matching，同时进行如下:
2

3
2. 和 SAC 一样训练 critic(s, a).
4

5
3. noise = randn()
6
采样 obs, action.
7
action = one_step_actor(obs, noise)
8
distill_loss = mse(action, flow_matching(obs, noise)) # 必须是同一个 noise
9
q_loss = -critic(obs, action)
10

11
推理用 one_step_actor.

你可能会问：为什么还需要一个 flow_matching 来 distill 呢，fm 本来不就纯模仿数据集吗，可以不可以直接 mse(数据集)？答案是不行，one_step_actor 需要模仿 noise 条件下的 flow matching，教程说是为了增强 expressiveness.