285-0 - How to?

flowchart TD
    PG["Vanilla Policy Gradient
[on-policy]"] -->|"Add V(s)"| AC
    AC["Actor-Critic
[on-policy]
·MC: Σ(γ^t r_t) - V(s)
·单步 TD: r + γ*V(s+1) - V(s)
·GAE"]
    PG -->|"mean_net + logstd 计算概率密度"| cont[支持连续动作]
    AC -->|"1: remove V and actor
2: add Q(s, a); a = argmax
3: ε-greedy
4) replay buffer 采样"| Q-learning["Q-learning
[off-policy, discrete]
但这玩意儿不稳定"]
    Q-learning -->|"Q_target"| DQN
    DQN -->|"计算 label Q 时改用 Q_online 来选 action"| Double-DQN
    Double-DQN -->|"1: Add actor，loss = -Q(actor(s))
2: entropy loss 替代 ε
3: 悲观估计"| SAC["Soft Actor-Critic
（DDPG 则是离散的）"]
    SAC -->|"1: add V(s)，使用 expectile loss
2: use V(s) in critic_loss
 and actor_loss"| IQL["Implicit Q-learning
[offline]"]
    AC -->|"1: IS clipped[0.8, 1.2]
2: Entropy loss
3: KL(π_old ‖ π_new)
4: replay buffer 带概率"| PPO["PPO
[off-policy]"]
    PG -->|"rollout N times. advantage = score - mean(score)"| GR["GR-REINFORCE
on-policy，可以 kl loss"]
    GR -->|"PPO IS clipped"| GRPO

Utils

https://rail.eecs.berkeley.edu/deeprlcourse/

伪代码

以下的 loss 省略 mean().
critic 指的是 Q(s, a)，不是指 V(s).

PG: on-policy, 离散和连续动作均可

1
s, a, r, nxt_s, done = rollout()
2
actor_loss = -actor(s).log_prob(a) * reward_to_go(r)
3
#             |<---------------->|
4
#      Gradient increases prob of good actions.

AC (baseline): on-policy

好处：降低优势方差. 这里没有 Q_net 而是用 V_net.

1
s, a, r, nxt_s, done = rollout()
2
value_net_loss = mse(
3
    value_net(s),
4
    reward_to_go(r)
5
)
6

7
actor_loss = -actor(s).log_prob(a) * (reward_to_go(r) - value_net(s).detach())

Double-DQN: off-policy, discrete-only

没有 actor. 直接 argmax Q.
好处：通过 off-policy 大幅提升 sample efficiency.

1
s, a, r, nxt_s, done = replaybuffer.sample()
2
critic_loss = mse(
3
    online_critic(s, a),
4
    [no grad] r + gamma * target_critic(nxt_s)[argmax_a online_critic(nxt_s, all a) * (1 - done)])
5
)

其中离散 critic: s -> value[b, num_actions] 输出所有 action 的价值.

SAC: off-policy

重新引入 actor 输出 distribution. 好处：支持连续动作.

1
s, a, r, nxt_s, done = replaybuffer.sample()
2
critic_loss = mse(
3
    online_critic(s, a),
4
    [no_grad] r + gamma * target_critic(nxt_s, actor(nxt_s).sample()) * (1 - done)
5
    #                                          |<-------->|
6
    #                                       this is a distribution
7
)
8
actor_loss = -online_critic(s, actor(s).rsample()) - entropy(actor(s)) * temperature
9
#             |<--------->|
10
#             critic 传播梯度，但不在 actor optimizer 中所以不会被更新.

其中连续 critic: s, a -> value[b,]，输出给定 action 价值
连续 actor: s -> a[b, action_dim][dtype=Distribution]

Implicit Q-learning: offline

有 Q, V, actor.
动机：由于 SAC critic 和 actor 均依赖下一步模拟执行（即贝尔曼方程），即 critic(s, actor(s).rsample())，在仅有静态数据集的情况下，rsample() 可能产生远离静态数据集的数据，优势估计通常偏大. 这里训练 value_net 的目的就是更换 critic_loss 和 actor_loss，但是 value_net 只能使用静态数据集.
为什么需要高 expectile：AC 等算法中 V(s) 和 critic 是 conditioned on current actor 或者 best actor possible. 而 IQL 禁止用 actor 采样，因此 V(s) 应当 conditioned on somewhat good actor.

1
s, a, r, nxt_s, done = replaybuffer.sample()
2
diff = target_critic(s, a).detach() - value_net(s) # make this smaller, but with weight.
3
value_net_loss = where(diff > 0, 0.7, 0.3) * diff**2
4
# 解释：如果 where(diff>0,0.5,0.5) 则 value_net 收敛到所有动作 Q 值的平均值; 如果是 0.7 则收敛到 Q 的近似最大值.
5
# e.g. 考虑同一个状态 s 的多个 a 价值分别是 1, 2, 9，expectile loss 更倾向于拟合 9.
6
# expectile 关心的是上下两边的平方误差力矩怎么平衡.
7

8
critic_loss = mse(
9
    online_critic(s, a),
10
    [no_grad] r + gamma * value_net(nxt_s) * (1 - done)
11
)
12

13
actor_loss = -exp(beta * (target_critic(s, a) - value_net(s)).detach()) * actor(s).log_prob(a)
14
#             |<---------- advantage weight, no grad ---------------->|
15
#                advantage 截断到 100. 动作优势大就提升数据集中该动作的概率.

actor loss 形式似乎启发式，并且回到了 PG-like.