285 hw3 - How to?

回忆提纲

实现了基于 replaybuffer 的离散 DQN 和连续 SAC. See my code in https://github.com/julyfun/hw285/tree/main/hw3 .

公式参考 https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw3.pdf

2.4: CartPole DQN

回忆提纲

1
# Double DQN
2
s, a, r, nxt_s, done = replaybuffer.sample()
3
critic_loss = mse(
4
    online_critic(s, a),
5
    (no grad) r + gamma * target_critic(nxt_s, argmax_a online_critic(nxt_s, all a) * (1 - done))
6
)

仅支持离散空间，因为没有 actor 我们必须选 action s.t. Q(s, a) 最大，在离散空间，Q_net 直接输出一个数组包含所有 a 的价值即可。

回答问题

• Submit your logs of CartPole-v1, and a plot with environment steps on the x-axis and eval return on the y-axis.

其中 target values 就是 r + gamma * Q(s_next, a_next) 估计，即 Q(s, a) 的拟合目标.

复现

1
conda install -c conda-forge swig
2
# 但是我们并不使用 conda，所以把 conda base bin 加入 PATH.
3

4
uv run src/scripts/run_dqn.py -cfg experiments/dqn/cartpole.yaml --eval_interval 2500

2.5: Double-DQN (Lunar-Lander)

回忆提纲

确实只有一行不同，和 Lec8 伪代码一致。

回答问题

• Run double-Q DQN on LunarLander-v2: You should expect a return of 200 at least once during training. Plot the eval return over training steps.

能看出来确实成功了。

• Run your DQN implementation on the MsPacman-v0 problem. Our default configuration will use double- Q learning by default. You are welcome to tune hyperparameters to get it to work better, but the default parameters should work (so if they don’t, you likely have a bug in your implementation). Your implementation should receive a score of around 1500 at least once during training.

复现

1
uv run src/scripts/run_dqn.py -cfg experiments/dqn/lunarlander.yaml

3.2: Soft Actor-Critic 的 critic (Half-Cheetah, Hopper)

回忆提纲

本节仅实现 SAC 的 critic 部分，任务全部改为连续空间. SAC 的 actor 输出是 mean & logstd 采样. update() 使用的 next_action 不再需要像 DQN 那样让 critic 来选，而是 policy 直接输出并 sample.

1
s, a, r, nxt_s, done = replaybuffer.sample()
2
critic_loss = mse(
3
    online_critic(s, a),
4
    (no_grad) r + gamma * target_critic(nxt_s, actor(nxt_s).sample()) * (1 - done)
5
    #                                          |<-------->|
6
    #                                       this is a distribution
7
)

回答问题

其实此图用了错的代码. 先不更新了.

3.3 用 Entropy 鼓励 policy 的随机性

回忆提纲

相比上面几乎只增加了 next_qs += (self.temperature * next_action_entropy).unsqueeze(0) # (num_critics, b)
- 此处如何利用 action distribution 计算 entropy? 是从 dist 中采样一个 action 然后算 -dist.log_prob(action)，十分暴力. 为啥能这样呢？首先 entropy 的定义是 H = E[ -log pi(a | s) ]，比如动作分布为 0.1, 0.9 比 0.5, 0.5 的熵略高. 虽然对于 Gaussian Policy，输出的 action dist 可以直接计算 H，但是代码中还支持 use_tanh，此时无法直接 dist.entropy().

回答问题

For a tanh-squashed 1D action space, this is approximately log 2 ≈0.69.

3.4 Actor Update with Reparametrization

回忆提纲

1
actor_loss = -online_critic(actor(obs).rsample()) - entropy * temperature
2
# 注：优化器仅带 actor 参数所以 critic 不用冻结.
3
# rsample() 就可以正确传播梯度，sample() 则不行.

回答问题

• Train on HalfCheetah-v4 with halfcheetah.yaml. Plot results with number of environment steps on the x-axis and evaluation return on the y-axis. You should expect to achieve a return of at least 6000 at least once during training. uv run src/scripts/run_sac.py -cfg experiments/sac/halfcheetah.yaml

3.5 Autotune Temperature

回忆提纲

本节目标是让 action distribution entropy 大于等于定值，同时最大化奖励，因此使用 Dual gradient descent

回答问题

• Run auto-tuned SAC on HalfCheetah-v4: uv run src/scripts/run_sac.py -cfg experiments/sac/halfcheetah_autotune.yaml

• Create a figure with two subplots: – Eval return over training for both fixed temperature (from Section 3.4) and auto-tuned temperature – Temperature value (α) over training for the auto-tuned run

temperature 来回变化，ave return 更早突破 1000.

• Answer the following questions: – Does auto-tuning improve or achieve comparable performance to the fixed temperature? – How does the temperature evolve during training? Does it increase, decrease, or s

纠错过程

一开始 loss -= entropy 这里忘记了 * temperature，最终奖励仅到 2500 左右，如上图红色部分.
log_alpha 相关搞错了很多地方，既然变量名为 log_alpha 那么取 temperature 自然需要 log_alpha.exp()，老是忘 exp()
讲义 ∇_α J(α) = E(-alpha * C) 这个公式似乎有问题，应该是 J = E(-alpha * C), 其对 log_alpha 求导仍然是 E(-alpha * C).

3.6 悲观估计 clipped double-Q

回忆提纲

仅需改一行. next_qs = next_qs.mean(dim=0) -> next_qs = next_qs.min(dim=0).values. 使用多个 Q，对 actor 的输出进行估值时直接取 min.

回答问题

• Run single-Q and clipped double-Q on Hopper-v4 using the corresponding configuration files. You should expect the clipped double-Q to achieve a return of at least 1500 at least once during training. Which one works best? Plot the logged eval return from each of them as well as q values. Discuss how these results relate to overestimation bias.

1
uv run src/scripts/run_sac.py -cfg experiments/sac/hopper_singleq.yaml
2
uv run src/scripts/run_sac.py -cfg experiments/sac/hopper_clipq.yaml

使用悲观估计 q values 确实更小了, avg_return 和 std_return 更稳定.

附录

如何使用其他目录的环境运行当前目录：

1
cd hw3
2
source ../hw2/.venv/bin/activate.fish
3
PYTHONPATH=src uv run --active \
4
    src/scripts/run_dqn.py  -cfg experiments/dqn/lunarlander.yaml