how to

4.5.weight-decay

Aug 6, 2024
notesjulyfun技术学习d2l
1 Minutes
109 Words

默认情况下,PyTorch 同时衰减权重和偏移

可以手动设置不衰减偏置:

1
def train_concise(wd):
2
net = nn.Sequential(nn.Linear(num_inputs, 1))
3
for param in net.parameters():
4
param.data.normal_()
5
loss = nn.MSELoss(reduction='none')
6
num_epochs, lr = 100, 0.003
7
# 偏置参数没有衰减
8
trainer = torch.optim.SGD([
9
{"params":net[0].weight,'weight_decay': wd},
10
{"params":net[0].bias}], lr=lr)
11
animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log',
12
xlim=[5, num_epochs], legend=['train', 'test'])
13
for epoch in range(num_epochs):
14
for X, y in train_iter:
15
trainer.zero_grad()
8 collapsed lines
16
l = loss(net(X), y)
17
l.mean().backward()
18
trainer.step()
19
if (epoch + 1) % 5 == 0:
20
animator.add(epoch + 1,
21
(d2l.evaluate_loss(net, train_iter, loss),
22
d2l.evaluate_loss(net, test_iter, loss)))
23
print('w的L2范数:', net[0].weight.norm().item())
Article title:4.5.weight-decay
Article author:Julyfun
Release time:Aug 6, 2024
Copyright 2025
Sitemap