32 · Computation graphs, the chain rule, and gradients
Foundations path: 31 → 32 → 33 → 34, then choose search, ads, or recommendation.
Scope: constructed small-array mechanism experiments on CPU; no real logs, formal model training, or business-impact claims.
For each section: predict → change one variable → run → explain. Execute from the beginning first; after edits, rerun affected downstream cells.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
Construct a scalar computation graph
z=wx+b, L=(z-y)^2. Calculate dL/dw=2(z-y)x by hand first. All values use float64.
import torch
import matplotlib.pyplot as plt
x, y = 2., 1.
w = torch.tensor(0.5, dtype=torch.float64, requires_grad=True)
b = torch.tensor(0.2, dtype=torch.float64, requires_grad=True)
z = w*x+b
loss = (z-y)**2
loss.backward()
assert torch.allclose(w.grad, 2*(z.detach()-y)*x)
print('z, loss, dw, db:', z.item(), loss.item(), w.grad.item(), b.grad.item())Saved notebook output · not executed on this page
z, loss, dw, db: 1.2 0.03999999999999998 0.7999999999999998 0.3999999999999999
Verify with finite differences
Change only the difference step. Central differences approximate gradients; excessively small steps can cause cancellation error.
def objective(value):
return (value*x+b.detach().item()-y)**2
steps = [1e-2, 1e-5, 1e-8, 1e-12]
estimates = [(objective(w.item()+h)-objective(w.item()-h))/(2*h) for h in steps]
assert abs(estimates[1]-w.grad.item()) < 1e-8
print(list(zip(steps, estimates)))
plt.semilogx(steps, [abs(v-w.grad.item()) for v in estimates], 'o-')
plt.xlabel('finite difference step'); plt.ylabel('absolute error'); plt.show()Saved notebook output · not executed on this page
[(0.01, 0.8000000000000007), (1e-05, 0.7999999999962897), (1e-08, 0.7999999995789153), (1e-12, 0.7999816087345124)]
<Figure size 640x480 with 1 Axes>
Gradient accumulation
Rebuild the same graph and call backward again. Predict whether gradients are automatically cleared.
first = w.grad.clone()
((w*x+b-y)**2).backward()
assert torch.allclose(w.grad, 2*first)
print('累加后的梯度:', w.grad.item())
w.grad = None
b.grad = None
((w*x+b-y)**2).backward()
assert torch.allclose(w.grad, first)Saved notebook output · not executed on this page
累加后的梯度: 1.5999999999999996
Change only learning rate
Start from the same w and fix current b. A descent direction does not guarantee descent for every step size.
rates = [0.05, 1.0]
new_losses = [objective(w.item()-lr*w.grad.item()) for lr in rates]
assert new_losses[0] < loss.item() < new_losses[1]
print('原损失:', loss.item(), '更新后:', dict(zip(rates, new_losses)))Saved notebook output · not executed on this page
原损失: 0.03999999999999998 更新后: {0.05: 0.014400000000000026, 1.0: 1.959999999999999}
Explain
Is a gradient a local slope or a final parameter? Why do training batches usually clear gradients? Does a failed large step disprove the chain rule?
In the next cell, record your prediction, actual result, reason for any difference, and applicability limits. Keep failed results.
reflection = {'预测': '', '结果': '', '解释与边界': ''}
reflectionSaved notebook output · not executed on this page
{'预测': '', '结果': '', '解释与边界': ''}