LastN, DIN, and SIM
Goal
Turn concepts into inspectable inputs, operations, gradients, and comparisons.
Evidence: real histories + teaching attention variants. Local teaching parameters are not production defaults; mechanism experiments are not full reproductions of the original papers.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
Setup
On the first pass, run from Setup in order: later cells reuse earlier variables. Then change one parameter and rerun that cell and its downstream cells. Constructed data does not establish real business impact.
from pathlib import Path
import sys
import inspect
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display, Code
PROJECT = next(p for p in [Path.cwd(), *Path.cwd().parents]
if (p / "reclab/models.py").exists())
if str(PROJECT) not in sys.path:
sys.path.insert(0, str(PROJECT))
from reclab.data import Dataset
from reclab.paths import ARTIFACTS, PROCESSED, read_json
plt.rcParams.update({"figure.figsize": (9, 3.8), "font.size": 11,
"axes.spines.top": False, "axes.spines.right": False,
"axes.labelcolor": "#182b2b", "text.color": "#182b2b",
"axes.facecolor": "white", "figure.facecolor": "white",
"grid.color": "#d4ddda", "axes.axisbelow": True})
dataset = Dataset()
print("Project:", PROJECT)
print("Dataset version:", dataset.manifest["dataset_version"])
print("Python:", sys.executable)
assert Path(sys.executable).parent == PROJECT / ".venv/bin", "请选择 Recommendation Lab 内核"Saved notebook output · not executed on this page
Project: /Users/lizuolin/local/recommendation-lab Dataset version: 2772cab0aca8cb81 Python: /Users/lizuolin/local/recommendation-lab/.venv/bin/python
Steps
Lecture references: 05_LastN_01.pdf, 05_LastN_02.pdf, 05_LastN_03.pdf
Real LastN histories and mean pooling
Saved item vectors at the training cutoff diagnose representations; they do not evaluate prediction gains on earlier samples.
Predict first: Why is the mean unchanged by reversal? What if the truncation window changes?
import torch
from reclab.learning.networks import TargetAttention
torch.manual_seed(42);torch.set_num_threads(4)
vectors=torch.tensor(np.load(ARTIFACTS/"runs/baseline-history32/item_vectors.npy"))
user=max(dataset.histories,key=lambda u:len(dataset.histories[u]))
ids=dataset.histories[user][-min(1000,len(dataset.histories[user])):]
history=vectors[ids][None,:,:];mask=torch.ones(1,len(ids),dtype=torch.bool)
assert torch.allclose(history.mean(1),history.flip(1).mean(1),atol=1e-6)
print("Real history length:",len(ids),"user:",dataset.users[user])Saved notebook output · not executed on this page
Real history length: 1000 user: 1150086
DIN-style candidate-dependent activation
Inputs are h, q, h-q, and h*q. This is a softmax teaching variant without Dice or the original regularization. Random weights do not represent learned interests.
Predict first: Why does the same user's representation change with the candidate? Can it still be cached as an independent user tower?
attention=TargetAttention(vectors.shape[1])
short=history[:,-40:];short_mask=mask[:,-40:]
qa=vectors[100][None,:];qb=vectors[2000][None,:]
pa,wa=attention(short,qa,short_mask);pb,wb=attention(short,qb,short_mask)
assert torch.allclose(wa.sum(-1),torch.ones(1))
display(Code(inspect.getsource(TargetAttention),language="python"))
plt.plot(wa[0].detach(),label="candidate100",color="#006e60")
plt.plot(wb[0].detach(),label="candidate2000",color="#855519",linestyle="--")
plt.xlabel("History position");plt.ylabel("Attention weight");plt.title("Untrained activation / candidate sensitivity");plt.legend();plt.show()Saved notebook output · not executed on this page
class TargetAttention(nn.Module):
"""DIN-style candidate-dependent activation, softmax-normalized teaching variant."""
def __init__(self, dim, hidden=16):
super().__init__()
self.activation = nn.Sequential(
nn.Linear(dim * 4, hidden), nn.ReLU(), nn.Linear(hidden, 1)
)
def forward(self, history, candidate, mask):
query = candidate.unsqueeze(1).expand_as(history)
logits = self.activation(
torch.cat([history, query, history - query, history * query], -1)
).squeeze(-1)
weights = logits.masked_fill(~mask, -1e9).softmax(-1) * mask
weights = weights / weights.sum(-1, keepdim=True).clamp_min(1e-8)
return (history * weights.unsqueeze(-1)).sum(1), weights<Figure size 900x380 with 1 Axes>
Actually backpropagate through the activation unit
An artificial vector-fitting task checks learnability, not CTR labels.
Predict first: Does a weight change mean real users are more interested? What evidence is missing?
optimizer=torch.optim.Adam(attention.parameters(),lr=.02);curve=[]
for _ in range(40):
optimizer.zero_grad();pooled,_=attention(short,qa,short_mask)
loss=torch.nn.functional.mse_loss(pooled,qa)
loss.backward();optimizer.step();curve.append(float(loss.detach()))
assert curve[-1]<curve[0]
print("Artificial vector-fit loss:",curve[0],"->",curve[-1])Saved notebook output · not executed on this page
Artificial vector-fit loss: 0.04444105550646782 -> 0.019344493746757507
SIM-style retrieval before activation
Select top K from real long histories by category and similarity. This illustrates decoupled retrieval and activation, not SIM's complete training or reported performance.
Predict first: Must computation and representation error change monotonically as K increases?
K=20;candidate_id=100;candidate=vectors[candidate_id][None,:]
category=dataset.items[candidate_id-1]["category"]
pool=[i for i in ids if dataset.items[i-1]["category"]==category] or ids
pool_vectors=vectors[pool]
selected=torch.topk(pool_vectors@candidate[0],min(K,len(pool))).indices
filtered=pool_vectors[selected][None,:,:]
with torch.no_grad():
full,_=attention(history,candidate,mask)
reduced,_=attention(filtered,candidate,torch.ones(1,filtered.shape[1],dtype=torch.bool))
print("Full/filtered tokens:",len(ids),filtered.shape[1])
print("Representation distance (not accuracy):",float(torch.norm(full-reduced)))
assert filtered.shape[1]<=KSaved notebook output · not executed on this page
Full/filtered tokens: 1000 1 Representation distance (not accuracy): 1.0099929571151733
Checks
Every assertion and output is traceable to code. They check operators, data snapshots, or mechanisms, not online gains.
Next steps
Record your prediction, change only one variable, and run the relevant experiment cell. Open reclab/learning/ to inspect helper functions.