多样性、规则与 DPP
Goal
把概念变成可检查的输入、运算、梯度与对照。
数据类型：真实向量 + 数值验收。本地教学参数不是生产默认值；机制实验不是原论文完整复现。
Setup
每本首次从 Setup 起顺序运行，后面的格会复用前面的变量。之后只改一个参数，重跑对应格及下游格。不要把构造数据当作真实业务收益。
PYTHON · 01
from pathlib import Path
import sys
import inspect
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display, Code
PROJECT = next(p for p in [Path.cwd(), *Path.cwd().parents]
if (p / "reclab/models.py").exists())
if str(PROJECT) not in sys.path:
sys.path.insert(0, str(PROJECT))
from reclab.data import Dataset
from reclab.paths import ARTIFACTS, PROCESSED, read_json
plt.rcParams.update({"figure.figsize": (9, 3.8), "font.size": 11,
"axes.spines.top": False, "axes.spines.right": False,
"axes.labelcolor": "#182b2b", "text.color": "#182b2b",
"axes.facecolor": "white", "figure.facecolor": "white",
"grid.color": "#d4ddda", "axes.axisbelow": True})
dataset = Dataset()
print("Project:", PROJECT)
print("Dataset version:", dataset.manifest["dataset_version"])
print("Python:", sys.executable)
assert Path(sys.executable).parent == PROJECT / ".venv/bin", "请选择 Recommendation Lab 内核"Notebook 保存的输出 · 本页未执行
Project: /Users/lizuolin/local/recommendation-lab Dataset version: 2772cab0aca8cb81 Python: /Users/lizuolin/local/recommendation-lab/.venv/bin/python
Steps
讲义对应：06_Rerank_01.pdf, 06_Rerank_02.pdf, 06_Rerank_03.pdf, 06_Rerank_04.pdf, 06_Rerank_05.pdf
真实向量上的 ILD、类目熵与 MMR
ILD 衡量协同向量距离，不等同于独立图文语义。列表变化不是用户反馈。
先预测： 同类目多不一定意味着向量相似吗？
PYTHON · 02
from reclab.learning.systems import ild,category_entropy,constrained_ranking,greedy_dpp,slow_greedy_dpp,exhaustive_dpp
from reclab.retrieval import mmr
vectors=np.load(ARTIFACTS/"runs/baseline-history32/item_vectors.npy")
scores=vectors@vectors[100];pool=np.argsort(-scores[1:])[:80]+1
rows=[]
for lam in [1.,.8,.5]:
selected,_=mmr(pool.tolist(),scores,vectors,k=10,relevance=lam)
rows.append({"lambda":lam,"mean_similarity":float(scores[selected].mean()),
"ILD":ild(vectors,selected),"category_entropy_bits":category_entropy([dataset.items[i-1]["category"] for i in selected])})
display(pd.DataFrame(rows))Notebook 保存的输出 · 本页未执行
| lambda | mean_similarity | ILD | category_entropy_bits | |
|---|---|---|---|---|
| 0 | 1.0 | 0.819358 | 0.191529 | 0.921928 |
| 1 | 0.8 | 0.819279 | 0.198008 | 0.921928 |
| 2 | 0.5 | 0.808305 | 0.277409 | 1.770951 |
规则与不可行约束
若没有可行候选，本实现明确返回部分结果，不伪造满足规则的完整列表。
先预测： 候选全部来自一个类目时，算法应该放松规则还是少返回？
PYTHON · 03
categories={i:dataset.items[i-1]["category"] for i in pool}
selected=constrained_ranking(pool.tolist(),categories,k=10,max_consecutive=2)
display(pd.DataFrame({"item":selected,"category":[categories[i] for i in selected]}))
for pos in range(2,len(selected)):
assert not (categories[selected[pos]]==categories[selected[pos-1]]==categories[selected[pos-2]])Notebook 保存的输出 · 本页未执行
| item | category | |
|---|---|---|
| 0 | 100 | 819 |
| 1 | 4153 | 819 |
| 2 | 1172 | 405 |
| 3 | 4126 | 532 |
| 4 | 3095 | 819 |
| 5 | 4311 | 819 |
| 6 | 1537 | 1445 |
| 7 | 3424 | 819 |
| 8 | 398 | 819 |
| 9 | 4520 | 1593 |
DPP：体积、慢速贪心、快速 Cholesky 和穷举
受控随机向量，便于数值验收。贪心 MAP 不是全局最优保证。
先预测： 把两个候选向量设成相同，Gram 行列式会怎样变化？
PYTHON · 04
rng=np.random.default_rng(42)
v=rng.normal(size=(8,8));v/=np.linalg.norm(v,axis=1,keepdims=True)
quality=np.exp(np.linspace(0,1,8));features=v*quality[:,None];kernel=features@features.T
assert np.linalg.eigvalsh(kernel).min()>-1e-8
orthogonal=np.eye(2);duplicate=np.array([[1.,0.],[1.,0.]])
print("Orthogonal/duplicate volume squared:",np.linalg.det(orthogonal@orthogonal.T),np.linalg.det(duplicate@duplicate.T))
display(Code(inspect.getsource(greedy_dpp),language="python"))
fast=greedy_dpp(kernel,3);slow=slow_greedy_dpp(kernel,3)
best_logdet,best_ids=exhaustive_dpp(kernel,3)
assert fast==slow
greedy_logdet=np.linalg.slogdet(kernel[np.ix_(fast,fast)])[1]
assert greedy_logdet<=best_logdet+1e-8
display(pd.DataFrame([{"method":"greedy","items":fast,"logdet":greedy_logdet},{"method":"global exhaustive","items":best_ids,"logdet":best_logdet}]))Notebook 保存的输出 · 本页未执行
Orthogonal/duplicate volume squared: 1.0 0.0
def greedy_dpp(kernel, k):
"""Incremental Cholesky greedy MAP. It is an approximation to global MAP."""
n = len(kernel)
k = min(k, n)
residual = kernel.diagonal().copy()
coefficients = np.zeros((k, n))
selected = []
for step in range(k):
item = int(np.argmax(residual))
if residual[item] <= 1e-12:
break
correction = coefficients[:step, item] @ coefficients[:step, :]
update = (kernel[item, :] - correction) / math.sqrt(residual[item])
coefficients[step, :] = update
residual = np.maximum(residual - update**2, 0)
selected.append(item)
residual[selected] = -np.inf
return selected| method | items | logdet | |
|---|---|---|---|
| 0 | greedy | [7, 6, 5] | 4.680314 |
| 1 | global exhaustive | (5, 6, 7) | 4.680314 |
Checks
所有断言和输出都可回溯到代码。它们验证算子、数据快照或机制，不是线上收益。
Next Steps
先记录预测，只改一个变量，再运行对应实验格。需要查看辅助函数时，打开 reclab/learning/。