基础、指标与 A/B
Goal
把概念变成可检查的输入、运算、梯度与对照。
数据类型：真实数据 + 模拟分桶。本地教学参数不是生产默认值；机制实验不是原论文完整复现。
Setup
每本首次从 Setup 起顺序运行，后面的格会复用前面的变量。之后只改一个参数，重跑对应格及下游格。不要把构造数据当作真实业务收益。
PYTHON · 01
from pathlib import Path
import sys
import inspect
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display, Code
PROJECT = next(p for p in [Path.cwd(), *Path.cwd().parents]
if (p / "reclab/models.py").exists())
if str(PROJECT) not in sys.path:
sys.path.insert(0, str(PROJECT))
from reclab.data import Dataset
from reclab.paths import ARTIFACTS, PROCESSED, read_json
plt.rcParams.update({"figure.figsize": (9, 3.8), "font.size": 11,
"axes.spines.top": False, "axes.spines.right": False,
"axes.labelcolor": "#182b2b", "text.color": "#182b2b",
"axes.facecolor": "white", "figure.facecolor": "white",
"grid.color": "#d4ddda", "axes.axisbelow": True})
dataset = Dataset()
print("Project:", PROJECT)
print("Dataset version:", dataset.manifest["dataset_version"])
print("Python:", sys.executable)
assert Path(sys.executable).parent == PROJECT / ".venv/bin", "请选择 Recommendation Lab 内核"Notebook 保存的输出 · 本页未执行
Project: /Users/lizuolin/local/recommendation-lab Dataset version: 2772cab0aca8cb81 Python: /Users/lizuolin/local/recommendation-lab/.venv/bin/python
Steps
讲义对应：01_Basics_01.pdf, 01_Basics_02.pdf, 01_Basics_03.pdf
指标的分母与推荐漏斗
模拟一份有曝光的日志。真实 Retailrocket 没有完整曝光分母。
先预测： 把购买数除以浏览数和除以曝光数，分别回答什么问题？
PYTHON · 02
impressions, clicks, carts, purchases = 10000, 800, 120, 36
rates = {"CTR (simulated)": clicks/impressions,
"cart given click": carts/clicks, "purchase given cart": purchases/carts,
"purchase per impression": purchases/impressions}
display(pd.Series(rates).to_frame("rate"))
assert np.isclose(rates["CTR (simulated)"]*rates["cart given click"]*rates["purchase given cart"], rates["purchase per impression"])
print("Real event types:", dataset.events.event.unique().tolist())Notebook 保存的输出 · 本页未执行
| rate | |
|---|---|
| CTR (simulated) | 0.0800 |
| cart given click | 0.1500 |
| purchase given cart | 0.3000 |
| purchase per impression | 0.0036 |
Real event types: ['view', 'addtocart', 'transaction']
全词表与采样评估
固定分数的构造案例。候选变容易，不是模型变好。
先预测： 删掉竞争者之后，Recall@10 会怎样变化？
PYTHON · 03
scores = np.arange(100, dtype=float); positive = 50
full_rank = 1 + int((scores > scores[positive]).sum())
sampled = np.r_[np.arange(10), positive]
sampled_rank = 1 + int((scores[sampled] > scores[positive]).sum())
display(pd.DataFrame({"candidates":[100,len(sampled)],"rank":[full_rank,sampled_rank]},index=["full","sampled"]))
assert sampled_rank < full_rankNotebook 保存的输出 · 本页未执行
| candidates | rank | |
|---|---|---|
| full | 100 | 50 |
| sampled | 11 | 1 |
同层互斥、跨层散列与 Holdout
20,000 个模拟标识。有限样本不保证列联表绝对相等；这里不做真实因果实验。
先预测： 两个层使用相同 salt，会发生什么？反转实验为什么需要等待长期指标？
PYTHON · 04
from reclab.learning.systems import bucket
display(Code(inspect.getsource(bucket), language="python"))
ids = np.arange(20000)
a = np.array([bucket(int(i),"retrieval") for i in ids])
b = np.array([bucket(int(i),"ranking") for i in ids])
groups = pd.DataFrame({"retrieval":np.select([a<20,a<40],["A","B"],default="control"),
"ranking":np.where(b<50,"C","control"),
"holdout":[bucket(int(i),"holdout")<10 for i in ids]})
display(pd.crosstab(groups.retrieval, groups.ranking, normalize="all"))
assert not np.any((a<20) & ((a>=20)&(a<40)))
assert bucket(123,"retrieval") == bucket(123,"retrieval")Notebook 保存的输出 · 本页未执行
def bucket(identifier, layer="retrieval", buckets=100):
if buckets < 1:
raise ValueError("buckets must be positive")
key = f"{layer}:{identifier}".encode()
return int.from_bytes(sha256(key).digest()[:8], "big") % buckets| ranking | C | control |
|---|---|---|
| retrieval | ||
| A | 0.10090 | 0.10405 |
| B | 0.10085 | 0.10025 |
| control | 0.29690 | 0.29705 |
Checks
所有断言和输出都可回溯到代码。它们验证算子、数据快照或机制，不是线上收益。
Next Steps
先记录预测，只改一个变量，再运行对应实验格。需要查看辅助函数时，打开 reclab/learning/。