Foundations, metrics, and A/B testing
Goal
Turn concepts into inspectable inputs, operations, gradients, and comparisons.
Evidence: real data + simulated assignment. Local teaching parameters are not production defaults; mechanism experiments are not full reproductions of the original papers.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
Setup
On the first pass, run from Setup in order: later cells reuse earlier variables. Then change one parameter and rerun that cell and its downstream cells. Constructed data does not establish real business impact.
from pathlib import Path
import sys
import inspect
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display, Code
PROJECT = next(p for p in [Path.cwd(), *Path.cwd().parents]
if (p / "reclab/models.py").exists())
if str(PROJECT) not in sys.path:
sys.path.insert(0, str(PROJECT))
from reclab.data import Dataset
from reclab.paths import ARTIFACTS, PROCESSED, read_json
plt.rcParams.update({"figure.figsize": (9, 3.8), "font.size": 11,
"axes.spines.top": False, "axes.spines.right": False,
"axes.labelcolor": "#182b2b", "text.color": "#182b2b",
"axes.facecolor": "white", "figure.facecolor": "white",
"grid.color": "#d4ddda", "axes.axisbelow": True})
dataset = Dataset()
print("Project:", PROJECT)
print("Dataset version:", dataset.manifest["dataset_version"])
print("Python:", sys.executable)
assert Path(sys.executable).parent == PROJECT / ".venv/bin", "请选择 Recommendation Lab 内核"Saved notebook output · not executed on this page
Project: /Users/lizuolin/local/recommendation-lab Dataset version: 2772cab0aca8cb81 Python: /Users/lizuolin/local/recommendation-lab/.venv/bin/python
Steps
Lecture references: 01_Basics_01.pdf, 01_Basics_02.pdf, 01_Basics_03.pdf
Metric denominators and the recommendation funnel
Simulate an impression log. Real Retailrocket data does not provide a complete impression denominator.
Predict first: What questions are answered by purchases divided by views, and purchases divided by impressions?
impressions, clicks, carts, purchases = 10000, 800, 120, 36
rates = {"CTR (simulated)": clicks/impressions,
"cart given click": carts/clicks, "purchase given cart": purchases/carts,
"purchase per impression": purchases/impressions}
display(pd.Series(rates).to_frame("rate"))
assert np.isclose(rates["CTR (simulated)"]*rates["cart given click"]*rates["purchase given cart"], rates["purchase per impression"])
print("Real event types:", dataset.events.event.unique().tolist())Saved notebook output · not executed on this page
| rate | |
|---|---|
| CTR (simulated) | 0.0800 |
| cart given click | 0.1500 |
| purchase given cart | 0.3000 |
| purchase per impression | 0.0036 |
Real event types: ['view', 'addtocart', 'transaction']
Full-vocabulary versus sampled evaluation
A constructed example with fixed scores. Easier candidates do not mean a better model.
Predict first: What happens to Recall@10 after removing competing candidates?
scores = np.arange(100, dtype=float); positive = 50
full_rank = 1 + int((scores > scores[positive]).sum())
sampled = np.r_[np.arange(10), positive]
sampled_rank = 1 + int((scores[sampled] > scores[positive]).sum())
display(pd.DataFrame({"candidates":[100,len(sampled)],"rank":[full_rank,sampled_rank]},index=["full","sampled"]))
assert sampled_rank < full_rankSaved notebook output · not executed on this page
| candidates | rank | |
|---|---|---|
| full | 100 | 50 |
| sampled | 11 | 1 |
Within-layer exclusion, cross-layer hashing, and holdouts
Use 20,000 simulated identifiers. Finite samples do not guarantee perfectly equal contingency-table counts. This is not a real causal experiment.
Predict first: What happens if two layers use the same salt? Why must a reverse experiment wait for long-term metrics?
from reclab.learning.systems import bucket
display(Code(inspect.getsource(bucket), language="python"))
ids = np.arange(20000)
a = np.array([bucket(int(i),"retrieval") for i in ids])
b = np.array([bucket(int(i),"ranking") for i in ids])
groups = pd.DataFrame({"retrieval":np.select([a<20,a<40],["A","B"],default="control"),
"ranking":np.where(b<50,"C","control"),
"holdout":[bucket(int(i),"holdout")<10 for i in ids]})
display(pd.crosstab(groups.retrieval, groups.ranking, normalize="all"))
assert not np.any((a<20) & ((a>=20)&(a<40)))
assert bucket(123,"retrieval") == bucket(123,"retrieval")Saved notebook output · not executed on this page
def bucket(identifier, layer="retrieval", buckets=100):
if buckets < 1:
raise ValueError("buckets must be positive")
key = f"{layer}:{identifier}".encode()
return int.from_bytes(sha256(key).digest()[:8], "big") % buckets| ranking | C | control |
|---|---|---|
| retrieval | ||
| A | 0.10090 | 0.10405 |
| B | 0.10085 | 0.10025 |
| control | 0.29690 | 0.29705 |
Checks
Every assertion and output is traceable to code. They check operators, data snapshots, or mechanisms, not online gains.
Next steps
Record your prediction, change only one variable, and run the relevant experiment cell. Open reclab/learning/ to inspect helper functions.