15 · Is evaluation trustworthy, and what should change next?
Goal
Calculate the official metric by hand, verify correct weighting across markets, and inspect frozen audit results and failure cases. After audit results are published, this data is a visible benchmark. Repeatedly optimizing it means it can no longer be called an untouched test set.
Setup
Official protocol · Winning paper · Metric implementation.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
from pathlib import Path
import sys, json, inspect
ROOT = Path.cwd()
if not (ROOT / "reclab").is_dir():
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT))
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display, Code
from reclab.competition import data
from reclab.competition.teaching import load_run, style, bar_scores
style()
pd.set_option("display.max_columns", 12)
pd.set_option("display.max_rows", 40)
run, rows, features = load_run()
print("实际训练快照:", run.name)Saved notebook output · not executed on this page
实际训练快照: 20261030T115556803593-cloud-restore
Step 1 · nDCG with one positive
If the positive ranks r and r≤10, the score is 1/log2(r+1); otherwise it is 0. Average over all users.
The official protocol combines users from t1 and t2 before averaging, so simply averaging the two market scores is incorrect.
from reclab.competition.metrics import evaluate, per_user
display(pd.DataFrame({"positive_rank": [1,2,3,10,11], "ndcg10": [1/np.log2(r+1) if r <=10 else 0 for r in [1,2,3,10,11]]}))
predictions = pd.read_parquet(run / "valid_predictions.parquet")
selection = json.loads((run / "selection.json").read_text())
chosen = selection["selected"]
audit = rows.partition.eq("audit").to_numpy()
report = evaluate(rows.loc[audit], predictions.loc[audit, chosen])
display(report)
by_market = report[report.market.ne("combined")]
weighted = np.average(by_market.ndcg10, weights=by_market.users)
assert np.isclose(weighted, report.iloc[-1].ndcg10)
print("仅按 select 冻结的主模型:", chosen)Saved notebook output · not executed on this page
| positive_rank | ndcg10 | |
|---|---|---|
| 0 | 1 | 1.000000 |
| 1 | 2 | 0.630930 |
| 2 | 3 | 0.500000 |
| 3 | 10 | 0.289065 |
| 4 | 11 | 0.000000 |
| market | users | ndcg10 | hr10 | |
|---|---|---|---|---|
| 0 | t1 | 543 | 0.711020 | 0.799263 |
| 1 | t2 | 1062 | 0.615750 | 0.718456 |
| 2 | combined | 1605 | 0.647982 | 0.745794 |
仅按 select 冻结的主模型: lgb_memory
Step 2 · Use the same audit users for every comparison
lgb_stats uses statistics only; lgb_memory adds collaborative filtering; lgb_all adds the small graph model; lgb_neg9 only reduces training negatives.
All configurations were fixed before inspecting audit. The winner's hidden-test 0.6773 and local audit scores belong to different evaluations and are not plotted as one leaderboard.
metrics = pd.read_csv(run / "metrics.csv")
table = metrics[(metrics.partition == "audit") & (metrics.market == "combined")].copy()
display(table[["model", "users", "ndcg10", "hr10"]].sort_values("ndcg10", ascending=False))
bar_scores(table, "Fixed audit users · local champion-core reproduction")Saved notebook output · not executed on this page
| model | users | ndcg10 | hr10 | |
|---|---|---|---|---|
| 86 | lgb_memory | 1605 | 0.647982 | 0.745794 |
| 95 | lgb_all | 1605 | 0.647218 | 0.744548 |
| 104 | lgb_neg9 | 1605 | 0.646032 | 0.743925 |
| 50 | icf_all_rating | 1605 | 0.628597 | 0.720249 |
| 41 | icf_all_iuf | 1605 | 0.626545 | 0.717134 |
| 23 | icf_s1 | 1605 | 0.622547 | 0.713396 |
| 32 | icf_s3 | 1605 | 0.611734 | 0.696573 |
| 14 | icf_local | 1605 | 0.610977 | 0.694081 |
| 68 | graph_layers2 | 1605 | 0.465419 | 0.603738 |
| 77 | lgb_stats | 1605 | 0.394811 | 0.602492 |
| 5 | target_item_rating_count | 1605 | 0.340036 | 0.537695 |
| 59 | graph_layers0 | 1605 | 0.103664 | 0.188785 |
<Figure size 990x528 with 1 Axes>
Step 3 · Put an improvement on an inspectable scale
Bootstrap paired per-user score differences, stratified by market. The interval reflects sampling uncertainty for these users; it does not correct model-selection bias from repeatedly inspecting audit.
The models compared here were already frozen in selection.json.
from reclab.competition.teaching import paired_bootstrap
interval = paired_bootstrap(rows.loc[audit], predictions.loc[audit, "icf_local"], predictions.loc[audit, chosen])
display(interval)Saved notebook output · not executed on this page
{'users': 1605,
'delta_b_minus_a': 0.037004412984351524,
'bootstrap_95_interval': [0.027052149858014626, 0.046701845992888215],
'scope': 'fixed published audit users; descriptive after repeated inspection'}Step 4 · What do failure cases look like?
Inspect users whose positive ranks poorly and their known history lengths. Use failures to form the next hypothesis, not to edit answers directly. Possible causes include weak co-occurrence links for unpopular items, mismatched source-market preferences, or insufficient graph-model training.
errors = per_user(rows.loc[audit], predictions.loc[audit, chosen])
target_counts = rows[["market", "userId"]].join(features[["target_user_rating_count"]]).drop_duplicates(["market", "userId"])
errors = errors.merge(target_counts, on=["market", "userId"], validate="one_to_one")
display(errors.sort_values("rank", ascending=False).head(10))
bins = pd.cut(errors.target_user_rating_count, bins=[-1,2,5,10,np.inf], labels=["0-2", "3-5", "6-10", "11+"])
display(errors.assign(history_band=bins).groupby(["market", "history_band"], observed=True).agg(users=("userId","size"), ndcg10=("ndcg10","mean")))Saved notebook output · not executed on this page
| market | userId | rank | ndcg10 | hr10 | target_user_rating_count | |
|---|---|---|---|---|---|---|
| 688 | t2 | t2U1002369 | 100 | 0.0 | 0.0 | 6.0 |
| 903 | t2 | t2U1006445 | 99 | 0.0 | 0.0 | 6.0 |
| 1295 | t2 | t2U1013942 | 98 | 0.0 | 0.0 | 10.0 |
| 1398 | t2 | t2U1015657 | 96 | 0.0 | 0.0 | 6.0 |
| 661 | t2 | t2U1002049 | 95 | 0.0 | 0.0 | 7.0 |
| 1107 | t2 | t2U1010355 | 95 | 0.0 | 0.0 | 9.0 |
| 640 | t2 | t2U1001803 | 93 | 0.0 | 0.0 | 10.0 |
| 830 | t2 | t2U1005041 | 92 | 0.0 | 0.0 | 6.0 |
| 227 | t1 | t1U1004477 | 92 | 0.0 | 0.0 | 10.0 |
| 1462 | t2 | t2U1016591 | 91 | 0.0 | 0.0 | 7.0 |
| users | ndcg10 | ||
|---|---|---|---|
| market | history_band | ||
| t1 | 0-2 | 42 | 0.920754 |
| 3-5 | 126 | 0.779186 | |
| 6-10 | 289 | 0.694936 | |
| 11+ | 86 | 0.562768 | |
| t2 | 0-2 | 58 | 0.988506 |
| 3-5 | 213 | 0.697184 | |
| 6-10 | 586 | 0.593280 | |
| 11+ | 205 | 0.489908 |
Check · What was reproduced?
- Direct comparisons pass: original ItemCF scoring formulas; original LightGCN propagation, BPR, and gradients; the full official dataset, 100 candidates, and nDCG@10.
- Reduced or adjusted locally: graph dimensions, depth, update count, and feature count; user-isolated fit/select/audit instead of original ten-fold row bagging; the other target market's qrels are excluded from feature graphs.
- Not reproduced: the full Word2Vec/Node2Vec/NCF/Swing/LLR/BiGraph ensemble, large-scale null-importance feature selection, and final hidden-test score. Original source remains available for inspection.
Next step · Run a complete experiment
Start by removing one market feature in 14 and examining only select. Then vary sampling or layers in 13. Implementation changes require retraining. To connect an edited graph model to the main workflow, run the command below; it creates a new run and preserves existing artifacts.
print("cd", ROOT)
print(".venv/bin/python -m reclab.competition run --tag my-first-ablation --dim 32 --layers 2 --epochs 12")
print("当前 run 的配置:", run / "config.json")
print("指标:", run / "metrics.csv")
print("隐藏测试预测（无公开答案，不报分）:", run / f"test_{chosen}.parquet")Saved notebook output · not executed on this page
cd /Users/lynnlee/Documents/Codex/2026-10-30/task/soutui-lab .venv/bin/python -m reclab.competition run --tag my-first-ablation --dim 32 --layers 2 --epochs 12 当前 run 的配置: /Users/lynnlee/Documents/Codex/2026-10-30/task/soutui-lab/artifacts/xmrec/20261030T115556803593-cloud-restore/config.json 指标: /Users/lynnlee/Documents/Codex/2026-10-30/task/soutui-lab/artifacts/xmrec/20261030T115556803593-cloud-restore/metrics.csv 隐藏测试预测（无公开答案，不报分）: /Users/lynnlee/Documents/Codex/2026-10-30/task/soutui-lab/artifacts/xmrec/20261030T115556803593-cloud-restore/test_lgb_memory.parquet