14 · Retrain a fusion model yourself
Goal
Inspect one ranking sample, train LightGBM, vary features or negative-sample count, and explain a prediction. The winning solution scores candidates with a binary classifier, not LambdaRank; this objective is preserved.
Setup
Ranker code · Full experiment entry point. Training cells use all official candidates for real t1 users and fixed user partitions. Only select scores are shown for experimentation.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
from pathlib import Path
import sys, json, inspect
ROOT = Path.cwd()
if not (ROOT / "reclab").is_dir():
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT))
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display, Code
from reclab.competition import data
from reclab.competition.teaching import load_run, style, bar_scores
style()
pd.set_option("display.max_columns", 12)
pd.set_option("display.max_rows", 40)
run, rows, features = load_run()
print("实际训练快照:", run.name)Saved notebook output · not executed on this page
实际训练快照: 20261030T115556803593-cloud-restore
Step 1 · Separate labels from features
Each user has one positive and 99 negative candidates. Candidates were supplied by the competition, not inserted into a retrieval pool after seeing the answer. The feature matrix excludes user ID, item ID, label, and split. Fit, select, and audit each contain whole users.
first_user = rows.query("market == 't1' and partition == 'fit'").userId.iloc[0]
mask = rows.market.eq("t1") & rows.userId.eq(first_user)
graph_columns = [c for c in features if c.startswith("graph_layers") and c != "graph_layers0"]
inspect_columns = ["target_user_rating_count", "target_item_rating_count", "icf_local", "icf_s1", "icf_all_iuf"] + graph_columns[:1]
sample = rows.loc[mask, ["userId", "itemId", "label", "partition"]].join(features.loc[mask, inspect_columns])
display(sample.sort_values("label", ascending=False).head(8))
print("样本正负比:", rows.loc[mask].label.value_counts().to_dict())Saved notebook output · not executed on this page
| userId | itemId | label | partition | target_user_rating_count | target_item_rating_count | icf_local | icf_s1 | icf_all_iuf | graph_layers2 | |
|---|---|---|---|---|---|---|---|---|---|---|
| 86 | t1U1005570 | P1026107 | 1 | fit | 5.0 | 10.0 | 0.000000 | 0.000000 | 0.000000 | 0.235096 |
| 0 | t1U1005570 | P1005811 | 0 | fit | 5.0 | 19.0 | 0.075901 | 0.046836 | 0.030729 | 0.229921 |
| 63 | t1U1005570 | P1024716 | 0 | fit | 5.0 | 7.0 | 0.000000 | 0.003632 | 0.007091 | 0.412555 |
| 72 | t1U1005570 | P1015969 | 0 | fit | 5.0 | 673.0 | 0.022080 | 0.014206 | 0.011226 | 0.992087 |
| 71 | t1U1005570 | P1012765 | 0 | fit | 5.0 | 81.0 | 0.032284 | 0.018921 | 0.014960 | 0.868360 |
| 70 | t1U1005570 | P1012762 | 0 | fit | 5.0 | 11.0 | 0.000000 | 0.000000 | 0.000000 | 0.203922 |
| 69 | t1U1005570 | P1028229 | 0 | fit | 5.0 | 245.0 | 0.027676 | 0.016709 | 0.007334 | 0.243087 |
| 68 | t1U1005570 | P1026274 | 0 | fit | 5.0 | 8.0 | 0.000000 | 0.003686 | 0.003105 | 0.263098 |
样本正负比: {0: 99, 1: 1}
Step 2 · Retrain a real model
Change one of COLUMNS, NEGATIVE_KEEP, or LEAVES, then run this cell and the next.
Keep positives and subsample only fit negatives. All 100 select/audit candidates remain unchanged.
Probabilities after negative subsampling cannot directly represent the original distribution, much less online CTR.
import importlib
import reclab.competition.ranking as ranking_module
import reclab.competition.teaching as teaching_module
importlib.reload(ranking_module)
importlib.reload(teaching_module)
COLUMNS = list(features.columns)
NEGATIVE_KEEP = 99
LEAVES = 15
model, selection, experiment_path = teaching_module.sandbox_ranker(
columns=COLUMNS, negative_keep=NEGATIVE_KEEP, leaves=LEAVES, tag="manual-baseline")
display(selection)
print("本次配置/模型保存:", experiment_path)Saved notebook output · not executed on this page
| market | users | ndcg10 | hr10 | |
|---|---|---|---|---|
| 0 | t1 | 538 | 0.696011 | 0.789963 |
| 1 | combined | 538 | 0.696011 | 0.789963 |
本次配置/模型保存: /Users/lynnlee/Documents/Codex/2026-10-30/task/soutui-lab/artifacts/xmrec/experiments/20261030T115947230728
Step 3 · Change one variable: 99 negatives → 9
This cell trains again, keeping labels, user partitions, features, and leaf count fixed. Keep results that do not improve. Do not delete an ablation because it contradicts your expectation.
sampled_model, sampled_selection, sampled_path = teaching_module.sandbox_ranker(
columns=COLUMNS, negative_keep=9, leaves=LEAVES, tag="negative-sampling-9")
comparison = pd.concat([selection.assign(experiment="99 negatives"), sampled_selection.assign(experiment="9 negatives")])
display(comparison[comparison.market.eq("t1")][["experiment", "users", "ndcg10", "hr10"]])Saved notebook output · not executed on this page
| experiment | users | ndcg10 | hr10 | |
|---|---|---|---|---|
| 0 | 99 negatives | 538 | 0.696011 | 0.789963 |
| 0 | 9 negatives | 538 | 0.691342 | 0.776952 |
Step 4 · What information does the tree use?
Gain measures training-time split improvement, not causal contribution. Then inspect TreeSHAP's log-odds decomposition for one prediction. Positive and negative contributions plus the bias equal the model's raw score, not its probability.
importance = pd.DataFrame({"feature": COLUMNS, "gain": model.booster_.feature_importance(importance_type="gain")}).sort_values("gain", ascending=False)
display(importance.head(10))
fig, ax = plt.subplots(figsize=(9,4))
top = importance.head(10).sort_values("gain")
ax.barh(top.feature, top.gain, color="#006e60")
ax.set(xlabel="LightGBM split gain (training)", title="Feature importance · t1 notebook experiment")
fig.tight_layout(); plt.show()Saved notebook output · not executed on this page
| feature | gain | |
|---|---|---|
| 14 | icf_local | 76522.053376 |
| 17 | icf_all_iuf | 15161.155351 |
| 18 | icf_all_rating | 8366.501459 |
| 16 | icf_s3 | 6512.573406 |
| 8 | target_user_rating_count | 2074.465636 |
| 6 | t1_train_item_count | 1647.045635 |
| 20 | graph_layers2 | 1561.719227 |
| 7 | t1_train_5core_item_count | 1298.185287 |
| 19 | graph_layers0 | 1192.538080 |
| 9 | target_user_rating_mean | 1163.521230 |
<Figure size 990x440 with 1 Axes>
candidate = rows.query("market == 't1' and partition == 'select'").index[0]
x = features.loc[[candidate], COLUMNS]
contributions = model.booster_.predict(x, pred_contrib=True, num_threads=1)[0]
raw = model.booster_.predict(x, raw_score=True, num_threads=1)[0]
assert np.isclose(contributions.sum(), raw)
display(pd.DataFrame({"feature": COLUMNS + ["bias"], "log_odds_contribution": contributions}).sort_values("log_odds_contribution", key=abs, ascending=False).head(12))
print("raw score:", round(float(raw), 5), "probability:", round(float(model.predict_proba(x)[0,1]), 5))Saved notebook output · not executed on this page
| feature | log_odds_contribution | |
|---|---|---|
| 21 | bias | -6.084144 |
| 17 | icf_all_iuf | -0.333658 |
| 6 | t1_train_item_count | -0.266225 |
| 13 | t2_train_5core_item_count | -0.109099 |
| 14 | icf_local | -0.079736 |
| 8 | target_user_rating_count | 0.076829 |
| 16 | icf_s3 | -0.075973 |
| 15 | icf_s1 | -0.072254 |
| 9 | target_user_rating_mean | -0.070087 |
| 12 | t2_train_item_count | -0.060302 |
| 7 | t1_train_5core_item_count | -0.041747 |
| 4 | s3_train_item_count | 0.036470 |
raw score: -7.00451 probability: 0.00091
Check and continue
Must t1's select score fall if icf_s1 is removed from COLUMNS? Predict first, then rerun step 2.