34 · Scores, probabilities, ranking, and evaluation boundaries
Foundations path: 31 → 32 → 33 → 34, then choose search, ads, or recommendation.
Scope: constructed small-array mechanism experiments on CPU; no real logs, formal model training, or business-impact claims.
For each section: predict → change one variable → run → explain. Execute from the beginning first; after edits, rerun affected downstream cells.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
Three questions about the same labels
Scores determine order. Sigmoid maps real values to (0,1), but does not automatically calibrate them. Reuse the project's probability evaluation.
import numpy as np
import matplotlib.pyplot as plt
from scipy.special import expit
from reclab.probability import evaluate
from sklearn.metrics import ndcg_score
y = np.array([1, 0, 1, 0, 0, 1])
scores = np.array([2., 1., .5, -.5, -1., -2.])
p = expit(scores)
base = evaluate(y, p, bins=3)
print({k: base[k] for k in ['auc', 'logloss', 'brier']})Saved notebook output · not executed on this page
{'auc': 0.5555555555555556, 'logloss': 0.8047555609137674, 'brier': 0.2803104793171875}
Change only temperature
Dividing by positive temperature preserves order and AUC but changes probability scores. Do not choose temperature on final audit.
temperatures = [0.25, 1., 4.]
results = [evaluate(y, expit(scores/t), bins=3) for t in temperatures]
assert all(np.isclose(r['auc'], base['auc']) for r in results)
assert len({round(r['logloss'], 6) for r in results}) == 3
for t, r in zip(temperatures, results):
print(t, {k:r[k] for k in ['auc','logloss','brier']})
plt.plot(temperatures, [r['logloss'] for r in results], 'o-')
plt.xlabel('temperature'); plt.ylabel('logloss'); plt.show()Saved notebook output · not executed on this page
0.25 {'auc': 0.5555555555555556, 'logloss': 2.048471115111226, 'brier': 0.3320704642448545}
1.0 {'auc': 0.5555555555555556, 'logloss': 0.8047555609137674, 'brier': 0.2803104793171875}
4.0 {'auc': 0.5555555555555556, 'logloss': 0.685871813125373, 'brier': 0.2462089522244778}
<Figure size 640x480 with 1 Axes>
Within-query ranking versus overall classification
AUC compares all positive/negative pairs; NDCG measures top-ranked relevance within queries. Split six candidates into two constructed queries.
relevance = y.reshape(2, 3)
query_scores = scores.reshape(2, 3)
ndcg = ndcg_score(relevance, query_scores, k=2)
shifted = query_scores.copy()
shifted[1] += 10 # 只改变第二个查询的整体偏置
assert np.isclose(ndcg_score(relevance, shifted, k=2), ndcg)
assert not np.isclose(evaluate(y, expit(shifted.flatten()))['auc'], base['auc'])
print('NDCG@2:', ndcg, '偏置前后总体AUC:', base['auc'], evaluate(y, expit(shifted.flatten()))['auc'])Saved notebook output · not executed on this page
NDCG@2: 0.30657359638272924 偏置前后总体AUC: 0.5555555555555556 0.22222222222222224
Change only the candidate set
Removing a high-scoring negative from the first query makes evaluation easier. Scores from different candidate pools cannot establish model improvement.
full = ndcg_score(relevance[:1], query_scores[:1], k=2)
keep = [0, 2]
reduced = ndcg_score(relevance[:1, keep], query_scores[:1, keep], k=2)
assert reduced > full
print('完整 / 删除高分负例后的 NDCG@2:', full, reduced)Saved notebook output · not executed on this page
完整 / 删除高分负例后的 NDCG@2: 0.6131471927654585 1.0
Explain
Why can logloss change with identical order? Why can query-specific bias preserve within-query rankings but change overall AUC? What extra evidence is needed for offline relevance, click probability, and online gain? Six constructed labels cannot establish real calibration or business impact.
In the next cell, record your prediction, actual result, reason for any difference, and applicability limits. Keep failed results.
reflection = {'预测': '', '结果': '', '解释与边界': ''}
reflectionSaved notebook output · not executed on this page
{'预测': '', '结果': '', '解释与边界': ''}