25 · A real search task and relevance evaluation
Recommendation often predicts items from user history; search must understand both query and document. Ranking infrastructure can be shared, but task definitions, labels, and denominators cannot. Start with the real data contract, calculate MRR, NDCG, and recall, and inspect what unjudged documents imply.
Predict first: If a query has three relevant papers and top 10 retrieves one, are Hit and Recall equal? Is every unlabeled paper truly irrelevant?
Reuse chapter 17's SciFact experiment without duplicating data or trainers. The full corpus has 5,183 documents; test has 300 queries and 339 positive judgments; train has 809 queries, of which the formal linear model uses the first 200 lexicographically. Every subset exercise states its scope and does not replace the original 300-query results.
Navigation: Next · Integrated experiment
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
from pathlib import Path
import os, sys
root = Path.cwd()
if not (root / 'reclab').exists(): root = root.parent
os.chdir(root); sys.path.insert(0, str(root))
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from reclab.search_lessons import context, query_vectors, score_query, labels
from reclab.search import BM25, rrf
from reclab.retrieval import topk, metrics
ctx = context()
print('Shared search run:', ctx['receipt']['run_id'])Saved notebook output · not executed on this page
Shared search run: 20261030T124927-d1467622
1. Specify an input/output contract
Documents contain _id/title/text; queries map query_id → text; qrels map query_id → {relevant document_idx}. Original IDs are retained, while internal numbering starts at 1, reserving 0 for the shared recommendation-retrieval function. Ignoring this convention silently masks the first document.
Try it: Check unique IDs, cross-table references, and query groups, not just total rows. Only counts and shapes are printed; original queries and abstracts stay local.
summary = pd.DataFrame([{'split':s, 'queries':len(q), 'positive_qrels':sum(map(len,q.values()))} for s,q in ctx['truth'].items()])
display(summary)
print('documents / vectors:',len(ctx['corpus']), ctx['vectors'].shape)
assert ctx['corpus']['_id'].is_unique
assert not (set(ctx['truth']['train']) & set(ctx['truth']['test']))
assert all(1 <= d <= len(ctx['corpus']) for split in ctx['truth'].values() for truth in split.values() for d in truth)Saved notebook output · not executed on this page
| split | queries | positive_qrels | |
|---|---|---|---|
| 0 | train | 809 | 919 |
| 1 | test | 300 | 339 |
documents / vectors: 5183 (5183, 384)
2. Each metric answers a different question
Recall@k=relevant retrieved / known relevant for that query. Hit@k asks whether any were found. MRR considers only the first relevant result. Binary NDCG discounts each relevant result by 1/log2(rank+1) and divides by ideal DCG. Compute per query, then macro-average; do not pool all document pairs into one average.
Predict → change one variable: Replace only the first irrelevant document with a relevant one. Which metrics change? This constructed ranking checks the mechanism before returning to real qrels.
truth = {'q':{1,2,3}}
for prediction in [[4,2,5],[1,2,5]]:
display(pd.DataFrame([metrics({'q':prediction},truth,3)], index=[str(prediction)]))
rel = np.array([0.,1.,0.]); discount=1/np.log2(np.arange(3)+2)
manual_ndcg=(rel*discount).sum()/discount.sum()
assert np.isclose(manual_ndcg, metrics({'q':[4,2,5]},truth,3)['ndcg@3'])
assert metrics({'q':[4,2,5]},truth,3)['hit_rate@3']==1
assert metrics({'q':[4,2,5]},truth,3)['recall@3']==1/3Saved notebook output · not executed on this page
| recall@3 | ndcg@3 | mrr@3 | hit_rate@3 | |
|---|---|---|---|---|
| [4, 2, 5] | 0.333333 | 0.296082 | 0.5 | 1.0 |
| recall@3 | ndcg@3 | mrr@3 | hit_rate@3 | |
|---|---|---|---|---|
| [1, 2, 5] | 0.666667 | 0.765361 | 1.0 | 1.0 |
3. Where does the denominator come from when judgments are incomplete?
The SciFact export provides positive judgments, not exhaustive negative judgments. Unlabeled documents score 0 under this reproducible convention, which does not prove true-world irrelevance. known_positive_fraction@10 is only the fraction with known positive judgments.
Find queries with multiple positives and inspect their macro-average contributions. Explain: Why is total hits / total positives not equal to query-macro Recall?
truth=ctx['truth']['test']
rows=[]
for r in ctx['per_query']:
relevant=len(truth[r['qid']]); recall=r['metrics']['bm25']['recall@10']
rows.append({'qid':r['qid'],'relevant':relevant,'recall':recall,'hits':recall*relevant})
q=pd.DataFrame(rows)
display(q[q.relevant>1].head(8))
print('macro recall:',q.recall.mean(),'pooled recall:',q.hits.sum()/q.relevant.sum())
assert np.isclose(q.recall.mean(),ctx['result']['rows'][0]['recall@10'])Saved notebook output · not executed on this page
| qid | relevant | recall | hits | |
|---|---|---|---|---|
| 8 | 1029 | 3 | 1.00 | 3.0 |
| 9 | 1041 | 2 | 1.00 | 2.0 |
| 23 | 1132 | 2 | 0.50 | 1.0 |
| 63 | 1274 | 3 | 1.00 | 3.0 |
| 81 | 133 | 5 | 0.20 | 1.0 |
| 95 | 1379 | 4 | 0.50 | 2.0 |
| 100 | 141 | 2 | 1.00 | 2.0 |
| 107 | 179 | 4 | 0.75 | 3.0 |
macro recall: 0.7909444444444444 pooled recall: 0.7728613569321534
4. Add an artificial judgment to observe bias
Change only the relevance label of a previously unjudged document, keeping ranking fixed. Metric changes come from label coverage, not a better retriever. These labels are explicitly constructed counterfactuals and are never written to real qrels.
r=ctx['per_query'][0]; qid=r['qid']
original=set(truth[qid]); prediction=[int(np.flatnonzero(ctx['corpus']['_id'].values==d)[0])+1 for d in r['top10']['bm25']]
unjudged=next((d for d in prediction if d not in original),None)
if unjudged is not None:
display(pd.DataFrame([metrics({qid:prediction},{qid:original},10),metrics({qid:prediction},{qid:original|{unjudged}},10)],index=['original judgments','constructed extra positive']))
print('真实qrels未修改:',truth[qid]==original)Saved notebook output · not executed on this page
| recall@10 | ndcg@10 | mrr@10 | hit_rate@10 | |
|---|---|---|---|---|
| original judgments | 0.0 | 0.000000 | 0.0 | 0.0 |
| constructed extra positive | 0.5 | 0.613147 | 1.0 | 1.0 |
真实qrels未修改: True
5. Sources, reproduction, and exercise
The 20 inputs in docs/search_sources.json record pinned repository commits, byte counts, and SHA hashes. The first context() verifies all inputs. Models, raw data, and per-query outputs stay out of Git; the formal run's configuration and code digest remain traceable.
Exercise: Construct two queries with one and five positives so macro and pooled Recall differ substantially. Explain why queries without targets cannot silently enter this protocol as zero scores. Next, keep labels fixed and change lexical retrieval.