35 · Needs, graded judgments, and evaluation
Units: S01, S02. Prerequisites: 31–34.
Sources: Wang Shusen’s pinned outline; Manning et al., Introduction to Information Retrieval (supplementary original textbook, chapters 1, 2, 6, and 8). This is a Lab-authored exercise. Wang's last four modules have a public outline only; this is not presented as the author's complete lecture material.
Evidence boundary: artificial tasks, sessions, and graded judgments—not graded SciFact labels.
Learning method: predict each comparison, change one variable, run, and explain the difference.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
import numpy as np
import pandas as pd
from IPython.display import display
from reclab.learning.search_curriculum import *Tasks come before metrics
Navigational, informational, and transactional tasks have different success conditions. Predict: can one NDCG value uniquely determine satisfaction? These sessions are artificial; dwell time and clicks do not directly equal success.
sessions=pd.DataFrame([['navigation',1.,80,False],['navigation',1.,2,True],['research',.8,120,True]],columns=['task','ndcg','seconds','task_success'])
display(sessions)
assert sessions.iloc[0].ndcg==sessions.iloc[1].ndcg and sessions.iloc[0].task_success!=sessions.iloc[1].task_successSaved notebook output · not executed on this page
| task | ndcg | seconds | task_success | |
|---|---|---|---|---|
| 0 | navigation | 1.0 | 80 | False |
| 1 | navigation | 1.0 | 2 | True |
| 2 | research | 0.8 | 120 | True |
Grades and a hand calculation
0=irrelevant, 1=partially relevant, 2=relevant, 3=directly satisfies the need; gain=2^rel−1 and discount=log2(rank+1). Swap only the first two results and predict the score change.
judge={'a':3,'b':0,'c':1,'d':2}
orders=[['a','d','c'],['d','a','c']]
rows=[{'order':o,'ndcg':graded_ndcg(o,judge)} for o in orders]
manual=(3/np.log2(2)+7/np.log2(3)+1/np.log2(4))/(7+3/np.log2(3)+1/2)
assert np.isclose(rows[1]['ndcg'],manual)
display(pd.DataFrame(rows))Saved notebook output · not executed on this page
| order | ndcg | |
|---|---|---|
| 0 | [a, d, c] | 1.000000 |
| 1 | [d, a, c] | 0.842828 |
Disagreement and pooling
Two annotators disagree about the same need. Record guidelines, disagreement, and adjudication. The pool is the union of two systems' top two results; unjudged documents outside it are not truly irrelevant by default. Predict which judgments a newly added system might lack.
j1=np.array([3,0,1,2]); j2=np.array([2,0,2,2])
print('agreement',np.mean(j1==j2),'absolute disagreement',np.abs(j1-j2).mean())
pool=set(['a','d'])|set(['c','a']); unseen=set(DOCUMENTS)-pool
assert unseen=={'b'}
print('pool',sorted(pool),'unjudged',unseen)Saved notebook output · not executed on this page
agreement 0.5 absolute disagreement 0.5
pool ['a', 'c', 'd'] unjudged {'b'}
Sensitivity to incomplete judgments
Change only b's judgment. Real experiments may obtain new annotations, not edit truth to improve scores. Existing SciFact loading binarizes labels; chapter 25 remains the real-data contract lesson.
old=dict(judge); old['b']=0
new=dict(judge); new['b']=3
print('unjudged-as-zero',graded_ndcg(['b','a','d'],old),'counterfactual adjudication',graded_ndcg(['b','a','d'],new))Saved notebook output · not executed on this page
unjudged-as-zero 0.6298989693877386 counterfactual adjudication 1.0
Interpreting the saved run
Two saved navigational sessions both have NDCG=1, but one succeeds and the other fails. Swapping the first two results lowers graded NDCG from 1 to about 0.843. Annotators agree exactly only half the time. Document b is unjudged outside the pool; a counterfactual judgment substantially changes scores. Define needs and annotation guidelines before interpreting metrics.
After changing parameters, interpret the new outputs. This fixed commentary is not a new measurement.
Review and exercise
Write down inputs, outputs, what stayed fixed, and conclusions this chapter cannot support. Predict and rerun one single-variable comparison above; retain failures. Return to the search learning path for the next semantic unit.