29 · Hybrid retrieval and real reranker training
Fusion combines complementary retrieval channels; reranking changes order within one candidate set. Neither recovers documents outside its available candidates. Actually construct training pairs, split by query, fit a ranker, and inspect candidate ceilings.
Predict first: Can score units dominate a direct sum of BM25 and cosine? Can random pair splitting place the same query in both training and validation?
Neural models stay frozen. The newly fitted model is a low-cost linear ranker. The original 300 test queries have already been inspected; the additional 40-query diagnostic is not a fresh blind audit.
Navigation: Previous · Next · Integrated experiment
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
from pathlib import Path
import os, sys
root = Path.cwd()
if not (root / 'reclab').exists(): root = root.parent
os.chdir(root); sys.path.insert(0, str(root))
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from reclab.search_lessons import context, query_vectors, score_query, labels
from reclab.search import BM25, rrf
from reclab.retrieval import topk, metrics
ctx = context()
print('Shared search run:', ctx['receipt']['run_id'])Saved notebook output · not executed on this page
Shared search run: 20261030T124927-d1467622
1. RRF uses ranks only
BM25 and cosine have different units. RRF sums 1/(60+rank), reducing scale dependence while discarding score-gap information. Change retrieval depth only and inspect 30 training queries first. The RRF constant stays 60.
qids=sorted(ctx['truth']['train'])[:30]; qv=query_vectors(qids)
rows=[]
for depth in [10,50,100]:
pred={q:topk(score_query(q,v,depth=depth)[2],10) for q,v in zip(qids,qv)}
rows.append({'depth_per_channel':depth,**metrics(pred,labels(qids,'train'),10)})
display(pd.DataFrame(rows))
print('Training-query exploration only; official depth remains 100.')Saved notebook output · not executed on this page
| depth_per_channel | recall@10 | ndcg@10 | mrr@10 | hit_rate@10 | |
|---|---|---|---|---|---|
| 0 | 10 | 0.900000 | 0.735520 | 0.689286 | 0.900000 |
| 1 | 50 | 0.866667 | 0.724944 | 0.685000 | 0.866667 |
| 2 | 100 | 0.833333 | 0.715308 | 0.681667 | 0.833333 |
Training-query exploration only; official depth remains 100.
2. Split by query, not candidate pair
Each query creates 20 highly dependent training pairs. Pair-wise random splitting can leak the same query into fit and select and overstate generalization. Reuse the formal run's 200 training queries: first 160 fit, last 40 select. A separate fixed 40-test-query subset is diagnostic only and never selects hyperparameters.
Features are [BM25, cosine, RRF, query-term coverage, log length], giving pairs×5 inputs. Only positive qrels are observed; the rest are unjudged weak negatives, not real negative impressions.
from reclab.search_lessons import candidate_rows,fit_ranker,rerank_rows,split_training_queries
train_ids=ctx['result']['train_queries']
fit_ids,select_ids=split_training_queries(train_ids)
audit_ids=sorted(ctx['truth']['test'])[:40]
assert not(set(fit_ids)&set(select_ids) or set(train_ids)&set(audit_ids))
fit_rows=candidate_rows(fit_ids); select_rows=candidate_rows(select_ids); audit_rows=candidate_rows(audit_ids)
x=np.concatenate([r['x'] for r in fit_rows]); y=np.concatenate([r['y'] for r in fit_rows])
print('fit/select/audit queries:',len(fit_ids),len(select_ids),len(audit_ids))
print('fit feature shape:',x.shape,'positive labels:',int(y.sum()),'unjudged weak negatives:',int((y==0).sum()))Saved notebook output · not executed on this page
fit/select/audit queries: 160 40 40 fit feature shape: (3200, 5) positive labels: 166 unjudged weak negatives: 3034
3. Actually fit models: add features only
Fix candidates, labels, C=1, seed, and training queries. Compare BM25 alone with all five features. StandardScaler learns mean and variance on fit queries only, never all data. Select supports comparison; test results do not decide which arm to retain. With a positive coefficient, a single BM25 feature is a monotonic transformation and cannot change its order. The comparison tests added information, not the mere fact of training.
Both actual fits create shared search records. config.workshop isolates them from formal serving. Directories save select scores and coefficients only; the additional 40-query diagnostic is displayed in this chapter and not used for selection.
from reclab.search_lessons import ranking_workshop
workshop_id,trained=ranking_workshop(fit_rows,select_rows)
print('Shared search workshop run:',workshop_id)
rows=[]
for name,(ranker,columns) in trained.items():
for split,items,truth_ids in [('select',select_rows,select_ids),('audit_diagnostic',audit_rows,audit_ids)]:
pred=rerank_rows(ranker,items,columns)
rows.append({'features':name,'split':split,**metrics(pred,labels(truth_ids,'train' if split=='select' else 'test'),10)})
display(pd.DataFrame(rows))
print('These new 160-query fits do not overwrite the official 200-query ranker.')Saved notebook output · not executed on this page
Shared search workshop run: 20261030T131425-ebe3235e
| features | split | recall@10 | ndcg@10 | mrr@10 | hit_rate@10 | |
|---|---|---|---|---|---|---|
| 0 | bm25_only | select | 0.8250 | 0.670245 | 0.622044 | 0.825 |
| 1 | bm25_only | audit_diagnostic | 0.7875 | 0.674287 | 0.641111 | 0.800 |
| 2 | five_features | select | 0.7625 | 0.667839 | 0.639653 | 0.800 |
| 3 | five_features | audit_diagnostic | 0.8125 | 0.702456 | 0.671280 | 0.825 |
These new 160-query fits do not overwrite the official 200-query ranker.
4. Coefficients are interpretable, not causal effects
A standardized linear coefficient describes how score changes with one feature while others are fixed. Correlated features, candidate-selection bias, and incomplete labels affect interpretation; it is not a business effect caused by adding that feature. Sigmoid outputs 0–1, but weak labels and candidate sampling prevent interpreting it as calibrated relevance probability, let alone CTR.
ranker,columns=trained['five_features']
feature_names=['bm25','cosine','rrf','query_term_coverage','log_characters']
display(pd.DataFrame({'feature':feature_names,'standardized_coefficient':ranker[1].coef_[0]}))
assert np.array_equal(ranker[0].mean_,x.mean(0)) or np.allclose(ranker[0].mean_,x.mean(0))
print('Scaler fitted only on 160-query training candidates.')Saved notebook output · not executed on this page
| feature | standardized_coefficient | |
|---|---|---|
| 0 | bm25 | 0.566910 |
| 1 | cosine | 0.549268 |
| 2 | rrf | 0.637531 |
| 3 | query_term_coverage | 0.544137 |
| 4 | log_characters | 0.243744 |
Scaler fitted only on 160-query training candidates.
5. Frozen cross-encoders and the candidate-recall ceiling
Bi-encoders encode separately; cross-encoders jointly input query/document and model token interactions, requiring inference per candidate. Actually run TinyBERT on only the first three diagnostic queries with 20 fixed candidates. It may demote correct documents; a frozen model does not guarantee domain adaptation.
from reclab.search_lessons import models
m=models(); cross={}
for row in audit_rows[:3]:
scores=m.rerank(ctx['queries'][row['qid']],[ctx['texts'][d-1] for d in row['ids']])
cross[row['qid']]=[row['ids'][i] for i in np.lexsort((row['ids'],-scores))]
q3=audit_ids[:3]
display(pd.DataFrame([metrics(cross,labels(q3),10)],index=['TinyBERT: 3-query demonstration']))
upper=np.mean([len(set(r['ids'])&ctx['truth']['test'][r['qid']])/len(ctx['truth']['test'][r['qid']]) for r in audit_rows])
linear=rerank_rows(ranker,audit_rows)
assert np.isclose(metrics(linear,labels(audit_ids),100)['recall@100'],upper)
print('40-query candidate Recall@20 upper bound:',upper)Saved notebook output · not executed on this page
/Users/lynnlee/Documents/Codex/2026-10-30/task/soutui-lab/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
| recall@10 | ndcg@10 | mrr@10 | hit_rate@10 | |
|---|---|---|---|---|
| TinyBERT: 3-query demonstration | 0.666667 | 0.666667 | 0.666667 | 0.666667 |
40-query candidate Recall@20 upper bound: 0.8875
Exercise: Keep five features and candidates fixed; change only C on the 40 select queries and retain every result. Final improvement requires fresh audit queries or another prespecified evaluation, not repeated selection on 40 already inspected test queries. New models remain local exercises and do not replace formal serving.