42 · BERT pair inference and real small-budget fine-tuning
Units: S12, S13. Prerequisites: 27, 29, 33; read docs/SEARCH_TRAINING_PROTOCOL.md first.
Sources: Wang Shusen’s pinned outline; Nogueira & Cho 2019; original BERT paper. This is a Lab-authored exercise. Wang's last four modules have a public outline only; this is not presented as the author's complete lecture material.
Evidence boundary: 32 fit / 8 select queries from real SciFact train; full-parameter TinyBERT fine-tuning, not Chinese training or full paper reproduction.
Learning method: predict each comparison, change one variable, run, and explain the difference.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
import numpy as np
import pandas as pd
from IPython.display import display
from reclab.learning.search_curriculum import *Bi-encoders and cross-encoders
Bi-encoders independently encode queries/documents and cache document vectors. Cross-encoders input [CLS]query[SEP]doc[SEP] and recompute attention per pair. Vector scoring costs roughly O(Kd); a cross-encoder performs K Transformer passes with attention quadratic in pair length. Inspect token_type_ids.
from reclab.search_lessons import models,context
m=models(); ctx=context()
q=sorted(ctx['truth']['train'])[32]
ids=__import__('reclab.retrieval',fromlist=['topk']).topk(ctx['bm25'].score(ctx['queries'][q]),10)
b=m.pair_tokenizer([ctx['queries'][q]]*2,[ctx['texts'][d-1] for d in ids[:2]],padding=True,truncation=True,max_length=96,return_tensors='pt')
print({k:list(v.shape) for k,v in b.items()},'segments',b['token_type_ids'][0].unique().tolist())Saved notebook output · not executed on this page
/Users/lynnlee/Documents/Codex/2026-10-30/task/soutui-lab/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
{'input_ids': [2, 96], 'token_type_ids': [2, 96], 'attention_mask': [2, 96]} segments [0, 1]
Truncation and batching
Fix candidates and change max_length from 32 to 96; query and document share the pair budget. Inference with batch=1 and batch=2 should approximately agree, but truncation may change ranking.
import torch
results={}
for length in [32,96]:
batch=m.pair_tokenizer([ctx['queries'][q]]*2,[ctx['texts'][d-1] for d in ids[:2]],padding=True,truncation=True,max_length=length,return_tensors='pt')
with torch.inference_mode(): score=m.reranker(**batch).logits[:,0].numpy()
results[length]=score
print(length,score)
with torch.inference_mode():
one=m.reranker(**{k:v[:1] for k,v in batch.items()}).logits[0,0].item()
assert np.isclose(one,results[96][0],atol=1e-5)Saved notebook output · not executed on this page
32 [ 0.90342236 -11.09127 ] 96 [ 7.8716245 -1.0740322]
Candidate-budget ceilings
Keep query and frozen model fixed; vary BM25 depth 2/5/10. Actually cross-encode every pair, reporting retrieved-positive count, final NDCG, and pair count. Reranking cannot recover absent positives. Latency measures local warm batched inference only.
from reclab.retrieval import metrics
import time
rows=[]
for depth in [2,5,10]:
candidate=ids[:depth]
start=time.perf_counter()
scores=m.rerank(ctx['queries'][q],[ctx['texts'][d-1] for d in candidate])
order=[candidate[i] for i in np.lexsort((candidate,-scores))]
truth=ctx['truth']['train'][q]
rows.append({'depth':depth,'pairs':len(candidate),'known_positive_candidates':len(set(candidate)&set(truth)),
'warm_ms':(time.perf_counter()-start)*1000,**metrics({q:order},{q:truth},10)})
display(pd.DataFrame(rows))Saved notebook output · not executed on this page
| depth | pairs | known_positive_candidates | warm_ms | recall@10 | ndcg@10 | mrr@10 | hit_rate@10 | |
|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 2 | 1 | 5.838958 | 1.0 | 1.0 | 1.0 | 1.0 |
| 1 | 5 | 5 | 1 | 9.108208 | 1.0 | 1.0 | 1.0 | 1.0 |
| 2 | 10 | 10 | 1 | 16.775958 | 1.0 | 1.0 | 1.0 | 1.0 |
Actual training
Run All creates a new workshop run. Use 32 fit / 8 select queries from original train only, at most 128 pairs, two epochs, CPU, batch 8, length 96, lr 2e−5. Qrels provide positives; unjudged BM25 documents from fit queries are weak negatives and may be false negatives. Select does not forcibly insert positives. BCEWithLogits training does not make outputs calibrated probabilities. The fixed budget is not tuned on 300 test queries.
from reclab.search_training_workshop import run
run_id,result=run()
print('teaching run',run_id,'pairs',result['pairs'],'seconds',result['fit_seconds'])
display(pd.json_normalize(result['rows']))
assert result['encoder_weight_l1_change']>0
print('encoder L1 update',result['encoder_weight_l1_change'],'candidate hit rate',result['candidate_hit_rate'])Saved notebook output · not executed on this page
teaching run 20261030T142455-38553a49 pairs 128 seconds 1.1206232080003247
| method | select.recall@10 | select.ndcg@10 | select.mrr@10 | select.hit_rate@10 | |
|---|---|---|---|---|---|
| 0 | frozen_tinybert | 0.833333 | 0.805515 | 0.8125 | 0.875 |
| 1 | finetuned_tinybert | 0.833333 | 0.805515 | 0.8125 | 0.875 |
encoder L1 update 1.8691143989562988 candidate hit rate 0.875
Training curves and candidate ceilings
Lower training loss need not improve select ranking. Unretrieved positives cannot be rescued by reranking. Checkpoints and predictions use the same registry; formal latest stays unchanged. Predict the cost of changing candidates from 10 to 20, without repeatedly searching recipes against select.
import matplotlib.pyplot as plt
plt.plot(result['losses']); plt.xlabel('optimizer step'); plt.ylabel('train BCE'); plt.show()
from reclab.search import latest
assert latest()[0]['run_id']==ctx['receipt']['run_id']
print(result['limits'])Saved notebook output · not executed on this page
<Figure size 640x480 with 1 Axes>
Tiny selected training subset; pretraining overlap unknown; unjudged are not verified negative; fixed top10 candidate ceiling; not paper reproduction or calibrated relevance probability.
Interpreting the saved run
In the fixed saved run, frozen and fine-tuned select NDCG@10 are both about 0.806 and MRR@10 both 0.8125: no ranking improvement. Attention-query weights change in L1, proving actual training, not better generalization. Candidate hit rate is 0.875 while macro positive Recall@10 is about 0.833; denominators differ. For the single-query 2/5/10 sweep, all depths already retrieve a positive, so extra budget increases inference cost without quality improvement.
After changing parameters, interpret the new outputs. This fixed commentary is not a new measurement.
Review and exercise
Write down inputs, outputs, what stayed fixed, and conclusions this chapter cannot support. Predict and rerun one single-variable comparison above; retain failures. Return to the search learning path for the next semantic unit.