17 · Real search: lexical retrieval, semantics, fusion, and reranking
This is the search track of the same Search Recommendation Ads Lab. It reuses reclab.experiments, ranking metrics, the coarse index, FastAPI, and the workbench. Queries are not fabricated from recommendation logs.
Predict first: Must general-purpose MiniLM beat BM25 in biomedicine? Can TinyBERT reranking recover papers outside its candidate set? Does probing every cluster equal exact retrieval?
Data boundary: All 5,183 BEIR SciFact documents and 300 test queries. Of 809 training queries, 200 are selected by lexicographic query ID only to fit a small linear ranker. Model settings, RRF constants, and candidate budgets are fixed in advance, not tuned to test scores. Neural models are frozen; this chapter performs no neural fine-tuning and is not a full BEIR reproduction. Unjudged documents are scored as nonrelevant, which does not establish true nonrelevance.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
from pathlib import Path
import os, sys
root = Path.cwd()
if not (root / "reclab").exists(): root = root.parent
os.chdir(root)
sys.path.insert(0, str(root))
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from reclab.search import latest, verify_sources, BM25, rrf
from reclab.retrieval import topk
receipt, result = latest()
verify_sources()
print(receipt['run_id'], result['scope'])
print('所有20项公开输入SHA已核验；数据/权重不进入Git。')Saved notebook output · not executed on this page
20261030T124927-d1467622 SciFact BEIR: all 5,183 documents, all 300 test queries; 200 lexicographically selected train queries; frozen MiniLM/TinyBERT, no neural fine-tuning. Unjudged documents treated as nonrelevant, not verified negatives. 所有20项公开输入SHA已核验；数据/权重不进入Git。
Inspect the actual experiment
MiniLM: six layers, 384 dimensions, normalized mean pooling, at most 256 wordpieces. TinyBERT: a two-layer cross-encoder, at most 256 joint tokens. BM25 uses full-corpus term statistics. RRF sums 1/(60+rank) over each channel's top 100. Reranking only sees the fused top 20 and cannot exceed that candidate-recall ceiling.
LogisticRegression fits the top 20 candidates of training queries using BM25, cosine, RRF, query-term coverage, and length features. Unjudged candidates are weak negatives and may include false negatives. Outputs rank candidates; they are not claimed to be calibrated relevance probabilities.
table = pd.DataFrame(result['rows']).set_index('method')
display(table.round(4))
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
table[['ndcg@10','recall@10']].plot.barh(ax=axes[0], title='SciFact: relevance (binary qrels)')
table[['latency_p50_ms','latency_p95_ms']].plot.barh(ax=axes[1], title='Warm CPU query latency (ms)')
plt.tight_layout(); plt.show()
print('候选Recall@20:', result['rerank_candidate_recall@20'])
print('近似索引:', result['approximation'])
assert result['approximation']['dense_coarse12']['exact_top10_agreement'] == 1.0Saved notebook output · not executed on this page
| recall@10 | ndcg@10 | mrr@10 | hit_rate@10 | recall@100 | latency_p50_ms | latency_p95_ms | known_positive_fraction@10 | |
|---|---|---|---|---|---|---|---|---|
| method | ||||||||
| bm25 | 0.7909 | 0.6617 | 0.6276 | 0.8100 | 0.8859 | 2.3056 | 3.0286 | 0.0873 |
| dense | 0.7883 | 0.6484 | 0.6068 | 0.8000 | 0.9250 | 13.2233 | 14.7887 | 0.0890 |
| hybrid_rrf | 0.8089 | 0.6855 | 0.6510 | 0.8267 | 0.9550 | 15.6706 | 17.3551 | 0.0907 |
| linear_rerank | 0.8107 | 0.7034 | 0.6747 | 0.8267 | 0.8752 | 17.1305 | 18.8177 | 0.0910 |
| tinybert_rerank | 0.7868 | 0.6670 | 0.6357 | 0.8000 | 0.8752 | 50.9085 | 55.7259 | 0.0890 |
| dense_coarse2 | 0.6242 | 0.5153 | 0.4848 | 0.6367 | 0.6942 | 12.9576 | 14.7254 | 0.0723 |
| dense_coarse12 | 0.7883 | 0.6484 | 0.6068 | 0.8000 | 0.9250 | 14.4022 | 16.2226 | 0.0890 |
<Figure size 1200x400 with 2 Axes>
候选Recall@20: 0.8752222222222222
近似索引: {'dense_coarse2': {'exact_top10_agreement': 0.7679999999999999, 'mean_candidates': 879.9366666666666}, 'dense_coarse12': {'exact_top10_agreement': 1.0, 'mean_candidates': 5183.0}}
Change one variable: fusion depth
Predict first: How does fusion change when each channel contributes only its top result versus its top two? The constructed rankings below check a mechanism; they are not SciFact scores. Change depth=1 to 2, rerun, and explain. For new real-data scores, explicitly run run(Config(candidate_depth=..., rerank_depth=...)); every run creates a new experiment and preserves prior training.
Latency is measured per-query CPU wall time, including tokenization, encoding, sorting, and applicable reranking. It excludes model loading, offline corpus encoding, disk, and HTTP. Methods are measured in mixed order; these numbers are not production QPS.
toy_rankings = [[2, 1, 3], [3, 1, 2]]
for depth in [1, 2]:
print('depth', depth, topk(rrf(toy_rankings, 3, depth=depth), 3))
# 查看正式算法源码；本格不自动训练。
import inspect
from reclab import search
print(inspect.getsource(search.rrf))Saved notebook output · not executed on this page
depth 1 [2, 3]
depth 2 [1, 2, 3]
def rrf(rankings, n_documents, depth=100, constant=60):
scores = np.zeros(n_documents + 1)
scores[0] = -np.inf
for ranking in rankings:
for rank, doc in enumerate(ranking[:depth], 1):
scores[doc] += 1 / (constant + rank)
# Documents outside the candidate union must never fill a truncated candidate list.
scores[scores == 0] = -np.inf
return scores
Failures and candidate ceilings
Inspect eight fixed queries ordered by TinyBERT's NDCG decline relative to BM25, rather than selecting only wins. Missing vocabulary, domain terminology, truncation, sparse judgments, and pretraining-domain differences are possible explanations to test, not causal conclusions.
Explain a comparison: Find a query where BM25 succeeds but reranking degrades. First check whether the target is in the top 20; then separate retrieval failure from ranking failure. The full per-query list is stored locally as per_query.json.
for failure in result['failures'][:3]:
print('Query ID', failure['qid'])
print('Relevant documents', failure['relevant_document_ids'])
display(pd.DataFrame(failure['metrics']).T[['ndcg@10','mrr@10','recall@10']].round(3))
# 本机 result['failures'] 可查看原查询/标题；Git只保留ID与精简指标。
# 数值上确认：在同一前20候选集内重排，Recall@100必须相同。
assert np.isclose(table.loc['linear_rerank','recall@100'], result['rerank_candidate_recall@20'])
assert np.isclose(table.loc['tinybert_rerank','recall@100'], result['rerank_candidate_recall@20'])Saved notebook output · not executed on this page
Query ID 1363 Relevant documents ['8290953']
| ndcg@10 | mrr@10 | recall@10 | |
|---|---|---|---|
| bm25 | 1.000 | 1.000 | 1.0 |
| dense | 0.000 | 0.000 | 0.0 |
| hybrid_rrf | 0.333 | 0.143 | 1.0 |
| linear_rerank | 1.000 | 1.000 | 1.0 |
| tinybert_rerank | 0.000 | 0.000 | 0.0 |
| dense_coarse2 | 0.000 | 0.000 | 0.0 |
| dense_coarse12 | 0.000 | 0.000 | 0.0 |
Query ID 659 Relevant documents ['1215116']
| ndcg@10 | mrr@10 | recall@10 | |
|---|---|---|---|
| bm25 | 1.0 | 1.0 | 1.0 |
| dense | 1.0 | 1.0 | 1.0 |
| hybrid_rrf | 1.0 | 1.0 | 1.0 |
| linear_rerank | 1.0 | 1.0 | 1.0 |
| tinybert_rerank | 0.0 | 0.0 | 0.0 |
| dense_coarse2 | 0.0 | 0.0 | 0.0 |
| dense_coarse12 | 1.0 | 1.0 | 1.0 |
Query ID 660 Relevant documents ['1215116']
| ndcg@10 | mrr@10 | recall@10 | |
|---|---|---|---|
| bm25 | 1.000 | 1.000 | 1.0 |
| dense | 0.000 | 0.000 | 0.0 |
| hybrid_rrf | 0.315 | 0.125 | 1.0 |
| linear_rerank | 0.000 | 0.000 | 0.0 |
| tinybert_rerank | 0.000 | 0.000 | 0.0 |
| dense_coarse2 | 0.000 | 0.000 | 0.0 |
| dense_coarse12 | 0.000 | 0.000 | 0.0 |
Reproduce and continue
.venv/bin/python -m reclab.search --download
.venv/bin/python scripts/execute_notebooks.py --only 17_Search_Experiments
New experiments appear in the same workbench and /api/experiments/search/{run_id}. Enter an English claim in the workbench to query the actual local corpus.
Dataset authors · BEIR data, CC BY-SA 4.0 · MiniLM, Apache-2.0 · TinyBERT, Apache-2.0. Pinned revisions and hashes are in docs/search_sources.json. English scientific-evidence retrieval does not represent general web search. Pretraining data may overlap; this is not claimed to be a strictly leakage-free benchmark or a source of medical conclusions.