27 · Semantic encoding and vector retrieval
Semantic retrieval maps queries and documents into one space with a shared encoder, then ranks by vector similarity. Inspect tokens, masked pooling, normalization, truncation, and actual query differences.
Predict first: After vector normalization, do dot product and cosine still differ? Must a general-purpose model understand biomedicine better than BM25?
Reuse the formal run's 5,183×384 document vectors without neural retraining. Encode 40 test queries for diagnosis; the original 300-query aggregate remains separate. This is exploration on an already inspected test set, not a new blind evaluation.
Navigation: Previous · Next · Integrated experiment
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
from pathlib import Path
import os, sys
root = Path.cwd()
if not (root / 'reclab').exists(): root = root.parent
os.chdir(root); sys.path.insert(0, str(root))
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from reclab.search_lessons import context, query_vectors, score_query, labels
from reclab.search import BM25, rrf
from reclab.retrieval import topk, metrics
ctx = context()
print('Shared search run:', ctx['receipt']['run_id'])Saved notebook output · not executed on this page
Shared search run: 20261030T124927-d1467622
1. From strings to tensors
The tokenizer outputs input_ids and attention_mask, both batch×tokens. Hidden states are batch×tokens×384. Pool only mask=1 positions, then L2-normalize; padding must not be averaged as text.
Print shapes and token counts only, without saving raw query text.
from reclab.search_lessons import models
m=models(); qids=sorted(ctx['truth']['test'])[:40]
encoded=m.tokenizer([ctx['queries'][q] for q in qids[:2]],padding=True,truncation=True,max_length=256,return_tensors='pt')
print({k:tuple(v.shape) for k,v in encoded.items()})
print('nonpadding lengths:',encoded['attention_mask'].sum(1).tolist())
with m.torch.inference_mode():
hidden=m.encoder(**encoded).last_hidden_state
mask=encoded['attention_mask'].unsqueeze(-1)
manual=m.torch.nn.functional.normalize((hidden*mask).sum(1)/mask.sum(1),dim=1).numpy()
actual=m.encode([ctx['queries'][q] for q in qids[:2]])
assert np.allclose(manual,actual,atol=1e-5)
print('hidden / pooled:',tuple(hidden.shape),actual.shape)Saved notebook output · not executed on this page
/Users/lynnlee/Documents/Codex/2026-10-30/task/soutui-lab/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
{'input_ids': (2, 17), 'token_type_ids': (2, 17), 'attention_mask': (2, 17)}
nonpadding lengths: [15, 17]
hidden / pooled: (2, 17, 384) (2, 384)
2. The geometry of normalization
For unit vectors, ||x-y||²=2-2x·y. Maximum dot product, maximum cosine, and minimum Euclidean distance therefore yield the same ranking with a shared stable tie rule. Without normalization, magnitude may dominate. Do not freely mix distance definitions.
vectors=ctx['vectors']; qv=query_vectors(qids)
assert np.allclose(np.linalg.norm(vectors,axis=1),1,atol=1e-5)
assert np.allclose(np.linalg.norm(qv,axis=1),1,atol=1e-5)
dot=vectors@qv[0]; distances=((vectors-qv[0])**2).sum(1)
assert np.allclose(distances,2-2*dot,atol=1e-5)
assert topk(np.r_[-np.inf,dot],10)==topk(np.r_[-np.inf,-distances],10)
print('normalized corpus shape:',vectors.shape,'float32 vector MiB:',vectors.nbytes/2**20)Saved notebook output · not executed on this page
normalized corpus shape: (5183, 384) float32 vector MiB: 7.59228515625
3. What information does truncation discard?
The formal configuration uses at most 256 wordpieces for title+abstract; the model's supported maximum is not the experiment's actual length. Count full token lengths for only the first 80 documents, then change inference length from 64 to 256 for those same documents. Vector differences show representation changes, not improved quality.
texts=ctx['texts'][:80]
lengths=np.array([len(m.tokenizer(t,add_special_tokens=True,truncation=False)['input_ids']) for t in texts])
print('80-doc sample token percentiles:',np.percentile(lengths,[50,90,100]),'fraction >256:',(lengths>256).mean())
from dataclasses import replace
old=m.config
try:
m.config=replace(old,max_length=64); short=m.encode(texts[:8])
finally: m.config=old
full=m.encode(texts[:8])
display(pd.DataFrame({'doc_offset':range(8),'cosine_64_vs_256':(short*full).sum(1),'full_token_count':lengths[:8]}))Saved notebook output · not executed on this page
Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors
80-doc sample token percentiles: [301. 532.7 975. ] fraction >256: 0.6
| doc_offset | cosine_64_vs_256 | full_token_count | |
|---|---|---|---|
| 0 | 0 | 0.781264 | 409 |
| 1 | 1 | 0.915053 | 234 |
| 2 | 2 | 0.816750 | 175 |
| 3 | 3 | 0.850522 | 295 |
| 4 | 4 | 0.862596 | 223 |
| 5 | 5 | 0.904354 | 372 |
| 6 | 6 | 0.889317 | 209 |
| 7 | 7 | 0.843466 | 238 |
4. Lexical versus semantic retrieval on the same queries
Hold the 40 queries, corpus, and metrics fixed and change only the retrieval channel. Report per-query differences and the average; one winning example is not a universal rule. Frozen MiniLM's pretraining may include related scientific material, so strict contamination-free evaluation is not claimed.
rows=[]
for q,v in zip(qids,qv):
bm,ds,_=score_query(q,v)
a=metrics({q:topk(bm,10)},labels([q]),10)['ndcg@10']
b=metrics({q:topk(ds,10)},labels([q]),10)['ndcg@10']
rows.append({'qid':q,'bm25':a,'dense':b,'delta':b-a})
comparison=pd.DataFrame(rows)
display(comparison.sort_values('delta').head(6))
display(comparison[['bm25','dense','delta']].mean().to_frame('40-query diagnostic mean'))
comparison.delta.plot.hist(bins=12,title='Dense minus BM25 NDCG: 40-query diagnostic');plt.show()Saved notebook output · not executed on this page
| qid | bm25 | dense | delta | |
|---|---|---|---|---|
| 7 | 1024 | 1.000000 | 0.00000 | -1.000000 |
| 1 | 100 | 1.000000 | 0.50000 | -0.500000 |
| 18 | 1107 | 1.000000 | 0.50000 | -0.500000 |
| 34 | 1180 | 0.386853 | 0.00000 | -0.386853 |
| 35 | 1185 | 1.000000 | 0.63093 | -0.369070 |
| 33 | 118 | 1.000000 | 0.63093 | -0.369070 |
| 40-query diagnostic mean | |
|---|---|
| bm25 | 0.680418 |
| dense | 0.643972 |
| delta | -0.036446 |
<Figure size 640x480 with 1 Axes>
5. Offline and online costs
Document encodings can be reused offline; each new query requires encoding. 5,183×384 float32 values occupy only about 7.6 MiB, but model loading, tokenization, and encoding still cost time. The formal latency table includes query encoding. Next chapter's index-only timing explicitly excludes it to avoid invalid comparisons.
Exercise: Replace the diagnostic query set with 40 fixed training queries and rerun section 4. Change only the query set, not model and parameters simultaneously. Explain why general semantic models may lose to BM25 on specialist terminology.