35 · Needs, graded judgments, and evaluation
Define user needs, graded relevance, disagreement, and incomplete-judgment sensitivity.
19 LESSONS · ONE CONNECTED PATH
Search
Connect user queries to document retrieval, relevance judgments and ranking.
Before you begin: Start with 31–34, then needs and relevance before queries, retrieval and ranking.
THE LEARNING PATH
Build one idea on the last.
01
Needs and relevance
S01–S02: tasks, satisfaction, graded judgments, disagreement, pooling and real qrels. Start with Foundations 31–34.
25 · A real search task and relevance evaluation
Define real search data contracts and calculate query-level relevance metrics.
02
Query understanding
S03–S06: normalization, Chinese tokenization, term and field weights, intent rejection, spelling and semantic drift. Constructed examples make the mechanisms visible.
36 · Normalization, Chinese tokenization, and new terms
Compare Chinese dictionary segmentation directions and distinguish words from subwords.
37 · Query-term weights and field matching
Separate soft query-term/field weights from hard eligibility constraints.
38 · Categories, intent, and abstention
Define topic and action labels, audit rule-based intent, and provide abstention fallback.
39 · Spelling correction, rewriting, and semantic drift
Inspect spelling correction, false retrieval, constraint preservation, and rewrite fallback.
03
Retrieval
S07–S11: positional / Boolean indexing → BM25 → frozen semantic representations → coarse-cluster candidate budgets → cache consistency. Chapter 28 is not HNSW.
40 · Inverted indexes, Boolean retrieval, and positional constraints
Check Boolean and positional indexes against a brute-force oracle and index generations.
26 · Inverted indexes and BM25
Inspect postings, BM25 term saturation, length normalization, and zero-score queries.
27 · Semantic encoding and vector retrieval
Inspect tokenization, masked pooling, vector geometry, and truncation in real retrieval.
28 · Approximate-index recall, candidates, and latency
Measure approximate-index error separately from task recall and end-to-end latency.
41 · Candidate caches, TTL, and consistency
Explore candidate-cache TTL, generation checks, stale data, and explicit failure fallback.
04
Relevance and ranking
S12–S15: BERT pair inputs and small-budget fine-tuning → ranking losses → real-data fusion → quality and diversity. Training workshops report teaching selection metrics.
42 · BERT pair inference and real small-budget fine-tuning
Inspect pair inference and run bounded, actual TinyBERT fine-tuning without claiming gains.
43 · Pointwise, pairwise, and listwise objectives
Optimize three ranking objectives and test score-shift invariants on constructed labels.
29 · Hybrid retrieval and real reranker training
Train query-isolated linear rerankers and inspect frozen cross-encoder candidate ceilings.
44 · Quality, diversity, and scenario-specific objectives
Explore relevance, quality filtering, topic coverage, and constructed exposure bias.
05
Suggestions and system diagnosis
S16–S18: constructed query sessions → traceable requests and fallbacks → error diagnosis in the real-data service.
45 · Prefix suggestions, related queries, and session recommendation
Separate prefix completion, related-query suggestions, and next-query transitions.
46 · End-to-end traces, failures, and course review
Trace query processing through candidates and cache to ranking and explicit degradation.
30 · Failure analysis, timing boundaries, and a shared service
Diagnose per-query failures and verify served document rankings against offline results.
06
Reference · integrated search experiment
The original SciFact experiment after the core sequence. Original models and previously inspected test results remain unchanged.
17 · Real search: lexical retrieval, semantics, fusion, and reranking
Compare BM25, frozen neural retrieval, fusion, and reranking on SciFact.
No lessons match. Try another concept or clear the search.
TRACE THE IDEAS
Sources & lecture notes
Follow the original material alongside the Lab. Source scope, original language and the boundaries of each teaching unit are explicit.
Shusen Wang · Search Engine Technology
Pinned six-module syllabus with nine linked PDFs. Query processing, retrieval, ranking and suggestions are primarily outlines; Lab experiments are independent teaching implementations.
Introduction to Information Retrieval · Manning / Raghavan / Schütze
Supplementary indexing, query processing, retrieval and evaluation. The original host closed connections during this network check; Wang’s lecture links remain available.
BERT: Pre-training of Deep Bidirectional Transformers
Architecture, inputs and pre-training background. This Lab does not reproduce BERT pre-training.
Passage Re-ranking with BERT
Reference for query–passage cross-encoder scoring. The small TinyBERT workshop is not a reproduction of the paper’s benchmark.
Chapter-to-source map 18 units
S01
Tasks, needs and search products
Distinguish the user’s information need from an offline metric.
- Shusen Wang · Search Engine Technology Basics, lectures 1–2
Boundary: Artificial sessions do not measure real satisfaction; NDCG is an offline proxy.
S02
Graded relevance, judgments and evaluation
Calculate graded gains and inspect disagreement and incomplete judgments.
- Shusen Wang · Search Engine Technology Relevance, lectures 1–2
Boundary: Constructed graded judgments; SciFact keeps binary and incomplete qrels.
S03
Normalization, Chinese tokenization and new words
Compare dictionary segmentations and explain ambiguity and out-of-vocabulary terms.
- Shusen Wang · Search Engine Technology Syllabus: Query processing → tokenization
Boundary: Artificial word examples, bidirectional dictionary matching and candidate counts; no trained neural Chinese tokenizer.
S04
Term weights and field matching
Change a field or term weight and inspect both retrieval gains and false matches.
- Shusen Wang · Search Engine Technology Relevance, lecture 3: text matching
Boundary: A four-document linear field mechanism, not BM25F or production term-weight learning.
S05
Categories, intent and rejection
Define intent labels and let uncertain rules reject rather than force a class.
- Shusen Wang · Search Engine Technology Syllabus: Query processing → categories and intent
Boundary: Manually labeled topics/actions and rule-based rejection; no real intent-accuracy claim.
S06
Query rewriting, spelling and semantic drift
Compare original and rewritten queries, including a fallback on semantic drift.
- Shusen Wang · Search Engine Technology Syllabus: Query processing → query rewriting
Boundary: Fixed correction and synonym rules, including deliberate failures; no generative-model gain.
S07
Inverted indexes, Boolean and positional constraints
Build positional postings and verify Boolean/phrase results against exhaustive scans.
- Introduction to Information Retrieval · Manning / Raghavan / Schütze Chapters 1–2: Boolean retrieval, terms and postings
- Shusen Wang · Search Engine Technology Syllabus: Retrieval → inverted indexes
Boundary: A four-document positional index, exhaustive checks and rebuild generations; not a distributed incremental engine.
S08
Lexical relevance and BM25
Inspect BM25 terms and compare rankings on real queries.
- Introduction to Information Retrieval · Manning / Raghavan / Schütze Chapter 11: probabilistic information retrieval
- Shusen Wang · Search Engine Technology Relevance, lecture 3
Boundary: The original real-data BM25 experiment in 26. Previously inspected test results are diagnostic.
S09
Semantic representations and dense retrieval
Trace tokenized text through frozen embeddings to nearest documents.
- Shusen Wang · Search Engine Technology Syllabus: Retrieval → vector retrieval
Boundary: Frozen MiniLM in chapter 27; no retrieval-encoder training.
S10
Approximate indexes and candidate budgets
Measure the recall–candidate-budget tradeoff against an exact oracle.
- Shusen Wang · Search Engine Technology Syllabus: vector retrieval; candidate-budget exercise by the Lab
Boundary: A 12-cluster teaching index in 28, not an implementation of HNSW or FAISS.
S11
Candidate caches and consistency
Invalidate cached candidates correctly when queries or index generations change.
- Shusen Wang · Search Engine Technology Syllabus: Retrieval → caching
Boundary: A logical clock and loader-call cost, not measured production cache gains.
S12
BERT relevance scoring and inference
Inspect pair inputs, truncation, batching and cross-encoder scoring.
42 · BERT pair inference and real small-budget fine-tuning29 · Hybrid retrieval and real reranker training
- Shusen Wang · Search Engine Technology Relevance, lecture 4: BERT inference
- BERT: Pre-training of Deep Bidirectional Transformers Section 3: BERT
Boundary: Real TinyBERT pair inputs with truncation, batching and budget comparisons in 42 use the training partition. Chapter 29 also retains diagnostics on previously inspected test queries.
S13
Training relevance models
Fine-tune a small relevance model with separated fit and selection queries.
- Shusen Wang · Search Engine Technology Relevance, lecture 5: BERT training
- Passage Re-ranking with BERT Pair inputs and reranker training
Boundary: 32 fit / 8 selection queries, 128 weakly supervised pairs and 2 epochs; no blind-test or improvement claim.
S14
Ranking objectives and multi-channel fusion
Compare pointwise, pairwise and listwise objectives and score fusion.
- Shusen Wang · Search Engine Technology Syllabus: Ranking → training fusion models
Boundary: Actual optimization of three losses on constructed graded labels; real fusion retains the weak negatives of 29.
S15
Quality, diversity and constraints
Separate relevance from quality and topic diversity under explicit constraints.
- Shusen Wang · Search Engine Technology Basics, lecture 2: satisfaction; constraint exercise is a Lab extension
Boundary: Artificial quality/topics and known-propensity mechanisms; no online gain or real IPS result.
S16
Query recommendations and suggestions
Generate and rank suggestions from sessions; expose leakage and template overlap.
- Shusen Wang · Search Engine Technology Syllabus: suggestions → scenarios, retrieval and ranking
Boundary: Constructed sessions with disclosed template overlap; reciprocal rank and prefix precision/recall illustrate mechanisms.
S17
Service traces, diagnosis and performance
Trace a request through candidates, scores, fallback and latency diagnosis.
46 · End-to-end traces, failures, and course review30 · Failure analysis, timing boundaries, and a shared service
- Shusen Wang · Search Engine Technology Basics, lecture 4: search pipeline
Boundary: An explicitly constructed trace API; the original real-data API and performance diagnosis remain in 30.
S18
Synthesis and course navigation
Explain an end-to-end search result and identify what the evidence cannot establish.
- Shusen Wang · Search Engine Technology Six-module syllabus; synthesis designed by the Lab
Boundary: Semantic sequence, sources, prerequisites and evidence. Industrial scale and advanced extensions remain outside the core.
Original lecture library 9 entries
These links open the original author or publisher material, in its original language and under its own terms.
- Search-engine basics
01_Basics_01.pdf · Chinese
Original Chinese author PDF; Lab teaching text is independently authored.
- What determines user satisfaction?
01_Basics_02.pdf · Chinese
Original Chinese author PDF; Lab teaching text is independently authored.
- Search evaluation metrics
01_Basics_03.pdf · Chinese
Original Chinese author PDF; Lab teaching text is independently authored.
- Search pipeline
01_Basics_04.pdf · Chinese
Original Chinese author PDF; Lab teaching text is independently authored.
- Defining and grading relevance
02_Rel_01.pdf · Chinese
Original Chinese author PDF; Lab teaching text is independently authored.
- Relevance evaluation
02_Rel_02.pdf · Chinese
Original Chinese author PDF; Lab teaching text is independently authored.
- Text matching scores
02_Rel_03.pdf · Chinese
Original Chinese author PDF; Lab teaching text is independently authored.
- BERT relevance inference
02_Rel_04.pdf · Chinese
Original Chinese author PDF; Lab teaching text is independently authored.
- Training BERT relevance models
02_Rel_05.pdf · Chinese
Original Chinese author PDF; Lab teaching text is independently authored.