44 · Quality, diversity, and scenario-specific objectives

Units: S15. Prerequisites: 35, 43.

Sources: Wang Shusen’s pinned outline. This is a Lab-authored exercise. Wang's last four modules have a public outline only; this is not presented as the author's complete lecture material.

Evidence boundary: constructed task objectives, not real impressions or business gains.

Learning method: predict each comparison, change one variable, run, and explain the difference.

English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.

PYTHON · 01

# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
                          if (p / "reclab").is_dir()), None)
if _translation_root is None:
    raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
    _translation_sys.path.insert(0, str(_translation_root))

PYTHON · 02

import numpy as np
import pandas as pd
from IPython.display import display
from reclab.learning.search_curriculum import *

Scenarios and constraints

Informational queries need aspect coverage; navigation prioritizes official destinations; transactions may require inventory. Identical candidates can have different ideal lists under different needs. Relevance and quality are not the same label.

PYTHON · 03

rel={'a':.95,'b':.94,'c':.90,'d':.80}
print(pd.DataFrame([{'id':d,'rel':rel[d],**DOCUMENTS[d]} for d in DOCUMENTS]))

Saved notebook output · not executed on this page

  id   rel                 title                                     body  \
0  a  0.95          apple repair           official phone battery service   
1  b  0.94           apple fruit   battery repair mentioned in fruit spam   
2  c  0.90  phone battery repair          independent apple service guide   
3  d  0.80    apple repair guide  apple phone battery repair instructions   

     topic  quality  
0  service      1.0  
1     food      0.1  
2  service      0.8  
3    guide      0.9

Quality filtering

Change only minimum quality. Report relevance and low-quality exposure count; improved quality does not imply improved NDCG.

PYTHON · 04

for threshold in [0.,.5]:
    ids=sorted([d for d in DOCUMENTS if DOCUMENTS[d]['quality']>=threshold],key=lambda d:-rel[d])[:2]
    print(threshold,ids,'rel sum',sum(rel[d] for d in ids),'low quality',sum(DOCUMENTS[d]['quality']<.5 for d in ids))

Saved notebook output · not executed on this page

0.0 ['a', 'b'] rel sum 1.89 low quality 1
0.5 ['a', 'c'] rel sum 1.85 low quality 0

Diversity penalty

Greedily score rel−lambda×the number already selected from the same topic. This transparent mechanism is not full MMR/DPP. Remove low-quality candidates first, then vary only lambda to inspect coverage and relevance cost.

PYTHON · 05

def diversify(lam):
    chosen=[]; rest={d for d in DOCUMENTS if DOCUMENTS[d]['quality']>=.5}
    while rest and len(chosen)<2:
        d=min(rest,key=lambda d:(-(rel[d]-lam*sum(DOCUMENTS[d]['topic']==DOCUMENTS[c]['topic'] for c in chosen)),d))
        chosen.append(d); rest.remove(d)
    return chosen
for lam in [0.,.3]:
    ids=diversify(lam); print(lam,ids,'topics',len({DOCUMENTS[d]['topic'] for d in ids}),'rel',sum(rel[d] for d in ids))
assert len({DOCUMENTS[d]['topic'] for d in diversify(.3)})==2

Saved notebook output · not executed on this page

0.0 ['a', 'c'] topics 1 rel 1.85
0.3 ['a', 'd'] topics 2 rel 1.75

Exposure bias

Equal true attractiveness with different positional exposure can produce different clicks. Known constructed propensities explain the mechanism only. Real Lab data lacks exposure probabilities, so no real IPS correction is claimed. Online validation requires randomized experiments and task guardrails.

PYTHON · 06

attraction=np.array([.6,.6]); exposure=np.array([1.,.2])
click_rate=attraction*exposure
print('observed clicks',click_rate,'constructed correction',click_rate/exposure)
assert np.allclose(click_rate/exposure,attraction)

Saved notebook output · not executed on this page

observed clicks [0.6  0.12] constructed correction [0.6 0.6]

Interpreting the saved run

Filtering b reduces top-two relevance sum from 1.89 to 1.85 while low-quality exposures fall from one to zero. Increasing diversity lambda from 0 to 0.3 then obtains two topics at another 0.10 relevance cost. Constructed exposure probabilities create different click rates at equal attractiveness; without real propensities, this correction cannot be transferred to real data.

After changing parameters, interpret the new outputs. This fixed commentary is not a new measurement.

Review and exercise

Write down inputs, outputs, what stayed fixed, and conclusions this chapter cannot support. Predict and rerun one single-variable comparison above; retain failures. Return to the search learning path for the next semantic unit.

44 · Quality, diversity, and scenario-specific objectives