38 · Categories, intent, and abstention
Units: S05. Prerequisites: 37.
Sources: Wang Shusen’s pinned outline. This is a Lab-authored exercise. Wang's last four modules have a public outline only; this is not presented as the author's complete lecture material.
Evidence boundary: artificial labels and rules, without training on real Chinese user intents.
Learning method: predict each comparison, change one variable, run, and explain the difference.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
import numpy as np
import pandas as pd
from IPython.display import display
from reclab.learning.search_curriculum import *Define labels
support requests repair help; transaction seeks prices/purchases; navigation seeks a named destination. Abstain when uncertain. Category describes topic; intent describes action. They are not interchangeable.
Artificial device/food categories are independent of intent. apple may refer to a device-related entity or fruit; unknown must remain possible.
examples=[('repair battery','support'),('buy phone','transaction'),('official login','navigation'),('apple','abstain'),('buy battery','abstain'),('weather','abstain')]
print(pd.DataFrame([(q,y,intent(q)) for q,y in examples],columns=['query','expected','predicted']))
for q in ["buy phone","repair phone","buy fruit","apple","phone pie"]:
print(q,"category",category(q),"intent",intent(q))
assert category("buy phone")==category("repair phone")=="device"
assert intent("buy phone")!=intent("repair phone")
assert category("phone pie")=="unknown"Saved notebook output · not executed on this page
query expected predicted 0 repair battery support support 1 buy phone transaction transaction 2 official login navigation navigation 3 apple abstain abstain 4 buy battery abstain abstain 5 weather abstain abstain buy phone category device intent transaction repair phone category device intent support buy fruit category food intent transaction apple category unknown intent abstain phone pie category unknown intent abstain
Audit rules and abstention
These are inspectable rules, not learned decision boundaries. Coverage counts only non-abstained examples. Predict the effect of removing the conflicting word buy.
assert intent('buy battery')=='abstain' and intent('battery')=='support'
covered=[(q,y) for q,y in examples if intent(q)!='abstain']
print('coverage',len(covered)/len(examples),'covered accuracy',np.mean([intent(q)==y for q,y in covered]))Saved notebook output · not executed on this page
coverage 0.5 covered accuracy 1.0
Distribution change
The original rules do not understand implicit requests or negation. Keep rules fixed, change wording, and retain failures.
shift=[('how long does it last','support'),('do not buy phone','support')]
for q,y in shift: print(q,'expected',y,'predicted',intent(q))Saved notebook output · not executed on this page
how long does it last expected support predicted abstain do not buy phone expected support predicted transaction
Routing budget
Intent can be a feature or a route with fallback, not a reason to discard every candidate arbitrarily. Apply the official constraint only to clear navigation; abstention falls back to general retrieval.
for q in ['official repair','apple','official login']:
label=intent(q); required=['official'] if label=='navigation' else []
print(q,label,field_rank(q,required=required))Saved notebook output · not executed on this page
official repair abstain [('a', 3.0), ('d', 3.0), ('c', 2.0), ('b', 1.0)]
apple abstain [('d', 3.0), ('a', 2.0), ('b', 2.0), ('c', 1.0)]
official login navigation [('a', 1.0)]
Interpreting the saved run
buy phone and repair phone share the artificial device category but have different actions. phone pie conflicts across topics and returns unknown. buy battery triggers transaction and support, so the rule abstains. do not buy phone incorrectly triggers transaction, exposing missing semantics. Coverage and accuracy among covered examples apply only to these artificial cases.
After changing parameters, interpret the new outputs. This fixed commentary is not a new measurement.
Review and exercise
Write down inputs, outputs, what stayed fixed, and conclusions this chapter cannot support. Predict and rerun one single-variable comparison above; retain failures. Return to the search learning path for the next semantic unit.