36 · Normalization, Chinese tokenization, and new terms
Units: S03. Prerequisites: 35.
Sources: Wang Shusen’s pinned outline; Manning et al., Introduction to Information Retrieval (supplementary original textbook, chapters 1, 2, 6, and 8). This is a Lab-authored exercise. Wang's last four modules have a public outline only; this is not presented as the author's complete lecture material.
Evidence boundary: artificial Chinese examples and dictionary mechanisms, not neural Chinese-model training.
Learning method: predict each comparison, change one variable, run, and explain the difference.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
import numpy as np
import pandas as pd
from IPython.display import display
from reclab.learning.search_curriculum import *A normalization contract
NFKC unifies full-width/half-width forms; lower merges English case. This may discard distinctions. Predict whether normalization is idempotent.
x='ＡＩ 手机 Battery'
print(x,'→',normalize(x))
assert normalize(normalize(x))==normalize(x)Saved notebook output · not executed on this page
ＡＩ 手机 Battery → ai 手机 battery
Bidirectional longest matching and ambiguity
Left-to-right longest matching does not guarantee a globally correct segmentation. Predict two segmentations of “研究生命起源” (study the origin of life). Keep the dictionary fixed and change direction only.
vocab={'研究','研究生','生命','命','起源'}
f=segment('研究生命起源',vocab); b=segment('研究生命起源',vocab,reverse=True)
print('forward',f,'backward',b)
assert f!=b and ''.join(f)==''.join(b)Saved notebook output · not executed on this page
forward ['研究生', '命', '起源'] backward ['研究', '生命', '起源']
OOV terms and new-word candidates
New-word discovery needs evidence such as frequency, internal association, and contextual diversity. Here bigram counts from repeated artificial phrases merely propose candidates; they do not prove wordhood. Add only one word.
samples=['量子芯片发布','量子芯片测试','量子芯片量产']
counts=Counter(s[i:i+2] for s in samples for i in range(len(s)-1))
print(counts.most_common(5))
print(segment('量子芯片',{'芯片'}),segment('量子芯片',{'芯片','量子'}))
assert segment('量子芯片',{'芯片','量子'})==['量子','芯片']Saved notebook output · not executed on this page
[('量子', 3), ('子芯', 3), ('芯片', 3), ('片发', 1), ('发布', 1)]
['量', '子', '芯片'] ['量子', '芯片']
Neural subwords are not Chinese word segmentation
Read the existing English TinyBERT tokenizer without downloading another model. WordPiece uses the model vocabulary and may produce UNK. It is not a supervised Chinese word-boundary model, and this English model cannot establish Chinese segmentation quality.
from reclab.search_lessons import models
tok=models().pair_tokenizer
for s in ['研究生命起源','electroencephalography']:
print(s,tok.tokenize(s))
print('vocab',tok.vocab_size)Saved notebook output · not executed on this page
/Users/lynnlee/Documents/Codex/2026-10-30/task/soutui-lab/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
研究生命起源 ['[UNK]', '[UNK]', '生', '[UNK]', '[UNK]', '[UNK]'] electroencephalography ['electro', '##ence', '##pha', '##log', '##raphy'] vocab 30522
Interpreting the saved run
Forward longest matching gives “研究生/命/起源” (graduate student / life / origin); reverse matching gives “研究/生命/起源” (study / life / origin). Both reconstruct the string but assign different boundaries. Adding “量子” (quantum) changes only OOV segmentation. Repeated bigrams do not prove new words. English TinyBERT yields multiple UNKs on Chinese; its subwords are not a valid evaluation of Chinese word segmentation.
After changing parameters, interpret the new outputs. This fixed commentary is not a new measurement.
Review and exercise
Write down inputs, outputs, what stayed fixed, and conclusions this chapter cannot support. Predict and rerun one single-variable comparison above; retain failures. Return to the search learning path for the next semantic unit.