19 · Impressions, labels, missingness, and leakage-aware splitting
Chapter 18 remains the overview; this chapter isolates one question. The formal 200k experiment is preserved. Teaching runs use separate records in the shared registry.
Predict first: what exactly is one row?
Each Criteo row is an observed impression. label=1/0 means clicked/unclicked, unlike an unobserved recommendation interaction. There are 13 integer features and 26 anonymous categorical fields. Their meanings are unpublished: do not rename them age, advertiser, or user. Category values are anonymous hashes, not ordered continuous numbers.
Read only the first 40,000 rows of the same fixed-SHA shard. X has shape [N,39], y [N]. Predict the positive rate and the numeric column with most missing values, then execute. This click rate describes a sampled log, not population CTR.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
from pathlib import Path
import sys
ROOT=Path.cwd()
if not (ROOT/'reclab').is_dir(): ROOT=ROOT.parent
sys.path.insert(0,str(ROOT))
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display
from reclab.ads import latest, load_data, split_groups, NUM, CAT, FEATURES
from reclab.paths import ARTIFACTS
from reclab.probability import evaluate
preview=ARTIFACTS/'notebook-previews';preview.mkdir(parents=True,exist_ok=True)
frame,total=load_data(40000)
y=frame.label.to_numpy()
print('X shape:',frame[FEATURES].shape,'y shape:',y.shape,'shard rows:',total)
print('Actual clicked:',int(y.sum()),'sample CTR:',float(y.mean()))
display(pd.DataFrame({'missing_rate':frame[NUM].isna().mean(),'dtype':frame[NUM].dtypes.astype(str)}))Saved notebook output · not executed on this page
X shape: (40000, 39) y shape: (40000,) shard rows: 766864 Actual clicked: 1200 sample CTR: 0.03
| missing_rate | dtype | |
|---|---|---|
| integer_feature_1 | 0.187575 | float64 |
| integer_feature_2 | 0.093375 | float64 |
| integer_feature_3 | 0.249075 | float64 |
| integer_feature_4 | 0.445175 | float64 |
| integer_feature_5 | 0.411575 | float64 |
| integer_feature_6 | 0.104150 | float64 |
| integer_feature_7 | 0.032600 | float64 |
| integer_feature_8 | 0.000000 | float64 |
| integer_feature_9 | 0.000000 | float64 |
| integer_feature_10 | 0.104150 | float64 |
| integer_feature_11 | 0.411575 | float64 |
| integer_feature_12 | 0.011525 | float64 |
| integer_feature_13 | 0.249075 | float64 |
Missingness is an input state, not a non-click
A missing numeric value differs from zero. Estimate medians on train only; missingness indicators let a linear model recognize imputation. Trees can learn missing-value routing. Empty categories receive an explicit marker; training-unseen values use an other bucket.
Predict: if rows with more missing fields have lower CTR, does missingness cause non-clicks? No: impression source, users, and context are undisclosed. Segmentation by missing count below is error diagnosis only.
missing=frame[NUM].isna().sum(axis=1)
display(pd.DataFrame({'missing':missing,'label':y}).groupby('missing').label.agg(['count','mean']))
fig,ax=plt.subplots(figsize=(8,3));ax.hist(missing,bins=np.arange(15)-.5)
ax.set(xlabel='Missing numeric fields per exposure',ylabel='Exposure count',title='Real 40k prefix: missingness')
fig.tight_layout();fig.savefig(preview/'19_data.png',dpi=130);plt.show()Saved notebook output · not executed on this page
| count | mean | |
|---|---|---|
| missing | ||
| 0 | 11173 | 0.034458 |
| 1 | 6562 | 0.028345 |
| 2 | 4131 | 0.030743 |
| 3 | 9145 | 0.024057 |
| 4 | 3130 | 0.030990 |
| 5 | 2364 | 0.032149 |
| 6 | 1304 | 0.039110 |
| 7 | 840 | 0.023810 |
| 8 | 732 | 0.023224 |
| 9 | 403 | 0.034739 |
| 10 | 168 | 0.041667 |
| 11 | 48 | 0.000000 |
<Figure size 800x300 with 1 Axes>
Change one variable: labels must not change splits
Random row splitting can place identical feature vectors in both training and evaluation. This implementation hashes all 39 features into groups, excluding label, so a group always stays in one partition. It prevents exact-vector duplicate leakage, not user, session, or temporal dependence. No public user IDs or reliable shard timestamps are available, so user/time isolation cannot be claimed.
Compute the original split, then flip only labels. Partitions must remain identical. Explain the roles of train, select, calibration, and audit before seeing results.
masks,groups=split_groups(frame)
flipped=frame.copy();flipped['label']=1-flipped.label
changed,newgroups=split_groups(flipped)
assert np.array_equal(groups,newgroups)
for a,ma in masks.items():
for b,mb in masks.items():
if a!=b: assert not set(groups[ma])&set(groups[mb])
display(pd.DataFrame({name:{'rows':int(mask.sum()),'clicks':int(y[mask].sum())} for name,mask in masks.items()}).T)
print('Label flip leaves all feature-group partitions unchanged.')Saved notebook output · not executed on this page
| rows | clicks | |
|---|---|---|
| train | 22024 | 677 |
| select | 6006 | 194 |
| calibration | 6008 | 161 |
| audit | 5962 | 168 |
Label flip leaves all feature-group partitions unchanged.
Duplicate-copy stress test: explicitly constructed inputs
The original prefix may contain very few duplicates. Artificially copy 100 feature rows to test the rule; do not count them as real impressions. Change only the splitting unit: random rows versus identical-feature groups. Random splitting scatters copies, while grouping keeps them together. This checks a program invariant, not a new data benchmark.
toy=pd.concat([frame.iloc[:500],frame.iloc[:100]],ignore_index=True)
gm,gh=split_groups(toy)
rng=np.random.default_rng(42);random_train=rng.random(len(toy))<.55
random_overlap=set(gh[random_train])&set(gh[~random_train])
print('Constructed duplicate stress test, random-row group overlap:',len(random_overlap))
assert not set(gh[gm['train']])&set(gh[~gm['train']])
print('Grouped overlap: 0; synthetic copies are excluded from all reported real metrics.')Saved notebook output · not executed on this page
Constructed duplicate stress test, random-row group overlap: 43 Grouped overlap: 0; synthetic copies are excluded from all reported real metrics.
Vocabularies also obey training boundaries
Learn category mappings only from train. Inspecting evaluation data to choose frequent categories changes the learning protocol. The original model retains at most 255 frequent training categories per field, mapping rare/unseen values to 0. Predict whether more retained categories must help, then inspect coverage. Run budget experiments on select, not repeated audit selection.
Exercise: change only limit from 32 to 64 and inspect unknown/other rates and encoded dimension. Do not adjust split thresholds to obtain attractive CTRs. Explain why every field being seen does not mean the user was seen.
from reclab.ads import vocabulary,encode
limit=32 # Exercise: change only this to 64.
vocab=vocabulary(frame.loc[masks['train']],limit)
x=encode(frame,vocab)
assert x.shape==(len(frame),39)
print('Encoded shape:',x.shape,'category other fraction on select:',float((x[masks['select'],13:]==0).mean()))
print('Interpret: fixed width does not imply fixed information; vocabulary breadth changes coverage.')Saved notebook output · not executed on this page
Encoded shape: (40000, 39) category other fraction on select: 0.2871423448346525 Interpret: fixed width does not imply fixed information; vocabulary breadth changes coverage.