21 · Nonlinearity, feature crosses, and tree capacity
Chapter 18 remains the overview; this chapter isolates one question. The formal 200k experiment is preserved. Teaching runs use separate records in the shared registry.
Predict first: where does an additive model fail?
A linear logit expresses additive feature relationships. Cross terms add joint-field states to the design matrix; trees express interactions through conditional paths. Interactions increase expressiveness but may require more data and overfit. A task being called CTR does not guarantee deeper models are better.
Check expressiveness on explicitly constructed XOR data, then change only tree leaves from 7 to 15 on 40k real impressions. These are separate mechanism and real-training experiments, not one benchmark.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
from pathlib import Path
import sys
ROOT=Path.cwd()
if not (ROOT/'reclab').is_dir(): ROOT=ROOT.parent
sys.path.insert(0,str(ROOT))
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display
from reclab.ads import latest, load_data, split_groups, NUM, CAT, FEATURES
from reclab.paths import ARTIFACTS
from reclab.probability import evaluate
preview=ARTIFACTS/'notebook-previews';preview.mkdir(parents=True,exist_ok=True)
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
rng=np.random.default_rng(42)
x=rng.integers(0,2,size=(4000,2));y=(x[:,0]!=x[:,1]).astype(int)
print('Constructed XOR X:',x.shape,'y:',y.shape)
plain=LogisticRegression(C=10).fit(x[:3000],y[:3000])
cross=make_pipeline(PolynomialFeatures(degree=2,interaction_only=True,include_bias=False),LogisticRegression(C=10)).fit(x[:3000],y[:3000])
for name,model in [('additive',plain),('one_interaction',cross)]:
r=evaluate(y[3000:],model.predict_proba(x[3000:])[:,1]);print(name,{k:r[k] for k in ['auc','logloss']})Saved notebook output · not executed on this page
Constructed XOR X: (4000, 2) y: (4000,)
additive {'auc': 0.5427435387673956, 'logloss': 0.6928292897764964}
one_interaction {'auc': 1.0, 'logloss': 0.004364692561560054}
Cross-feature shapes and interpretation limits
Two inputs with interaction_only produce [x1,x2,x1*x2]. This is not full DeepFM or DCNv2; it checks why interactions may help. With d fields, pairwise interactions number d(d−1)/2. Crossing every high-cardinality one-hot feature can easily exceed local resources.
Predict the new column's values for all four inputs, then enumerate to verify. Consider a rare real cross with only one or two positive examples.
grid=np.array([[0,0],[0,1],[1,0],[1,1]])
transformed=cross[0].transform(grid)
display(pd.DataFrame(transformed,columns=['x1','x2','x1*x2']))
assert transformed.shape==(4,3)
assert np.array_equal(transformed[:,2],grid[:,0]*grid[:,1])
print('All second-order interactions for d=39:',39*38//2)Saved notebook output · not executed on this page
| x1 | x2 | x1*x2 | |
|---|---|---|---|
| 0 | 0.0 | 0.0 | 0.0 |
| 1 | 0.0 | 1.0 | 0.0 |
| 2 | 1.0 | 0.0 | 0.0 |
| 3 | 1.0 | 1.0 | 1.0 |
All second-order interactions for d=39: 741
Real training: change only leaf count
Fix raw numeric features, top 255 training categories, 80 trees, learning rate .05, minimum 100 samples per leaf, and L2=1. Change only num_leaves. Categories use categorical splits, not ordered integer codes. The budget is smaller than the formal 200k experiment, so compare only these two arms.
Depth and leaf count differ: 15 leaves do not imply every path has depth four. Training and evaluation use the shared registry and select, not audit.
from reclab.ads_teaching import workshop
run_id,comparison=workshop('tree_capacity')
print('Shared workshop:',run_id,comparison['shapes'])
table=pd.DataFrame([{'method':r['method'],'fit_seconds':r['fit_seconds'],**{k:r['select'][k] for k in ['auc','logloss','brier','ece']}} for r in comparison['rows']])
display(table)
fig,axes=plt.subplots(1,2,figsize=(9,3));axes[0].bar(table.method,table.auc);axes[1].bar(table.method,table.logloss)
axes[0].set(ylabel='Select AUC');axes[1].set(ylabel='Select logloss');fig.suptitle('Real click task: capacity is the only changed variable');fig.tight_layout();fig.savefig(preview/'21_trees.png',dpi=130);plt.show()Saved notebook output · not executed on this page
Shared workshop: 20261030T130911-14b5a3de {'raw': [40000, 39], 'train': 22024, 'select': 6006}
| method | fit_seconds | auc | logloss | brier | ece | |
|---|---|---|---|---|---|---|
| 0 | leaves_7 | 0.126307 | 0.704286 | 0.134063 | 0.030587 | 0.003655 |
| 1 | leaves_15 | 0.175748 | 0.688730 | 0.137054 | 0.030825 | 0.007551 |
<Figure size 900x300 with 2 Axes>
Explain results with loss, not complexity alone
This leaf-count comparison isolates capacity on this data prefix. Comparing its run time to different models in chapter 20 is not a rigorous architecture benchmark. In the formal 200k experiment, the linear model beats both tree candidates, and 160 trees do not beat 80. The negative result is preserved.
Read formal aggregates only to understand why complexity need not help. Do not retune the current model and reuse the same audit to claim improvement.
formal_receipt,formal=latest()
display(pd.DataFrame([{'method':r['method'],'audit_auc':r['audit']['auc'],'audit_logloss':r['audit']['logloss']} for r in formal['rows'] if r['method'] in ['linear_onehot','gbdt_80','gbdt_160']]))
print('Formal run unchanged:',formal_receipt['run_id'])
print('Workshop select and formal audit have different sample/fit budgets; no mixed leaderboard.')Saved notebook output · not executed on this page
| method | audit_auc | audit_logloss | |
|---|---|---|---|
| 0 | linear_onehot | 0.721468 | 0.135423 |
| 1 | gbdt_80 | 0.706329 | 0.136029 |
| 2 | gbdt_160 | 0.703035 | 0.137404 |
Formal run unchanged: 20261030T125738-f82b6582 Workshop select and formal audit have different sample/fit budgets; no mixed leaderboard.
Exercise: interactions are not a universal solution
Change only constructed labels from XOR to x1, preserving inputs and training budget. Compare with and without the cross. If the target is additive, extra interactions may not help. These are constructed mechanism labels, not altered real click labels.
Distinguish expressiveness, statistical error, and optimization budget in your explanation. To study real categorical interactions, constrain vocabularies on train, define a new select protocol, and reserve a fresh final audit.
target=x[:,0].copy() # Only the synthetic target mechanism changed.
for name,model in [('additive',LogisticRegression(C=10)),('interaction',make_pipeline(PolynomialFeatures(2,interaction_only=True,include_bias=False),LogisticRegression(C=10)))]:
model.fit(x[:3000],target[:3000]);r=evaluate(target[3000:],model.predict_proba(x[3000:])[:,1]);print(name,r['auc'],r['logloss'])Saved notebook output · not executed on this page
additive 1.0 0.0009870029880096182 interaction 1.0 0.0009621421929528383