24 · Budget pacing, constraints, and integrated error analysis
Chapter 18 remains the overview; this chapter isolates one question. The formal 200k experiment is preserved. Teaching runs use separate records in the shared registry.
Predict first: why can equal total budgets produce different delivery times?
Without pacing, a strategy checks only remaining total budget and may exhaust it early. Uniform pacing additionally enforces cumulative spend≤B×request progress. It reads only current request position and spending, not future clicks or true_p. Later traffic is more valuable by construction here; uniform pacing may help here and lose value elsewhere.
Fix seed, value bidding, multiplier 1, and budget 60; toggle only pacing. Compare the same potential market, verify cumulative constraints, and connect the error boundaries of real click prediction and simulated decisions.
English companion. All instructional Markdown is translated. Experiment code, Chinese language examples, execution counts, and saved outputs remain as in the source edition; historical output text may therefore be Chinese. The initial setup cell anchors the working directory and Python import path for this companion's deeper folder; it does not change the experiment. Saved outputs are historical evidence, not a new run. Run from the top after preparing the required data and models.
# English companions live one directory deeper; keep the original experiment code unchanged.
from pathlib import Path as _TranslationPath
import os as _translation_os
import sys as _translation_sys
_translation_root = next((p for p in [_TranslationPath.cwd(), *_TranslationPath.cwd().parents]
if (p / "reclab").is_dir()), None)
if _translation_root is None:
raise RuntimeError("Open this notebook from inside the Search Recommendation Ads Lab checkout.")
_translation_os.chdir(_translation_root)
if str(_translation_root) not in _translation_sys.path:
_translation_sys.path.insert(0, str(_translation_root))
from pathlib import Path
import sys
ROOT=Path.cwd()
if not (ROOT/'reclab').is_dir(): ROOT=ROOT.parent
sys.path.insert(0,str(ROOT))
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display
from reclab.ads import latest, load_data, split_groups, NUM, CAT, FEATURES
from reclab.paths import ARTIFACTS
from reclab.probability import evaluate
preview=ARTIFACTS/'notebook-previews';preview.mkdir(parents=True,exist_ok=True)
from reclab.auction import market,simulate
synthetic=market(42)
policies={'unpaced':simulate(synthetic,paced=False),'paced':simulate(synthetic,paced=True)}
display(pd.DataFrame([{'policy':name,**{k:r[k] for k in ['spend','wins','clicks','expected_click_value','expected_advertiser_surplus','budget_ok']}} for name,r in policies.items()]))
print('Input probabilities:',synthetic['estimate'].shape,'budgets:',[60]*3)Saved notebook output · not executed on this page
| policy | spend | wins | clicks | expected_click_value | expected_advertiser_surplus | budget_ok | |
|---|---|---|---|---|---|---|---|
| 0 | unpaced | 179.950958 | 1654 | 193 | 325.099130 | 145.148171 | True |
| 1 | paced | 179.352946 | 2522 | 181 | 363.687988 | 184.335042 | True |
Input probabilities: (6000, 3) budgets: [60, 60, 60]
Constraints are per-request invariants, not end-of-run checks
Checking only final spend≤budget can miss overspend followed by correction. Every allocation checks payment≤bid; every request checks cumulative spend≤total budget. With pacing, it also checks the released budget. Curves show 20 checkpoints, but internal assertions cover all 6,000 requests.
Predict when the unpaced curve approaches its limit, then relate the plot to the assumed later value increase.
fig,ax=plt.subplots(figsize=(8,3))
for name,r in policies.items():
t=np.array([x['fraction'] for x in r['trace']]);spend=np.array([x['spend'] for x in r['trace']])
assert (spend<=60+1e-8).all()
if name=='paced':assert (spend<=60*t[:,None]+1e-8).all()
ax.plot(t,spend.sum(axis=1),label=name)
ax.plot([0,1],[0,180],'k--',label='Paced combined ceiling')
ax.set(xlabel='Request fraction',ylabel='Simulated cumulative spend',title='Only pacing changed');ax.legend();fig.tight_layout();fig.savefig(preview/'24_pacing.png',dpi=130);plt.show()Saved notebook output · not executed on this page
<Figure size 800x300 with 1 Axes>
Change one variable: a tighter budget
Keep pacing on and reduce budget from 60 to 30, holding market, multiplier, and strategy fixed. Click value need not fall proportionally: eligibility and competition change. Each advertiser must satisfy its own budget; aggregate spend cannot cancel an individual overspend.
This cell reruns the simulation, not a prewritten table. Try budget=10, record lost impressions, spend, and value, and explain why this is not a change in model AUC.
budget=30. # Exercise: change only this value to 10.
tight=simulate(synthetic,budget=budget,paced=True)
display(pd.DataFrame(tight['advertisers']))
assert all(r['spend']<=budget+1e-8 for r in tight['advertisers'])
assert all(max(t['spend'])<=budget*t['fraction']+1e-8 for t in tight['trace'])
print('Tighter budget:',{k:tight[k] for k in ['spend','wins','expected_click_value','budget_ok']})Saved notebook output · not executed on this page
| spend | wins | clicks | |
|---|---|---|---|
| 0 | 29.842661 | 508 | 31 |
| 1 | 29.837349 | 489 | 27 |
| 2 | 29.819974 | 453 | 28 |
Tighter budget: {'spend': 89.49998375283522, 'wins': 1450, 'expected_click_value': 173.46772010420707, 'budget_ok': True}
The integrated error chain: scores → eligibility → payment
The real task measures discrimination and probability accuracy on one sampled distribution. It has no bidders, reserve prices, budgets, or propensities. The simulation studies allocation under these assumptions, not off-policy evaluation on real Criteo data.
Predict: if estimates increase by 1.5 while true probabilities and potential clicks stay fixed, should budget constraints break? They must always hold, while allocation and value may change. This is an explicit simulated prediction-bias stress test, not an estimate of real CTR calibration.
biased={**synthetic,'estimate':np.clip(synthetic['estimate']*1.5,.001,.99)}
bias_run=simulate(biased,paced=True)
assert bias_run['budget_ok']
display(pd.DataFrame([{'prediction':name,**{k:r[k] for k in ['spend','wins','expected_click_value','expected_advertiser_surplus','budget_ok']}} for name,r in [('original',policies['paced']),('overestimated',bias_run)]]))
print('Only synthetic estimated probabilities changed; true probabilities and potential clicks are shared.')Saved notebook output · not executed on this page
| prediction | spend | wins | expected_click_value | expected_advertiser_surplus | budget_ok | |
|---|---|---|---|---|---|---|
| 0 | original | 179.352946 | 2522 | 363.687988 | 184.335042 | True |
| 1 | overestimated | 179.076553 | 2366 | 277.435022 | 98.358469 | True |
Only synthetic estimated probabilities changed; true probabilities and potential clicks are shared.
Return to one system: layered acceptance and next steps
Data: real impression labels, pinned sources, hashes, and training boundaries. Prediction: actual training, select-based choice, independent calibration, and preserved audit negative results. Decision: simulation assumptions plus payment and budget invariants. Domains share experiment lifecycle and workbench, not inapplicable labels or metrics.
Confirm teaching runs did not replace the formal ads run, then inspect its prespecified segments and calibration uncertainty. Write a failure diagnosis distinguishing observations, assumptions, and needed new data. Never combine NDCG, AUC, and simulated revenue into one comparable leaderboard.
from reclab.experiments import catalog
receipt,result=latest()
print('Formal 200k run:',receipt['run_id'])
print('Workshop records:',[(r['run_id'],r['config']['workshop'],r['state']) for r in catalog() if r['domain']=='ads' and r['config'].get('workshop')])
display(pd.DataFrame([{'segment':s['segment'],'n':s['raw']['n'],'CTR':s['raw']['click_rate'],'raw_logloss':s['raw']['logloss'],'sigmoid_logloss':s['sigmoid']['logloss']} for s in result['segments']]))
print(result['paired_sigmoid'])
print('Acceptance: budget invariants passed; real and synthetic claims remain separated; negative model/calibration results retained.')Saved notebook output · not executed on this page
Formal 200k run: 20261030T125738-f82b6582
Workshop records: [('20261030T130907-22ac6037', 'linear_features', 'completed'), ('20261030T130911-14b5a3de', 'tree_capacity', 'completed')]
| segment | n | CTR | raw_logloss | sigmoid_logloss | |
|---|---|---|---|---|---|
| 0 | low_numeric_missing | 17016 | 0.036495 | 0.147043 | 0.146935 |
| 1 | high_numeric_missing | 12912 | 0.028733 | 0.121515 | 0.121515 |
| 2 | any_unseen_category | 11668 | 0.035996 | 0.146929 | 0.146712 |
| 3 | all_categories_seen | 18260 | 0.031325 | 0.129065 | 0.129103 |
{'calibrated_minus_raw_logloss': -6.130390957040765e-05, 'iid_row_bootstrap_95': [-0.00016953362229380587, 6.253575759107997e-05], 'replicates': 300, 'limitation': 'conditional on fitted models; repeated/related rows and one shard limit independence and generalization'}
Acceptance: budget invariants passed; real and synthetic claims remain separated; negative model/calibration results retained.