Five experiments. One connected system.

01Establish a baseline

Complete the local setup and run ./stream-lab smoke before changing anything. Open the local query console, select the stated engine, and use the supplied read-only presets or the SQL below. Exact counts and timings vary because the producer is continuous and random.

For every case record the selected order ID, query time and output. “Expected” below describes intended behavior; it is not a claim that your environment has already passed.

02Trace one order

Question: where does one business record become several change events? Run the ClickHouse query below, select one order ID, and inspect the same order in the MySQL preset. In Kafka UI, inspect that key in dwd-order-wide.

Expected: the same order ID connects the source, Kafka key and serving row, while successive messages can describe different versions. Look at Flink job records-in/out to relate propagation to computation.

Interpretation: a message count is not an order count. This case verifies a chosen record’s path; it does not prove completeness for all records.

SELECT order_id, order_status, payment_status,
       order_amount, updated_at
FROM analytics.dwd_order_wide FINAL
ORDER BY updated_at DESC
LIMIT 10;

03Watch a delayed payment update a join

Question: how can a joined row exist before its payment? Run the MySQL query below, find a new order with no payment, and refresh. Compare it with the ClickHouse order view.

Expected: the source initially has no payment row; the joined view uses PENDING. The mock producer schedules later payments and may produce PAID or FAILED. The configured delay is a generator input, not a measured end-to-end latency guarantee.

Interpretation: the regular left join retains state and updates the keyed result. If you miss the transition, select another new order; do not infer that every order must become PAID.

SELECT o.order_id, o.order_status, p.payment_status,
       o.created_at, p.paid_at
FROM orders o
LEFT JOIN payments p ON o.order_id = p.order_id
ORDER BY o.updated_at DESC
LIMIT 10;

04Compare continuous detail with a bounded summary

Question: why can a fresh detail row be absent from the latest hourly summary? Record the latest order and its category, run ./stream-lab batch, then query the ClickHouse hourly table.

Expected: the bounded job groups rows by creation hour and category. gmv sums order_amount for all selected orders; paid_gmv includes only payment_status = PAID. These are different metrics.

Interpretation: batch results are refreshed after execution and downstream ingestion. The background loop waits five minutes after a batch attempt finishes; it is not a wall-clock five-minute SLA. Check job completion and the result version before comparing.

-- Then run in the ClickHouse console:
SELECT hour_bucket, category, order_count,
       paid_order_count, gmv, paid_gmv
FROM analytics.ads_hourly_sales FINAL
ORDER BY hour_bucket DESC, gmv DESC
LIMIT 20;

05Reconcile two lake-table counts

Question: what can a count comparison tell you? Run ./stream-lab batch, then inspect the latest audit row. The two inputs are the Fluss/Paimon union table and the independent Iceberg table.

Expected: count_delta is paimon_fluss_count minus iceberg_count. It may be nonzero while sinks expose different progress. Record both counts and audit_time; repeat after the pipeline catches up.

Interpretation: zero means those reads returned equal counts, not identical keys or field values. Persistent differences call for checkpoint, sink-commit and tiering inspection. Stronger verification requires a fixed cohort or common cutoff and field-level checks.

SELECT audit_time, paimon_fluss_count, iceberg_count, count_delta
FROM analytics.lake_consistency_audit FINAL
ORDER BY audit_time DESC
LIMIT 1;

06Recover from a controlled worker interruption

Question: does recovered computation produce the expected records? First confirm that the streaming jobs and completed checkpoints exist. Keep this experiment within the local lab and record the baseline.

Use the commands below to interrupt the TaskManager briefly and restart it. Observe the job transition and checkpoint history in the local Flink UI. Then run smoke and batch checks and revisit your recorded order IDs.

Expected: the configured restart strategy attempts recovery. A task failure within a running session is different from recreating the entire cluster. This deployment does not configure JobManager high availability, and retaining Docker volumes alone does not prove that every reopened job resumes from its former checkpoint.

Interpretation: capture the actual recovery result, including failures. Do not describe a healthy process or a zero count delta as proof of global exactly-once delivery.

./stream-lab compose stop taskmanager
# Inspect the failure in your local Flink UI.
./stream-lab compose start taskmanager
./stream-lab smoke
./stream-lab batch