01Before you begin
You should be comfortable with SELECT, GROUP BY, primary keys and a left join. Reading requires only a browser. Local experiments additionally need a terminal, Docker and authorized source access.
Use a learning notebook with four fields: prediction, action, observation, explanation. An expected result in this guide is a hypothesis to test, not a saved run result.
02Records and changes
Question: if an order is updated three times, how many business orders and how many change events exist? Predict the answer before inspecting the source and serving tables.
Read the MySQL source schema and CDC SQL, then run the order-tracing case. Explain why an initial table snapshot needs to connect to a position in the ongoing change log.
- Outcome: distinguish a current table from an append-only event history or keyed changelog.
- Check yourself: why is replaying a change twice different from inserting two distinct orders?
- Next step: observe one order before and after payment.
03State and time
Question: what must the join remember when an order arrives before its payment? Trace which row is emitted first and which fields change later.
The lab’s regular left joins also react to customer and product updates. They are not historical “as-of” dimension joins. Read the SQL before assuming an old order retains the dimension values it had at purchase time.
- Outcome: explain join state, keyed updates and the tradeoff introduced by state TTL.
- Check yourself: could a payment arriving after state expiry join as you expect?
- Extension: design a separate watermark/window exercise; these mechanisms are not implemented by the current SQL.
04Storage and serving
Question: why keep both a transport log and an analytical table? Compare Kafka message updates with ClickHouse queries over unique order keys.
Then compare a Fluss/Paimon union read with an independent Iceberg read. Ask what each query includes at its read point; do not infer a performance winner from this small local setup.
- Outcome: separate computation, transport, table storage and query serving responsibilities.
- Check yourself: why can a raw ClickHouse count differ from a FINAL count?
- Extension: add field-level comparisons over a fixed key set before drawing a consistency conclusion.
05Recovery and evidence
Question: after a worker restart, what would prove the job recovered correctly? A green container is only one observation. You also need checkpoint history, job progress and checked output.
First obtain a healthy baseline. During a controlled local failure, record the last completed checkpoint, affected job and a small set of order IDs. After recovery, inspect those same records and rerun the batch audit.
- Outcome: separate an available service from recovered computation and verified results.
- Check yourself: can equal counts prove equal rows? What could both sinks be missing?
- Extension: define a common cutoff, compare keys and amounts, and test deletes independently.
06Your completion artifact
Write a short explanation of one order’s journey, attach the queries you used, record the code revision and run time, and state at least one limitation of the evidence. You have completed the learning path when another person can follow your observations and reproduce your checks.