01Follow the data
The continuous path is synthetic traffic → MySQL → Flink CDC → Fluss source tables → Flink regular joins. The joined result fans out to a Fluss wide table, an independent Iceberg table and a keyed Kafka topic. Kafka feeds ClickHouse through Kafka Engine tables and materialized views.
Fluss tiers its wide table into Paimon. Bounded Flink jobs read the combined Fluss/Paimon table for hourly aggregation and compare its count with Iceberg. Those batch results also travel through Kafka into ClickHouse.
02Who owns what?
| Layer | Implementation | Responsibility |
|---|---|---|
| Business source | MySQL retail tables | Customers, products, orders and one payment per order; row Binlog changes |
| Change capture | Flink CDC job 01 | Initial snapshot and incremental changes into primary-key tables |
| Streaming storage | Fluss retail tables | Maintain current keyed records and expose changes to downstream computation |
| Continuous computation | Flink SQL job 02 | Regular left joins maintain a denormalized order view |
| Distribution and serving | Kafka → ClickHouse | Keyed result transport, materialized ingestion, analytical queries |
| Lake storage | Paimon and Iceberg | Paimon is Fluss tiered storage; Iceberg is a separate v2 upsert sink |
| Bounded computation | Flink SQL jobs 10 and 11 | Hourly aggregates and a minimal cross-lake count comparison |
| Supporting infrastructure | RustFS, ZooKeeper, local console | Object storage, Fluss coordination and local inspection |
03Three data shapes to keep separate
- A business table answers “what is true now?” A Binlog records how database rows changed.
- A keyed changelog can contain multiple versions of one order. Counting messages does not count distinct orders.
- A lake query observes committed table state. Different sinks may expose a change at different times.
04Exactly-once is a boundary, not a global label
The source code configures Flink checkpoints and transactional delivery for the realtime Kafka sink. That configuration alone does not prove a lossless end-to-end pipeline. Recovery, consumer isolation, sink commit behavior and the final query must all be checked.
ClickHouse stores successive versions with ReplacingMergeTree(version), and the dashboard uses FINAL for query-time deduplication. The current Kafka ingestion does not translate tombstones into ClickHouse row deletions. Source deletion, serving retention and lake snapshot cleanup are distinct concerns.
The audit compares counts from separately read tables. Equal counts can hide changed values or different keys; a nonzero delta can reflect progress at different commit points.
05Storage and time choices in this implementation
Paimon uses the lab’s S3-compatible object store through Fluss tiering. Iceberg uses a Hadoop catalog on a shared Docker volume. The architecture therefore demonstrates two lake paths; it does not demonstrate Iceberg on S3 or a shared production catalog.
The joins are regular dynamic-table joins with a seven-day state TTL. No watermark, event-time window or explicit late-event side output is defined. TTL bounds idle state, and can change the result of a later update after state has expired.
Most SQL and ClickHouse timestamp fields use Asia/Shanghai. Treat that as the lab’s declared business time zone; changing only the container TZ variable does not migrate all query semantics.
06Find the contract in the source
| Question | Source file |
|---|---|
| What creates the traffic? | apps/mock/main.py |
| What enters CDC? | sql/01_cdc_to_fluss.sql |
| How is an order joined and distributed? | sql/02_wide_pipeline.sql |
| What does the batch job calculate? | sql/10_batch_aggregate.sql |
| What does the audit actually compare? | sql/11_lake_audit.sql |
| How does ClickHouse store versions? | clickhouse/init/001_analytics.sql |
| Which images and persistent volumes run? | compose.yml |