Course + reproducible experiments

Think in work,
data movement
and recovery.

A foundation-first course in distributed computation. Read the ideas, predict a result, inspect real Spark evidence, then change the experiment.

Start the course →Inspect evidence

Enter at the right depth

New to distributed computing? Know loops, functions, key/value pairs and basic sums; read Setup, then Chapters 01–07 in order. Already use an engine? Begin with the prediction and exit task of each chapter, inspect the saved plan, then reproduce before skipping ahead. No domain-specific experience is required.

Verified courses → executable exercises New: final AQE plan walkthrough

The core learning path

01 / CORE

Why parallelize?

Work, slots, overhead and correctness

02 / CORE

Data-parallel models

MapReduce, dependencies, stages and tasks

03 / CORE

Aggregate before moving

Mergeable summaries and Shuffle evidence

04 / CORE

Join strategies and skew

Broadcast, redistribution and salting tradeoffs

05 / CORE

Retry and recompute

Lineage, cache and checkpoint boundaries

06 / CORE

Resource diagnosis

Spill, task spread and performance reasoning

07 / CORE

Final adaptive plans

Coalescing, controls and independent verification

One complete learning loop

Explain the invariant → predict the communication → run the code → inspect tasks and plans → change one factor → defend the conclusion. Every chapter has a runnable example, hands-on task, interpretation and self-check answers.

Reproduce locally Download the companion

Original explanations with upstream papers and official references. Finite batch fundamentals; advanced distributed algorithms, multi-host scaling and production capacity planning are outside this edition.