Two university courses supply complementary foundations: Berkeley CS267 for parallel machines, communication and load balance; MIT 6.5840 for distributed-system failure and MapReduce. Neither substitutes for the pinned Spark 3.5.8 implementation guides. Choose these relevant units instead of treating an entire graduate course as a prerequisite for this introductory Lab.
The reviewed catalogue below is searchable in both languages and downloadable as structured JSON. Each learning unit names a particular lecture, paper section or assignment specification and a separate original runnable exercise. Predict the result before running, verify the stated invariant, then explain the observed plan and counters. The mapping does not grant credit or reproduce a university assignment solution.
Verification date: 14 November 2026, Asia/Shanghai. Published material is distinguished from tentative schedules. The CS267 2016 archive is deliberately historical; its hardware figures are not 2026 capacity advice. The 2025 official page says its videos require a Berkeley login. MIT's selected Spring 2026 Lecture 1 notes and Lab 1 specification are already published, while its schedule retains a tentative label. Public reading access is separate from enrollment, grading and tuition. No private course account or paid material is used.
From the project root, search without a browser or service:
MIT's collaboration policy forbids publishing lab solutions. Read the official policy before attempting its Go assignment privately. Our examples do not use its starter code or implement its assignment coordinator. Third-party material remains upstream under its own terms; this repository stores links, bibliographic facts and original learning tasks.
Prerequisites: Conventional programming and mathematics for the application algorithms; this is a parallel-computing course, not a Spark API course.
Public reading: Course outline and linked lecture PDFs are readable without an account. We checked the selected PDFs, not every archived video.
Enrollment / fees: Self-study of these public files needs no registration or purchase. University credit, staff support and historical machine accounts are separate; no current enrollment or fee offer is asserted.
Edition and limits: 2016 archive. Hardware and cloud figures are historical; retain the communication, granularity and locality questions. The 2025 official page exists but says videos require a Berkeley login; it is not substituted for this open archive.
published materials (schedule labels itself tentative) · Verified 2026-11-14
Prerequisites: Official prerequisites: 6.1910 (6.004), plus 6.1800 (6.033) or 6.1810, or equivalent; substantial programming experience helps. Lab 1 uses Go.
Public reading: Lecture 1 notes, schedule, MapReduce paper and Lab 1 specification are public. Selected 2026 materials were available on the verification date; a tentative schedule alone is not proof of a future lecture.
Enrollment / fees: Reading the public pages is free and needs no course login. MIT assessment/credit and course systems require their own enrollment; this Lab offers neither. Fees were not independently verified.
Edition and limits: 2026 teaching materials include the 2004 MapReduce classic. Our Spark task exception exercise is narrower than MIT's multi-process worker-crash assignment.
Predict first: Predict the hottest output partition under 80% hot-key skew; what extra work does salting add?
Evidence to defend: All variants: 240000 rows, weighted sum 384001; compare partition histograms and SortMergeJoin/BroadcastHashJoin plans. Keep the slower salted observation.
Limits: Unique dimension keys. The old join plans are pre-action strategy evidence, not final AQE plans; AQE skew join remains untested.
Original Lab exercise; upstream assignment solutions are not included.
Predict first: Predict why merely enabling AQE may leave 32 reducers, and whether 1-byte versus 1-MiB targets produce the same final plan.
Evidence to defend: Full per-key count/sum oracle; compare pre/final plans, AQEShuffleRead SQL metrics and actual TaskEnd stages. Run the 1-byte counterexample in the walkthrough.
Limits: Only post-shuffle coalescing is implemented. Join adaptation, distributed training and inference remain reading extensions.
Original Lab exercise; upstream assignment solutions are not included.
assignment specification · 2026 · Public reading / verified 2026-11-14
Read the specification only. Do not publish a solution; the upstream collaboration policy forbids it. Our exercises do not use the assignment starter or implement its coordinator.
AI extension / original paper · 2022 · Public reading / verified 2026-11-14
Published OSDI 2022. Read iteration-level scheduling for variable-length requests; no inference engine runs in this Lab.
AI reading bridge · not implemented
Training: read ZeRO (2019 preprint, selected 2020 v3). Relate state sharding to per-worker memory and collective communication; data shards and optimizer-state shards are different. Write which values each rank owns, and when they must move. Spark aggregation is a prerequisite reasoning exercise, not a ZeRO implementation or GPU benchmark.
Inference: read Orca (OSDI 2022), especially iteration-level scheduling. Relate variable request lengths to the long-tail and task-granularity questions from Join skew and AQE. Explain why a request can finish while its batch peers continue. Tail latency, token throughput, GPU collectives and serving correctness were not measured; this Lab runs no model and makes no LLM evaluation claim.
These are already published primary papers, not a forecast or a claim to cover the latest 2026 systems. No paid calls or credentials are needed.