Distributed Data Systems LabLynn Lee / Engineering notebook

Primary sources / Verified 2026-11-14

Read with purpose.
Test the claims.

Two real university courses, specific lectures and original runnable exercises mapped to all six units. Use this as a study route, not a course enrollment or assignment-solution service.

Only link public source materials. Do not mirror course content or publish official assignment solutions. Public reading access is separate from enrollment, assessment and fees. Reading resources do not imply completed exercises.
Carnegie Mellon University

15-445/645 / Intro to Database Systems

Fall 2025 · published archive

Programming/debugging in C++; course projects use C++20. This Lab uses Python rather than reproducing BusTub assignments.

Linked slides/notes are publicly readable. Formal campus enrollment, grading and any fees are separate and not arranged here. Public material access is not course credit.

Reading and solution policy

The official FAQ asks learners not to publish course project implementations. This companion contains separate original exercises only.

Official prerequisites / policy
Massachusetts Institute of Technology

6.5830/6.5831 / Database Systems

Spring 2026 · published 2026 materials

6.1210 (formerly 6.006) or equivalent algorithms; familiarity with computer systems such as 6.1800 (formerly 6.033). No prior database course required. Official labs use Go.

The selected PDFs are public without login. Campus enrollment, grading, attendance and fees are separate; no registration or paid access is promised.

Reading and solution policy

The syllabus prohibits publishing lab/problem-set solutions. The linked course is a reading source; our synthetic experiments are not its solutions.

Official prerequisites / policy

Six learning loops

01 / Read → predict → run → explain

Lecture #03 / Database Storage I · Lecture 4 / 2026-02-12 / Intro to DB Internals · File metadata and page headers / Parquet metadata

Inspect the real footer, identify what it knows about rows/chunks, then explain why it cannot tell you a replica count or table commit.

python lab.py parquet
python scripts/inspect_parquet.py build/parquet/sorted-4096-zstd.parquet

Total row-group rows agree with 131072; schema has four columns; storage/table authority is a reasoning question, not inferred from row-group count.

Open the complete unit →
02 / Read → predict → run → explain

Lecture #06 / Storage Models & Compression · Lecture 10 / 2026-03-10 / OLAP & Column Stores · File metadata and page headers / Parquet metadata

Predict candidate groups before comparing real read spans; change one predicate boundary in your own copy.

python lab.py parquet

All 45 queries return the same source-oracle rows; sorted 4096 layout retains 2/32 candidates while shuffled retains 32/32.

Open the complete unit →
03 / Read → predict → run → explain

Lecture 15 / 2026-04-07 / Distributed Databases · Lecture #23 / Distributed Database Systems I · NameNode and DataNodes; Data Replication; Robustness / HDFS Architecture 3.4.2 · 2003 classic; architecture and fault-tolerance sections / The Google File System

Count blocks, stop one DataNode and verify reading and restored redundancy separately.

python scripts/bootstrap.py hdfs
python lab.py hdfs

Thirteen blocks; RF=3 on four starting nodes; after one stops, three live replicas per block and zero pending/under-replicated counters.

Open the complete unit →
04 / Read → predict → run → explain

Lecture #05 / Database Storage II · Lecture #20 / Multi-Version Concurrency Control · Lecture #21 / Database Logging · Lecture 12 / 2026-03-19 / Optimistic Concurrency Control, Snapshot Isolation

Read the primary-index path in the executed plan and the version/conflict schedule. Separately list what would be needed to test WAL durability or LSM compaction.

python scripts/bootstrap.py database
python lab.py database

Plan identifies accounts_pkey; observed serializable conflict is not a WAL fsync or compaction measurement. Those remain reading-only.

Open the complete unit →
05 / Read → predict → run → explain

Lecture 12 / 2026-03-19 / Optimistic Concurrency Control, Snapshot Isolation · Lecture 15 / 2026-04-07 / Distributed Databases · Lecture 16 / 2026-04-09 / Distributed Transactions · Lecture #20 / Multi-Version Concurrency Control

Run the transfer and stale-total write-skew schedule, then retry the entire business decision against the fresh total.

python scripts/bootstrap.py database
python lab.py database

KV Put trace reaches distinct serving nodes; transfer total=300; one real 40001; final total=220; the second withdrawal is declined.

Open the complete unit →
06 / Read → predict → run → explain

Lecture 20 / 2026-04-28 / Cloud DB · Schema evolution; Partition evolution; Query the data / PyIceberg API: schema evolution, partition evolution and scans · SQL Catalog; Development only / PyIceberg configuration: SQL Catalog / SQLite · Snapshots; Schemas; Partition specs; File metadata / Iceberg Table Specification · Lecture 17 / 2026-04-14 / Embeddings, RAG, and Vector DBs

Create real snapshots, rename a field by identity, evolve a partition spec and compare a pinned read with a naive directory listing.

pip install -r requirements-iceberg.txt
python lab.py iceberg

Snapshot rowsets 3/5/6; field ID 2 survives rename; file specs 0 and 1 coexist; old hashes persist; uncommitted id=999 stays outside the table.

Open the complete unit →

Search the source inventory

Download the structured inventory JSON

2026: dataset lineage for retrieval

A retrieval/embedding build should record its corpus snapshot, schema, transform version and model version. The Iceberg experiment verifies a stable corpus rowset hash and changing latest pointer; it does not generate embeddings or assess RAG.

MIT Spring 2026 / Lecture 17 / Embeddings, RAG, and Vector DBs

The GFS paper is a 2003 classic. MIT 2026 selected lectures are published PDFs, verified readable; no planned or speculative release is presented as completed. Tutorials may evolve; executed compatibility is recorded separately.