PySpark
Joins, aggregations, memory & skew, plan reading.
4 topic ladders · 3 tiers
Practice · Coding
Drill the rep.
Until hesitation goes.
PySpark and pandas drills against fictional power-grid datasets. Joins, aggregations, memory & skew, plan reading — Junior to Senior. Server-graded, deep-linked back into the lesson when you miss.
The same engines run in production at
NetflixUberShopifyLinkedIn
Why drills
Theory builds the map. Practice builds the reflex.
Every track is a fictional power-grid scenario at production scale. You write the transform, predict the plan, diagnose the stack trace — and the platform server-grades the answer the same way Spark would. Get it right and the kWh banks; get it wrong and you deep-link straight back into the lesson that taught the missing piece.
Three languages
One discipline.
Distributed pipelines, single-machine pandas, set-based SQL — the languages a data engineer reaches for, drilled with the same rigor on each.
Python
pandas the way data engineers actually write it.
1 topic ladder · 3 tiers
SQL
Joins, windows, dimensional modeling, query plans.
Catalogue in build
Inside the rep
Read. Reason. Submit.
Every question opens with the concept laid out, the artifact you are reading, and a single decision. No scaffolding tricks — the rep is what you would do at work, scaled down to one focused minute.
ContextDataset
~8 min
Aggregate Before Join, or Join Before Aggregate
Context
Two engineers solve the same problem: produce per-region average daily_kw with substation metadata. Snippet A joins meters (12M rows) to substations (120 rows) first, then aggregates. Snippet B aggregates meters first, then joins. Both produce identical output. The question is which is faster and why.
Evidence
Input sample
meters: 12M rows, key columns include substation_id, region, daily_kw. substations: 120 rows (~50KB), well below autoBroadcastJoinThreshold. Meters Size: 12M rows Substations Size: 120 rows (broadcast-eligible)
Snippet A — join first, then aggregate
result_a = (meters .join(substations, on='substation_id', how='inner') .groupBy('region', 'name', 'capacity_mw') .agg(avg('daily_kw').alias('avg_kw')))Snippet B — aggregate first, then join
result_b = (meters .groupBy('substation_id', 'region') .agg(avg('daily_kw').alias('avg_kw')) .join(substations, on='substation_id', how='inner'))Question
Which snippet is faster on a 12M-row meters dataset, and why?
- Snippet A is faster because the broadcast of substations happens once before any aggregation work; the post-join groupBy operates on enriched data without shuffling the substation columns separately
- Snippet A is faster because Spark cannot broadcast substations after a groupBy stage; the broadcast-eligibility check happens before the first wide transformation in the plan and a groupBy disqualifies later joins from broadcast
- Both are equivalent in Catalyst because the optimizer reorders aggregate-vs-join freely when the join key is the same as the groupBy key; the engineer's choice of order is decorative when statistics are accurate
- Snippet B is faster because the groupBy reduces meters from 12M rows to ~120 rows BEFORE the join — the join then operates on a tiny aggregated DataFrame; aggregation is a row-reducer and reducing before joining is almost always cheaper
Live preview · Mid Pyspark · PJSI1 — Plan Reading Foundations
By the numbers
173
Tasks
5
Topics
3
Tiers
2
Languages
SQL ladder coming soon. Computer Science, Logic, Math & Statistics drills landing after that.
Pick a topic. Drill the rep.
Free during beta. Sign in to start banking kWh.