Each section below answers one big question about data engineering, and each page inside it answers one smaller question on the way there.
Streaming Pipelines
How do I build a correct, fault-tolerant streaming pipeline in Spark?
- What is a watermark, and what happens to late data?
- How does state cleanup work in a stream-stream join?
- How does Spark recover from a crash without duplicates?
Spark Performance
Why is my Spark job slow, and how do I find out?
- What is a shuffle, and why is it expensive?
- What is data skew, and how do I fix it?
- How does Spark choose how to join two tables?
Lakehouse Table Formats
How do table formats give a data lake database-like behavior?