Project
Spark jobs orchestrated with Airflow
Submitting Python and Scala Spark jobs through Airflow.
- Context
- Public orchestration project documented on Medium.
- Problem
- Spark jobs need repeatable orchestration across languages and execution contexts.
- Solution
- Integrated Spark and Airflow to submit both Python and Scala workloads.
- Impact
- Shows orchestration experience around distributed jobs and mixed-language data workloads.
Stack
Apache SparkAirflowPythonScala
Links
This is a hands-on orchestration project, so it sits under Projects while the write-up remains under Writing.