Project
Smart City real-time data engineering on AWS
Real-time mobility pipeline using Kafka, Spark, S3, Glue, Redshift, Lambda and Power BI.
- Context
- Public streaming architecture project documented on Medium.
- Problem
- Route, weather and incident data need to be streamed, processed and made available for analytics.
- Solution
- Built a streaming pipeline with Kafka and Spark, persisted data in S3, cataloged it with Glue, queried it through Redshift and visualized it in Power BI.
- Impact
- Shows practical orchestration of a multi-service streaming analytics architecture.
Stack
AWSKafkaSparkS3GlueRedshiftPower BI
Links
This is listed as a project because it has a public repo and a detailed architecture write-up.