Project
Real-time sentiment pipeline with Spark, OpenAI, Kafka and Elasticsearch
Streaming sentiment analysis pipeline using Spark, OpenAI, Kafka and Elasticsearch.
- Context
- Public streaming data engineering project documented on Medium.
- Problem
- Customer review streams need enrichment, classification and search-oriented storage.
- Solution
- Built a real-time flow from TCP socket ingestion to Spark processing, OpenAI sentiment classification, Kafka messaging and Elasticsearch indexing.
- Impact
- Shows streaming ingestion, AI enrichment and operational indexing in one pipeline.
Stack
PythonSpark StreamingOpenAIKafkaElasticsearchDocker
Links
This is listed as a project because it links to a public implementation and an end-to-end technical write-up.