HashtagCashtag is a big data pipeline designed for analyzing user sentiment on the US stock market. It aggregates Twitter and stock market data using a Lambda architecture, processing both batch and streaming data. The primary objective is to provide insights into stock trends and sentiment correlations. Apache Kafka, Spark, and Cassandra are utilized for data ingestion, processing, storage, and serving, respectively.
The project employs a Lambda architecture for both batch and stream processing, allowing for real-time and historical analysis. It utilizes Cassandra for efficient time-series data storage, optimized through careful schema design. The use of Azkaban for workflow scheduling offers visual interface, task parallelization, and failure recovery. A simplified sentiment analysis component demonstrates practical application of the pipeline.
- Data Ingestion: Pulls data from Twitter Streaming API and a third-party stock data provider (netfonds.no) using Python scripts.
- Real-time Analysis: Employs Spark Streaming for calculating real-time stock mentions and sentiment.
- Batch Processing: Leverages Spark for historical data aggregation, including mentions, sentiment, and stock data analysis.
- Time Series Storage: Uses Cassandra for efficient storage and retrieval of time-series data, essential for historical and real-time views.
- Stream Processing: Implements incremental algorithms in Spark Streaming to complement batch processing and provide faster insights.
- Scalability: Lambda architecture facilitates horizontal scaling for handling high data volumes.
- Developer Experience: Azkaban provides a user-friendly interface for managing and monitoring the data pipeline workflows.
HashtagCashtag is a functional prototype showcasing a complete data pipeline. The project has a sample of readily available code for data ingestion, batch processing, stream processing and serving. While the codebase is functional, it's currently limited to a specific set of stocks due to API rate limits. The documentation is comprehensive, providing detailed explanations of data ingestion, batch and speed layer operations, and Cassandra schema design.
This project is beneficial for data scientists and analysts seeking to build real-time sentiment analysis pipelines for financial markets. It addresses the need for correlating social media sentiment with stock price fluctuations. HashtagCashtag provides a robust and scalable framework for analyzing time-series data, offering a practical alternative to manual data collection and analysis methods.