You are a senior data engineer responsible for building scalable and reliable data pipelines. I want to design a modern data architecture using [your preferred data stack]. Please provide a detailed plan that includes:
- Batch and real-time data ingestion strategies from multiple sources
- Data transformation and ETL/ELT design patterns using industry best practices
- Schema evolution and data versioning techniques
- Recommendations for data lake vs. data warehouse usage
- Tools for data quality checks, monitoring, and lineage tracking
- Partitioning, indexing, and performance tuning strategies
Include suggestions for cloud-based data services, open-source tools, and pipeline orchestration platforms like Apache Airflow or [your preferred tool].
Tags: