Lead Data Engineer
Lead Data Engineer / Architect (Azure)
Relevant Experience required:
- 12+ years Responsibilities
- Design, build, and maintain scalable ETL/ELT pipelines using Azure Data Factory, Azure Synapse, and Azure Databricks.
- Architect and manage secure enterprise cloud storage, including ADLS Gen2, Delta Lake, and relational/NoSQL databases.
- Ingest and process both high-volume batch data and real-time streaming workloads.
- Optimize database queries, PySpark jobs, and data pipeline performance to minimize execution time and cloud costs.
- Collaborate with BI and analytics teams to model dimensional data marts (Star/Snowflake schemas) for reporting.
- Implement data governance, automated data quality validation, and end-to-end lineage tracking (e.g., via Microsoft Purview).Secure data pipelines and storage using Azure Role-Based Access Control (RBAC), Key Vaults, and encryption standards.
- Automate infrastructure deployments and code releases using Azure DevOps, GitHub Actions, and CI/CD best practices.