Lead Data Engineer

We're working with a leading global specialty insurance firm and wholesale brokerage disruptor on this exciting opportunity. We are seeking a powerhouse technical leader to architect and scale sophisticated data environments within the Azure ecosystem, transforming raw data into high-value business intelligence.As the Lead Data Engineer, you will own the Databricks strategy and lead a high-performing team to build resilient, real-time data pipelines. This is a high-impact role where you will bridge the gap between complex engineering—utilizing Delta Lake, Azure Data Factory, and PySpark—and strategic business objectives in a fast-paced, remote-friendly environment.The Role Lead the design and implementation of scalable data pipelines using Azure Data Factory and Databricks notebooks for both batch and real-time big data processing. Architect and manage advanced data storage solutions using Delta Parquet files in Azure Data Lake Storage (ADLS), optimizing for maximum query performance and cost-efficiency. Drive data excellence by integrating Profisee Master Data Management (MDM) and implementing Change Data Capture (CDC) for real-time data synchronization. Establish rigorous data governance frameworks using Azure Purview for automated cataloging, classification, and lineage tracking. Mentor and coach junior engineers while leading the charge on CI/CD pipeline automation via Azure DevOps and optimizing complex SQL Server environments.What You'll Need 10+ years of deep Data Engineering experience (or 8 years with a Master’s degree) with a proven track record of technical leadership. 6+ years of hands-on expertise specifically within Databricks and Big Data analytics environments. Mastery of the Azure Data Stack, including Azure Data Factory (ADF), SQL Server Management, and Delta Parquet File Management. Advanced proficiency in SQL Server Integration Services (SSIS), SSRS, and Tableau for high-level business intelligence reporting. Strong background in automated deployment, process optimization, and working with Profisee MDM and Azure Purview tools.What's On Offer Highly competitive salary range of $112,000 - $140,000 based on experience and impact. Remote-friendly flexibility with a focus on work-life balance and physical, financial, and emotional well-being. Comprehensive "Best-in-Class" benefits package including paid parental leave, mental health services, and generous PTO. Opportunity to work at the forefront of insurance technology with a focus on innovation and professional growth.Apply via Haystack today!