Databricks Data Engineer, Lead
Overview:
As a lead Databricks Data Engineer, you will design, build, and maintain robust data pipelines in Databricks while providing technical leadership. You will drive end-to-end ETL, data modeling, and security/compliance, collaborating with cross-functional teams to deliver scalable analytics platforms. This role combines hands-on delivery with mentorship in a fast-paced, agile environment. You will bridge raw data and insights, and contribute to data-driven decisions at scale. Compensation / Benefitshealth, life, disability, financial, and retirement benefitspaid leavetuition assistancework-life programsprofessional developmentrecognition awards
Responsibilities:
- Design, build, and maintain end-to-end data pipelines in Databricks
- Develop and optimize ETL processes and data flows
- Design data models (Bronze/Silver/Gold) and medallion architecture
- Ensure data security, governance, and compliance
- Collaborate with cross-functional teams and stakeholders
- Improve performance and scalability of data solutions
- Mentor and lead a multi-disciplinary analytics and data engineering team
- Oversee assessment, design, build, and maintenance of scalable platforms for clients
Key requirements:
- 8+ years designing, developing, deploying, and managing ETL/data pipelines 3+ years building and optimizing Databricks pipelines (Delta Lake, Unity Catalog)3+ years Python and PySpark for ETL and transformations
- Experience in data modeling and medallion architecture (Bronze/Silver/Gold)
- Experience with AWS services (Glue, S3, RDS)
- Performance optimization for ETL (partitioning, bucketing, caching)
- Ability to define schemas and optimize data structures for analytics
- Public Trust and security clearance suitability (Professional background check)
- Strong communication and ability to explain technical concepts to non-technical audiences
- Mentorship and leadership in a collaborative, agile environment
- Cross-functional collaboration and stakeholder management
- Databricks notebooks, clusters, and job orchestration
- Delta Lake and Databricks DeltaUnity Catalog