JOBSEARCHER

Databricks Data Engineer, Lead

Overview: As a lead Databricks Data Engineer, you will design, build, and maintain robust data pipelines in Databricks while providing technical leadership. You will drive end-to-end ETL, data modeling, and security/compliance, collaborating with cross-functional teams to deliver scalable analytics platforms. This role combines hands-on delivery with mentorship in a fast-paced, agile environment. You will bridge raw data and insights, and contribute to data-driven decisions at scale. Compensation / Benefitshealth, life, disability, financial, and retirement benefitspaid leavetuition assistancework-life programsprofessional developmentrecognition awards
Responsibilities:
  • Design, build, and maintain end-to-end data pipelines in Databricks
  • Develop and optimize ETL processes and data flows
  • Design data models (Bronze/Silver/Gold) and medallion architecture
  • Ensure data security, governance, and compliance
  • Collaborate with cross-functional teams and stakeholders
  • Improve performance and scalability of data solutions
  • Mentor and lead a multi-disciplinary analytics and data engineering team
  • Oversee assessment, design, build, and maintenance of scalable platforms for clients
Key requirements:
  • 8+ years designing, developing, deploying, and managing ETL/data pipelines 3+ years building and optimizing Databricks pipelines (Delta Lake, Unity Catalog)3+ years Python and PySpark for ETL and transformations
  • Experience in data modeling and medallion architecture (Bronze/Silver/Gold)
  • Experience with AWS services (Glue, S3, RDS)
  • Performance optimization for ETL (partitioning, bucketing, caching)
  • Ability to define schemas and optimize data structures for analytics
  • Public Trust and security clearance suitability (Professional background check)
  • Strong communication and ability to explain technical concepts to non-technical audiences
  • Mentorship and leadership in a collaborative, agile environment
  • Cross-functional collaboration and stakeholder management
  • Databricks notebooks, clusters, and job orchestration
  • Delta Lake and Databricks DeltaUnity Catalog