Databricks Platform Engineering Lead
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
Databricks Platform Engineering Lead / Technical Delivery Lead Warren, NJ 3 Days Hybrid
Project Overview:
We are seeking an experienced Databricks Platform Engineering Lead to support an enterprise-wide pricing application within a global reinsurance organization. The application leverages Databricks as its data and analytics platform and integrates with a SQL Server-based pricing application. The resource will lead the engineering, governance, and ongoing enhancement of the Databricks platform supporting the pricing application. The role requires a strong combination of hands-on platform engineering, Databricks administration, data engineering, and technical delivery leadership. The ideal candidate must be capable of managing the Databricks platform foundation while providing technical oversight to data-engineering teams building ETL/ELT pipelines, transformations, and pricing data products.
Technical and Delivery Expertise:10+ years of overall technology experience.5+ years of hands-on experience with Databricks, cloud data platforms, or platform engineering.3+ years of experience in a Platform Lead, Databricks Lead, Cloud Platform Engineer, Data Platform Lead, or similar role.
Strong hands-on experience with:
- Azure Databricks
- Databricks account and workspace administration
- Unity Catalog
- Apache Spark and PySparkPython and SQLDelta Lake and Lakehouse architecture
- Azure Data Lake Storage
- Azure Data Factory or comparable integration tools
- Databricks Workflows and job orchestration
- Infrastructure as code using TerraformCI/CD and automated deployment practices
- Strong understanding of:
- Databricks workspace provisioning and administration
- Development, test, and production environment management
- Compute provisioning and cluster configuration
- Cluster policies, pools, runtimes, and serverless compute
- Platform security, identity, and access management
- Service principals, managed identities, and secrets management
- Azure networking, private endpoints, and secure connectivity
- Unity Catalog architecture, permissions, lineage, and governance
- Platform monitoring, logging, auditability, and observability
- Databricks performance, capacity, and cost optimization
- Production support and platform-incident management
- Good understanding of data-engineering practices, including:
- Batch and streaming data ingestionETL/ELT pipeline development
- Medallion architecture
- Data transformations and enrichment
- Data quality, validation, and reconciliation
- Schema management and data lineage
- Spark workload and query optimization
Experience leading globally distributed platform, data-engineering, application, QA, DevOps, and production-support teams. Ability to participate in architecture reviews, solution-design discussions, code reviews, security assessments, and production-readiness evaluations. Strong understanding of Agile delivery, release management, risk management, dependency management, and stakeholder governance.