Lead Data Platform Engineer

Compensation: Base, $210k - $250kLocation: San Diego, CAHybrid: 2x WeekThe RoleAs Lead Data Platform Engineer, you'll architect and implement our centralized data platform on Databricks. You'll establish governance patterns using Unity Catalog, optimize for cost and performance at scale, and enable our existing Data Engineers to build confidently on the platform. This is a data infrastructure role—focused on pipelines, storage, governance, and platform operations.The Business ChallengeWe operate multiple product lines (Transfer Pricing, R&D Services, RoyaltyStat, Provisioning), each with distinct databases containing enterprise financial data—journal entries, general ledgers, and financial statements. Our immediate challenge is migrating multi-terabyte datasets from legacy systems to a unified Databricks lakehouse while establishing governance patterns that enable multi-product operations at scale.What You'll BuildData Structuring: Design data models and implement unified schemas across multiple disparate product lines.Unity Catalog Architecture: Design and implement multi-catalog governance strategy supporting data isolation, cross-product data sharing, and comprehensive lineage tracking across our product portfolioDelta Lake Optimization: Establish patterns for Z-ordering, compaction, and liquid clustering at multi-TB scale. Define table structures, partitioning strategies, and retention policies that balance query performance with storage costsETL Pipeline Framework: Build declarative pipeline patterns using Delta Live Tables. Create orchestration workflows for ingesting data from internal sources such as SQL databases and S3Third Party Integrations: Integrate with third party data sources such as ERP systems (Netsuite etc.) and external data providers (S&P etc.) with automated ingest, robust error handling and monitoring.Platform Operations: Implement cost monitoring and optimization strategies, establish data quality frameworks, create self-service patterns enabling Data Engineers to work independently while maintaining governance standardsBusiness Problems You'll SolveKey Legacy Product Migrations: Lead the architecture for migrating multi-terabyte datasets from legacy systems to Databricks—establishing patterns that will be reused across multiple product linesMulti-Product Data Architecture: Design Unity Catalog structures enabling secure data separation between product lines while allowing controlled cross-product analytics where appropriateCost-Efficient Scale: Build infrastructure that scales efficiently—through intelligent caching, query optimization, and compute management strategies that avoid linear cost growthPlatform Reliability: Establish monitoring, alerting, and data quality validation ensuring the platform operates reliably as foundation for both analytics and AI workloadsRequired ExperienceDatabricks Expertise (Required)Unity Catalog: Production experience with multi-catalog governance, metastore design, and lineage tracking.Data Structuring: Experience designing and building unified schemas across multiple disparate product lines.Delta Lake: Expert-level experience with Z-ordering, compaction, liquid clustering, and performance tuning at multi-TB scaleDelta Live Tables: Strong hands-on experience building declarative ETL pipelines, including change data capture and expectations/constraintsDatabricks Workflows: Experience with job orchestration, scheduling, and operational monitoringBusiness Intelligence: Experience enabling company-wide analytics and reporting with modern business intelligence tools and maintaining source of truth data and metrics.PySpark & Databricks SQL: Strong proficiency for code review, performance tuning, and query optimizationCore Platform Engineering5-8 years in data engineering or data platform roles, with 3+ years hands-on Databricks experienceTrack record leading at least one significant platform build or migration projectAWS experience (S3, IAM, VPC) with ability to collaborate on infrastructure decisionsInfrastructure-as-code experience (Terraform preferred)Technical LeadershipDemonstrated ability architecting data platforms from first principles and defending technical decisionsStrong written and verbal communication— document architecture decisions and present to both technical and business stakeholdersPreferred But Not RequiredExperience with financial data, accounting systems (NetSuite), or enterprise ERP platformsBackground building platforms that serve AI/ML workloads (experience preparing data for downstream ML consumption, RAG and retrieval, and LLMs.Understand advanced intelligence concepts such as relationship surfacing with knowledge graphsFamiliarity with data governance frameworks and compliance requirements for regulated industriesWhat We Offer(The following only applies to US-based positions)A collaborative team culture with opportunities for career development.Ample opportunities to be recognized, build valuable skills, and grow your career.Generous vacation policy, including paid parental leave.Comprehensive health plans with FSA and HSA options.401(k) retirement plan.Life and disability insurance coverage.Supplemental benefits like a dependent care savings plan, pet insurance, will preparation, and an employee assistance program.About UsAt Exactera, a FinTech SaaS start-up founded in 2016, we stand at the intersection of human and machine intelligence. Our corporate tax solutions are powered by AI and cloud-based technologies, serving customers worldwide. With over $100 million in funding from Savant Venture Fund and Insight Partners, we are poised for growth. We are committed to diversity, inclusion, and equal opportunities for all.