Lead DATA ENGINEER - Cloud Migration
Role OverviewThis role is part of a multi year enterprise initiative to modernize data platforms by migrating from legacy and on prem environments to cloud native, governed, and scalable architectures .The role focuses on migrating enterprise data workloads to Databricks on strategic cloud platforms , enabling standardized data engineering, analytics, centralized reporting, reconciliation utilities, and AI/ML use cases while adhering to controls, security, resilience, and regulatory compliance .Key ResponsibilitiesLead and execute migration of legacy data platforms (on prem / non standard tools) to Databricks on cloud under the Olympus programPerform application, data, and pipeline refactoring to cloud native Databricks patternsDrive migration planning including dependency analysis, sequencing, and cutover strategySupport coexistence models and transition from dual run to cloud only executionDatabricks Lakehouse EngineeringDesign and implement Databricks Lakehouse architecture (Bronze / Silver / Gold)Build scalable batch and streaming pipelines using PySpark, Spark SQLLeverage Delta Lake for reliability, versioning, and performanceOptimize compute usage and cost in line with enterprise cloud efficiency goalsEnterprise Data Controls & GovernanceEmbed data quality, reconciliation, and completeness controls as part of migrationEnsure migrated workloads meet EDO governance, MCA, and audit requirementsMaintain lineage, traceability, and explainability across migrated assetsSupport risk critical use cases (Finance, Ops, Recon, Reporting)Cloud Security & ResilienceImplement cloud aligned RBAC, identity controls, and secure access patternsEnforce data encryption, masking, and classification standardsEnsure workloads meet operational resilience and recovery expectationsPartner with cloud platform and security teams for certification and sign offReporting, Analytics & AI EnablementEnable downstream BI, regulatory reporting, and MI workloads on DatabricksSupport centralized reporting programs (e.g., ARA, GRU related use cases)Prepare data foundations for AI / ML and Agentic workflows post migrationRequired Qualifications8 12+ years in data engineering / platform modernizationStrong hands on experience with Databricks in large scale enterprisesProven experience delivering cloud migration programs (on prem cloud)Deep expertise in Apache Spark, PySpark, Spark SQLExperience embedding controls, reconciliation, and data quality in migrationsExperience in regulated environments (banking / financial services preferred)Preferred QualificationsExperience with Client Olympus or equivalent enterprise cloud programsKnowledge of legacy data platforms and modernization patternsFamiliarity with Finance, Ops, Recon, or Balance Sheet data domainsExposure to MLflow, AI pipelines, or GenAI enablement on cloudStrong understanding of run the bank vs change the bank executionBehavioral & Delivery ExpectationsStrong ownership and execution mindsetComfortable operating in large, multi vendor transformation programsAbility to engage with Technology, Operations, Risk, and Audit stakeholdersDisciplined approach to migration risk, controls, and documentation