Lead Software Engineer - Python, Databricks and AWS

Overview As a Lead Software Engineer in Corporate Technology, you drive design and delivery of trusted, scalable technology in a cross-functional, agile setting. You architect data lakehouse solutions, build robust data pipelines, and govern data access and security. You lead engineering standards, mentor others, and promote enterprise AI-assisted practices to raise quality and speed. This role blends hands-on delivery with strategic tech leadership in a regulated financial environment. Compensation / Benefitsbase salary and eligible incentive compensationcomprehensive health care coverageretirement savings planon-site health and wellness centerstuition reimbursementmental health support ResponsibilitiesArchitect lakehouse layers (bronze/silver/gold) and domain data productsDeliver scalable ingestion from AWS sources into Databricks (batch + streaming) with CDC where neededBuild maintainable pipelines using Delta Live Tables or modular Jobs with tests and documentationOperationalize workloads with Databricks Workflows/Jobs, retries, checkpointing, idempotency and safe re-runsGovernance by design: enforce least privilege, data classification, auditing, lineage, and controlled sharingDrive enterprise AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes with consistent validation standardsApply SDLC toolchain knowledge and automation to enhance value and efficiencyPerformance and cost management: tune Spark/Delta workloads, size clusters, optimize storage, and manage spendLead and mentor: set engineering standards, run design reviews, upskill engineers in Spark/DatabricksCI/CD and IaC: Terraform for Databricks and AWS resources; promote across environmentsTesting: unit/integration tests, data quality checks, contract testing, replay/backfill procedures, version control and runbooks Key requirementsFormal training or certification in software engineering plus 5+ years SDLC experienceStrong data engineering with leadership delivering multi-team data platformsHands-on experience building and operating a Databricks Lakehouse hosted in AWSDeep experience with Delta Lake (ACID, partitioning, schema evolution)Proven Spark on Databricks performance tuning, clustering, skew mitigation, joins, caching, file sizingExperience with streaming and batch pipelines (Structured Streaming, incremental processing, backfills, late-arriving data)Strong AWS fundamentals for data platforms (S3, IAM, KMS, VPC, logging/auditing)Experience implementing data governance/security controls in Databricks (Unity Catalog, permissions, credential passthrough)Experience leading use of AI-assisted development tools with validation of outputs for correctness, performance, and securityUnderstanding of responsible AI in engineering workflows and secure handling of dataProven ownership of reliability: monitoring/alerting, incident response, RCA, SLO/SLA managementleadership and mentorshipcross-functional collaborationstrong communication of trade-offs and plansDatabricks LakehouseDelta LakeSpark on Databricks