{"schemaVersion":"jobsearcher.job.v1","id":"53be0bd8dc5647bf6406f2f2","url":"https://jobsearcher.com/jobs/53be0bd8dc5647bf6406f2f2","canonicalUrl":"https://jobsearcher.com/jobs/53be0bd8dc5647bf6406f2f2","title":"Lead Software Engineer - Databricks, Spark, AWS","description":"This is your chance to change the path of your career and guide multiple teams to success at one of the world's leading financial institutions.\r\nAs a Lead Software Engineer at JPMorgan Chase within Corporate Sector, Chief Technology Office, youare an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives.\r\nJob Responsibilities:Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark (Core, SQL, Structured Streaming).\r\nEstablish lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) and ensure performance at scale.\r\nOwn Databricks cluster strategy and setup: runtime selection, autoscaling, driver/executor sizing, Spark configs, init scripts, cluster policies, pools, and instance profiles.\r\nOrchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.\r\nDesign secure data ingestion and transformation frameworks leveraging AWS services:S3 for data lake storage and lifecycle management\r\nGlue for catalog/metadata and ETL jobs\r\nIAM and Secrets Manager for role-based access and credential management\r\nCloudWatch for logging, metrics, and alerting\r\nLambda for serverless utilities\r\nKinesis and/or Kafka/MSK for streaming ingestionEnforce data quality, lineage, and governance using Unity Catalog and/or Glue Catalog; embed expectations and validation into pipelines.\r\nDrive Spark performance engineering: partitioning strategies, file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, and job right-sizing to optimize cost.\r\nBuild reusable libraries, frameworks, and APIs in Python and/or Java; oversee unit, integration, and data validation testing.\r\nImplement CI/CD for data projects (Git-based workflows), Terraform Infrastructure deployments environment promotion, and automated deployments; champion engineering standards and code reviews.\r\nPartner with platform security and networking teams to enforce encryption, network controls, and least-privilege access; ensure compliance with organizational policies.\r\nLead incident response and root-cause analysis; establish SLAs, observability, and runbooks; drive continuous improvement in reliability and cost efficiency.\r\nPartner with platform security and networking teams to enforce encryption, network controls, and least-privilege access; ensure compliance with organizational policies.\r\nLead incident response and root-cause analysis; establish SLAs, observability, and runbooks; drive continuous improvement in reliability and cost efficiency.\r\nRequired qualifications, capabilities, and skills:Formal training or certification on software engineering concepts and 5+ years applied experience.\r\n10+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR.\r\nStrong proficiency in Python and/or Java for data processing, platform tooling, and automation.\r\nHands-on Databricks expertise (Delta Lake, Unity Catalog, Workflows, Repos/notebooks, SQL Warehouses).\r\nSolid AWS experience: S3, IAM, Glue, CloudWatch, Kinesis / MSK, DynamoDB\r\nProven track record architecting and operating ETL/ELT pipelines (batch and streaming), with schema design/evolution, SLAs, and reliability engineering.\r\nDeep skills in Spark performance tuning and Databricks cluster setup/optimization.\r\nStrong SQL and analytics data modeling (dimensional/star schema; lakehouse best practices).\r\nSecurity-first mindset: roles/instance profiles, secret management, encryption-at-rest/in-transit, and network controls.\r\nDemonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship; excellent communication with stakeholders.\r\nPreferred qualifications, capabilities, and skills:Experience with Delta Live Tables and advanced governance (catalogs, grants, auditing) in Databricks.\r\nAWS networking knowledge (VPC, subnets, routing, security groups) and data egress controls.\r\nExperience with Terraform for Infra deployments\r\nCost optimization experience: autoscaling strategies, spot vs on-demand, auto-termination, storage layouts and compaction.\r\nFamiliarity with Kafka/MSK or Kinesis Data Streams/Firehose for real-time ingestion.\r\nCI/CD and automation tooling for data (Git workflows, artifact management) and testing frameworks (pytest, JUnit).\r\nObservability for data systems (freshness/completeness metrics, lineage, SLAs, alerting).\r\nExperience in financial services or other regulated industries.#J-18808-Ljbffr","company":"Testing Site Jpmc","rawCompany":"testing site jpmc","city":"Plano","state":"TX","isRemote":false,"isActive":false,"createdAt":"2026-11-04T02:16:46.872Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Lead Software Engineer - Databricks, Spark, AWS","description":"This is your chance to change the path of your career and guide multiple teams to success at one of the world's leading financial institutions.\r\nAs a Lead Software Engineer at JPMorgan Chase within Corporate Sector, Chief Technology Office, youare an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives.\r\nJob Responsibilities:Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark (Core, SQL, Structured Streaming).\r\nEstablish lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) and ensure performance at scale.\r\nOwn Databricks cluster strategy and setup: runtime selection, autoscaling, driver/executor sizing, Spark configs, init scripts, cluster policies, pools, and instance profiles.\r\nOrchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.\r\nDesign secure data ingestion and transformation frameworks leveraging AWS services:S3 for data lake storage and lifecycle management\r\nGlue for catalog/metadata and ETL jobs\r\nIAM and Secrets Manager for role-based access and credential management\r\nCloudWatch for logging, metrics, and alerting\r\nLambda for serverless utilities\r\nKinesis and/or Kafka/MSK for streaming ingestionEnforce data quality, lineage, and governance using Unity Catalog and/or Glue Catalog; embed expectations and validation into pipelines.\r\nDrive Spark performance engineering: partitioning strategies, file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, and job right-sizing to optimize cost.\r\nBuild reusable libraries, frameworks, and APIs in Python and/or Java; oversee unit, integration, and data validation testing.\r\nImplement CI/CD for data projects (Git-based workflows), Terraform Infrastructure deployments environment promotion, and automated deployments; champion engineering standards and code reviews.\r\nPartner with platform security and networking teams to enforce encryption, network controls, and least-privilege access; ensure compliance with organizational policies.\r\nLead incident response and root-cause analysis; establish SLAs, observability, and runbooks; drive continuous improvement in reliability and cost efficiency.\r\nPartner with platform security and networking teams to enforce encryption, network controls, and least-privilege access; ensure compliance with organizational policies.\r\nLead incident response and root-cause analysis; establish SLAs, observability, and runbooks; drive continuous improvement in reliability and cost efficiency.\r\nRequired qualifications, capabilities, and skills:Formal training or certification on software engineering concepts and 5+ years applied experience.\r\n10+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR.\r\nStrong proficiency in Python and/or Java for data processing, platform tooling, and automation.\r\nHands-on Databricks expertise (Delta Lake, Unity Catalog, Workflows, Repos/notebooks, SQL Warehouses).\r\nSolid AWS experience: S3, IAM, Glue, CloudWatch, Kinesis / MSK, DynamoDB\r\nProven track record architecting and operating ETL/ELT pipelines (batch and streaming), with schema design/evolution, SLAs, and reliability engineering.\r\nDeep skills in Spark performance tuning and Databricks cluster setup/optimization.\r\nStrong SQL and analytics data modeling (dimensional/star schema; lakehouse best practices).\r\nSecurity-first mindset: roles/instance profiles, secret management, encryption-at-rest/in-transit, and network controls.\r\nDemonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship; excellent communication with stakeholders.\r\nPreferred qualifications, capabilities, and skills:Experience with Delta Live Tables and advanced governance (catalogs, grants, auditing) in Databricks.\r\nAWS networking knowledge (VPC, subnets, routing, security groups) and data egress controls.\r\nExperience with Terraform for Infra deployments\r\nCost optimization experience: autoscaling strategies, spot vs on-demand, auto-termination, storage layouts and compaction.\r\nFamiliarity with Kafka/MSK or Kinesis Data Streams/Firehose for real-time ingestion.\r\nCI/CD and automation tooling for data (Git workflows, artifact management) and testing frameworks (pytest, JUnit).\r\nObservability for data systems (freshness/completeness metrics, lineage, SLAs, alerting).\r\nExperience in financial services or other regulated industries.#J-18808-Ljbffr","datePosted":"2026-11-04T02:16:46.872Z","dateModified":"2026-11-04T02:16:46.872Z","hiringOrganization":{"@type":"Organization","name":"Testing Site Jpmc","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Plano","addressRegion":"TX","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"53be0bd8dc5647bf6406f2f2"},"url":"https://jobsearcher.com/jobs/53be0bd8dc5647bf6406f2f2"}}