{"schemaVersion":"jobsearcher.job.v1","id":"aa4e2e62f445a8fc45aa8e1d","url":"https://jobsearcher.com/jobs/aa4e2e62f445a8fc45aa8e1d","canonicalUrl":"https://jobsearcher.com/jobs/aa4e2e62f445a8fc45aa8e1d","title":"Senior Spark Data Engineer - Databricks & ML Pipelines","description":"We are seeking a Senior Data Engineer with deep expertise in Apache Spark and hands-on experience delivering production-grade solutions on Databricks. This role centers on designing, optimizing, and operating large-scale Spark data platforms that power analytics, machine learning (ML), and artificial intelligence (AI) use cases.\r\nThe ideal candidate is a Spark practitioner who understands how to build efficient distributed data pipelines, optimize performance and cost, and enable ML workflows at scale. You will lead Spark pipeline design, collaborate closely with data scientists and analysts, and help operationalize ML and AI solutions across the enterprise.\r\nKey Responsibilities\r\nDesign, build, and maintain distributed data pipelines using Apache Spark on Databricks\r\nLead development of high-performance Spark jobs for batch and near-real-time processing\r\nImplement complex data transformations using PySpark (DataFrame & Spark SQL APIs)\r\nApply Spark best practices for partitioning, caching, joins, and shuffles\r\nTune Spark jobs for performance, scalability, and cost efficiency\r\nRequired Qualifications\r\n12+ years of experience in Data Engineering.\r\nStrong hands-on experience building and operating Apache Spark applications.\r\nProven experience running Spark workloads in Databricks (production environments).\r\nAdvanced proficiency in PySpark, Spark SQL, and Python.\r\nExperience working with large-scale, distributed datasets in cloud environments (AWS preferred).\r\nDemonstrated experience supporting ML or AI pipelines on Spark.\r\nStrong understanding of data modeling, distributed systems, and data architecture.\r\nExperience designing reliable, automated, and observable Spark pipelines.\r\nAbout ValueMomentum\r\nValueMomentum is a product development company. We have 25+ years in the market and we have products like Ifoundry, rating engine and Biz dynamics. We have been partnered with Guidewire, Duck Creek and Insurity . We have headquartered in NJ and PA. we have specialised in P&C insurance, we have 75+ clients implementing and delivering. we do have offshore development centres India, Canada, and UK. we have overall 6000+ employees overall the globe.\r\nJ-18808-Ljbffr","company":"Valuemomentum","rawCompany":"valuemomentum","city":"Edison","state":"NJ","isRemote":false,"isActive":false,"createdAt":"2026-06-25T00:55:16.997Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Senior Spark Data Engineer - Databricks & ML Pipelines","description":"We are seeking a Senior Data Engineer with deep expertise in Apache Spark and hands-on experience delivering production-grade solutions on Databricks. This role centers on designing, optimizing, and operating large-scale Spark data platforms that power analytics, machine learning (ML), and artificial intelligence (AI) use cases.\r\nThe ideal candidate is a Spark practitioner who understands how to build efficient distributed data pipelines, optimize performance and cost, and enable ML workflows at scale. You will lead Spark pipeline design, collaborate closely with data scientists and analysts, and help operationalize ML and AI solutions across the enterprise.\r\nKey Responsibilities\r\nDesign, build, and maintain distributed data pipelines using Apache Spark on Databricks\r\nLead development of high-performance Spark jobs for batch and near-real-time processing\r\nImplement complex data transformations using PySpark (DataFrame & Spark SQL APIs)\r\nApply Spark best practices for partitioning, caching, joins, and shuffles\r\nTune Spark jobs for performance, scalability, and cost efficiency\r\nRequired Qualifications\r\n12+ years of experience in Data Engineering.\r\nStrong hands-on experience building and operating Apache Spark applications.\r\nProven experience running Spark workloads in Databricks (production environments).\r\nAdvanced proficiency in PySpark, Spark SQL, and Python.\r\nExperience working with large-scale, distributed datasets in cloud environments (AWS preferred).\r\nDemonstrated experience supporting ML or AI pipelines on Spark.\r\nStrong understanding of data modeling, distributed systems, and data architecture.\r\nExperience designing reliable, automated, and observable Spark pipelines.\r\nAbout ValueMomentum\r\nValueMomentum is a product development company. We have 25+ years in the market and we have products like Ifoundry, rating engine and Biz dynamics. We have been partnered with Guidewire, Duck Creek and Insurity . We have headquartered in NJ and PA. we have specialised in P&C insurance, we have 75+ clients implementing and delivering. we do have offshore development centres India, Canada, and UK. we have overall 6000+ employees overall the globe.\r\nJ-18808-Ljbffr","datePosted":"2026-06-25T00:55:16.997Z","dateModified":"2026-06-25T00:55:16.997Z","hiringOrganization":{"@type":"Organization","name":"Valuemomentum","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Edison","addressRegion":"NJ","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"aa4e2e62f445a8fc45aa8e1d"},"url":"https://jobsearcher.com/jobs/aa4e2e62f445a8fc45aa8e1d"}}