Databricks Spark SME

Databricks Spark SME (Transactional Data)Hybrid – Houston, TX12-Month ContractWe are supporting a client who is looking for a Databricks Spark SME to join their data engineering team on a 12-month contract. This role will focus on optimizing Spark workloads processing large-scale transactional datasets and improving performance, latency, and cost efficiency across the platform.'This is a hands-on engineering role, ideal for someone with a strong software engineering background and deep expertise in Spark internals and distributed data processing.ResponsibilitiesAct as the Spark subject matter expert for performance optimization across the Databricks platformAnalyze and optimize Spark jobs, clusters, and query performanceTroubleshoot and resolve latency issues in real-time streaming pipelinesOptimize cost and compute performance across Databricks workloadsImprove processing efficiency for large-scale transactional data pipelinesWork closely with data engineering teams to design high-performance Spark-based data processing frameworksOptimize serverless and distributed compute workloads for large data processingCollaborate with stakeholders across engineering and platform teams to ensure scalable and efficient solutionsRequirementsStrong software engineering background (Java, Hadoop, or similar)Deep expertise with Apache Spark, including Spark internals and performance tuningHands-on experience with the Databricks platformExperience working with large transactional datasetsStrong experience with real-time or near real-time streaming pipelines (e.g., Kafka, Spark Structured Streaming)Experience resolving performance, latency, and scaling challenges in distributed data systemsHands-on experience working in AWS environmentsExperience optimizing large-scale data processing workloads and compute costsExperience with serverless compute on DatabricksNice to HaveExperience supporting high-throughput transactional systemsExperience in high-scale data environments