{"schemaVersion":"jobsearcher.job.v1","id":"719478a2e0ab1f70786e5fe0","url":"https://jobsearcher.com/jobs/719478a2e0ab1f70786e5fe0","canonicalUrl":"https://jobsearcher.com/jobs/719478a2e0ab1f70786e5fe0","title":"Spark Developer","description":"Job Title: Spark Developer (Search Integration)\r\nPosition Type: Contract.\r\nLocation: Pleasanton, CA – Onsite- Hybrid - Looking for resources who can work 3 days hybrid onsite in Pleasanton, CA\r\nNote\r\nWe are looking for a Spark Developer with OpenSearch/Algolia expertise who can design, build, and optimize scalable data pipelines to ingest, transform, and index large-scale datasets into search engines for fast retrieval.\r\nUtilize Scala/Python and Spark SQL to process data from various sources (S3, Kafka) for real-time indexing in OpenSearch or Algolia.\r\nFocus: Spark (ETL/Streaming) + Search Engines (OpenSearch/Algolia)\r\nObjective: Power real-time, relevant, and fast search experiences.\r\nResponsibilities\r\nData Pipelines : Design, develop, and maintain high-performance Spark jobs (Scala or PySpark) to process, transform, and clean large datasets.\r\nIndex Management : Ingest data into OpenSearch or Algolia, optimizing index strategy, mapping, and document structuring for maximum search efficiency.\r\nOptimization : Tune Spark applications (data partitioning, caching, shuffle tuning) and search engines (query performance, indexing speed).\r\nStreaming/Batch : Implement both batch ETL jobs and real-time streaming solutions (Spark Streaming/Kafka) to keep search indexes updated.\r\nCollaboration : Work with backend teams to integrate search functionality into applications and debug search relevance issues.\r\nRequired Skills and Qualifications\r\nCore Spark: Strong experience with Apache Spark RDD/DataFrame APIs, Scala, or Python (PySpark).\r\nSearch Tech: Experience in indexing, querying, and managing clusters in OpenSearch (formerly Elasticsearch) or Algolia.\r\nBig Data Ecosystem: Proficiency with HDFS, S3, Kafka, and data warehousing solutions.\r\nDatabase Knowledge: Experience with SQL/NoSQL databases (PostgreSQL, Cassandra, DynamoDB).\r\nPerformance Tuning: Expertise in optimizing distributed systems and troubleshooting latency issues.\r\nTypical Projects\r\nBuilding a product catalog search engine using Spark to transform ERP data into JSON, indexed into OpenSearch.\r\nImplementing real-time update pipelines to sync user activity logs into Algolia for instant search results.\r\nJ-18808-Ljbffr","company":"Mindquest Technology Solutions","rawCompany":"mindquest technology solutions","city":"Pleasanton","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-04-09T09:52:41.167Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"519290","title":"Web Search Portals and All Other Information Services","slug":"web-search-portals-and-all-other-information-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Spark Developer","description":"Job Title: Spark Developer (Search Integration)\r\nPosition Type: Contract.\r\nLocation: Pleasanton, CA – Onsite- Hybrid - Looking for resources who can work 3 days hybrid onsite in Pleasanton, CA\r\nNote\r\nWe are looking for a Spark Developer with OpenSearch/Algolia expertise who can design, build, and optimize scalable data pipelines to ingest, transform, and index large-scale datasets into search engines for fast retrieval.\r\nUtilize Scala/Python and Spark SQL to process data from various sources (S3, Kafka) for real-time indexing in OpenSearch or Algolia.\r\nFocus: Spark (ETL/Streaming) + Search Engines (OpenSearch/Algolia)\r\nObjective: Power real-time, relevant, and fast search experiences.\r\nResponsibilities\r\nData Pipelines : Design, develop, and maintain high-performance Spark jobs (Scala or PySpark) to process, transform, and clean large datasets.\r\nIndex Management : Ingest data into OpenSearch or Algolia, optimizing index strategy, mapping, and document structuring for maximum search efficiency.\r\nOptimization : Tune Spark applications (data partitioning, caching, shuffle tuning) and search engines (query performance, indexing speed).\r\nStreaming/Batch : Implement both batch ETL jobs and real-time streaming solutions (Spark Streaming/Kafka) to keep search indexes updated.\r\nCollaboration : Work with backend teams to integrate search functionality into applications and debug search relevance issues.\r\nRequired Skills and Qualifications\r\nCore Spark: Strong experience with Apache Spark RDD/DataFrame APIs, Scala, or Python (PySpark).\r\nSearch Tech: Experience in indexing, querying, and managing clusters in OpenSearch (formerly Elasticsearch) or Algolia.\r\nBig Data Ecosystem: Proficiency with HDFS, S3, Kafka, and data warehousing solutions.\r\nDatabase Knowledge: Experience with SQL/NoSQL databases (PostgreSQL, Cassandra, DynamoDB).\r\nPerformance Tuning: Expertise in optimizing distributed systems and troubleshooting latency issues.\r\nTypical Projects\r\nBuilding a product catalog search engine using Spark to transform ERP data into JSON, indexed into OpenSearch.\r\nImplementing real-time update pipelines to sync user activity logs into Algolia for instant search results.\r\nJ-18808-Ljbffr","datePosted":"2026-04-09T09:52:41.167Z","dateModified":"2026-04-09T09:52:41.167Z","hiringOrganization":{"@type":"Organization","name":"Mindquest Technology Solutions","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Pleasanton","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"719478a2e0ab1f70786e5fe0"},"url":"https://jobsearcher.com/jobs/719478a2e0ab1f70786e5fe0"}}