{"schemaVersion":"jobsearcher.job.v1","id":"31b96c68ba5464fe7fa968c2","url":"https://jobsearcher.com/jobs/31b96c68ba5464fe7fa968c2","canonicalUrl":"https://jobsearcher.com/jobs/31b96c68ba5464fe7fa968c2","title":"Spark Developer","description":"Hi All,Job Title: Spark Developer (Search Integration)Position Type: Contract . Location: Pleasanton, CA – Onsite- Hybrid - Looking for resources who can work 3 days hybrid onsite in Pleasanton, CA Note: We are looking for a Spark Developer with OpenSearch/Algolia expertise who can design, build, and optimize scalable data pipelines to ingest, transform, and index large-scale datasets into search engines for fast retrieval. Utilize Scala/Python and Spark SQL to process data from various sources (S3, Kafka) for real-time indexing in OpenSearch or Algolia.Focus: Spark (ETL/Streaming) + Search Engines (OpenSearch/Algolia)Objective: Power real-time, relevant, and fast search experiences..ResponsibilitiesData Pipelines: Design, develop, and maintain high-performance Spark jobs (Scala or PySpark) to process, transform, and clean large datasets.Index Management: Ingest data into OpenSearch or Algolia, optimizing index strategy, mapping, and document structuring for maximum search efficiency.Optimization: Tune Spark applications (data partitioning, caching, shuffle tuning) and search engines (query performance, indexing speed).Streaming/Batch: Implement both batch ETL jobs and real-time streaming solutions (Spark Streaming/Kafka) to keep search indexes updated.Collaboration: Work with backend teams to integrate search functionality into applications and debug search relevance issues.Required Skills and Qualification: Core Spark: Strong experience with Apache Spark RDD/DataFrame APIs, Scala, or Python (PySpark).Search Tech: Experience in indexing, querying, and managing clusters in OpenSearch (formerly Elasticsearch) or Algolia.Big Data Ecosystem: Proficiency with HDFS, S3, Kafka, and data warehousing solutions.Database Knowledge: Experience with SQL/NoSQL databases (PostgreSQL, Cassandra, DynamoDB).Performance Tuning: Expertise in optimizing distributed systems and troubleshooting latency issuesTypical Projects:Building a product catalog search engine using Spark to transform ERP data into JSON, indexed into OpenSearch.Implementing real-time update pipelines to sync user activity logs into Algolia for instant search results.Optimizing large-scale data re-indexing processes to reduce latency.Please do share me your updated resume to","company":"Mindquest Technology Solutions","rawCompany":"mindquest technology solutions","city":"Oakland","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-04-09T09:52:41.167Z","occupations":[{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1254.00","title":"Web Developers","slug":"web-developers"}],"industries":[{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Spark Developer","description":"Hi All,Job Title: Spark Developer (Search Integration)Position Type: Contract . Location: Pleasanton, CA – Onsite- Hybrid - Looking for resources who can work 3 days hybrid onsite in Pleasanton, CA Note: We are looking for a Spark Developer with OpenSearch/Algolia expertise who can design, build, and optimize scalable data pipelines to ingest, transform, and index large-scale datasets into search engines for fast retrieval. Utilize Scala/Python and Spark SQL to process data from various sources (S3, Kafka) for real-time indexing in OpenSearch or Algolia.Focus: Spark (ETL/Streaming) + Search Engines (OpenSearch/Algolia)Objective: Power real-time, relevant, and fast search experiences..ResponsibilitiesData Pipelines: Design, develop, and maintain high-performance Spark jobs (Scala or PySpark) to process, transform, and clean large datasets.Index Management: Ingest data into OpenSearch or Algolia, optimizing index strategy, mapping, and document structuring for maximum search efficiency.Optimization: Tune Spark applications (data partitioning, caching, shuffle tuning) and search engines (query performance, indexing speed).Streaming/Batch: Implement both batch ETL jobs and real-time streaming solutions (Spark Streaming/Kafka) to keep search indexes updated.Collaboration: Work with backend teams to integrate search functionality into applications and debug search relevance issues.Required Skills and Qualification: Core Spark: Strong experience with Apache Spark RDD/DataFrame APIs, Scala, or Python (PySpark).Search Tech: Experience in indexing, querying, and managing clusters in OpenSearch (formerly Elasticsearch) or Algolia.Big Data Ecosystem: Proficiency with HDFS, S3, Kafka, and data warehousing solutions.Database Knowledge: Experience with SQL/NoSQL databases (PostgreSQL, Cassandra, DynamoDB).Performance Tuning: Expertise in optimizing distributed systems and troubleshooting latency issuesTypical Projects:Building a product catalog search engine using Spark to transform ERP data into JSON, indexed into OpenSearch.Implementing real-time update pipelines to sync user activity logs into Algolia for instant search results.Optimizing large-scale data re-indexing processes to reduce latency.Please do share me your updated resume to","datePosted":"2026-04-09T09:52:41.167Z","dateModified":"2026-04-09T09:52:41.167Z","hiringOrganization":{"@type":"Organization","name":"Mindquest Technology Solutions","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Oakland","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"31b96c68ba5464fe7fa968c2"},"url":"https://jobsearcher.com/jobs/31b96c68ba5464fe7fa968c2"}}