{"schemaVersion":"jobsearcher.job.v1","id":"d002faee4b1c28521149ae4b","url":"https://jobsearcher.com/jobs/d002faee4b1c28521149ae4b","canonicalUrl":"https://jobsearcher.com/jobs/d002faee4b1c28521149ae4b","title":"Senior Data Engineer","description":"We're hiring on behalf of a Haystack partner!The Role Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark and cloud-native data technologies Develop and optimize ETL/ELT workflows to ingest, transform, and curate data from diverse source systems into analytics-ready datasets Implement data modeling and transformation logic to support reporting, dashboards, and downstream analytical and machine learning workloads Build and manage data processing workloads within modern lakehouse platforms Ensure data quality, reliability, and consistency by implementing validation checks, monitoring, and reconciliation processes Optimize Spark jobs for performance, cost efficiency, and scalability across large and complex datasetsWhat You'll Need Strong experience as a Data Engineer building and operating production data pipelines Hands-on experience with Apache Spark for large-scale data processing Proficiency in Python, SQL, and data transformation best practices Experience with cloud-based data platforms and storage (e.g., Data Lakes, Lakehouse architectures) Familiarity with Microsoft Fabric, OneLake, or similar analytics platforms (strong plus) Experience designing and optimizing data models for analytical workloadsWhat's On Offer Opportunity to work on scalable, reliable, and high-performing data pipelines Collaborative environment with analytics engineers, data scientists, and product teams Contribute to Agile delivery processesApply via Haystack today!","company":"Haystack","rawCompany":"haystack","city":"Chicago","state":"IL","isRemote":false,"isActive":false,"createdAt":"2026-08-05T09:44:48.088Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Senior Data Engineer","description":"We're hiring on behalf of a Haystack partner!The Role Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark and cloud-native data technologies Develop and optimize ETL/ELT workflows to ingest, transform, and curate data from diverse source systems into analytics-ready datasets Implement data modeling and transformation logic to support reporting, dashboards, and downstream analytical and machine learning workloads Build and manage data processing workloads within modern lakehouse platforms Ensure data quality, reliability, and consistency by implementing validation checks, monitoring, and reconciliation processes Optimize Spark jobs for performance, cost efficiency, and scalability across large and complex datasetsWhat You'll Need Strong experience as a Data Engineer building and operating production data pipelines Hands-on experience with Apache Spark for large-scale data processing Proficiency in Python, SQL, and data transformation best practices Experience with cloud-based data platforms and storage (e.g., Data Lakes, Lakehouse architectures) Familiarity with Microsoft Fabric, OneLake, or similar analytics platforms (strong plus) Experience designing and optimizing data models for analytical workloadsWhat's On Offer Opportunity to work on scalable, reliable, and high-performing data pipelines Collaborative environment with analytics engineers, data scientists, and product teams Contribute to Agile delivery processesApply via Haystack today!","datePosted":"2026-08-05T09:44:48.088Z","dateModified":"2026-08-05T09:44:48.088Z","hiringOrganization":{"@type":"Organization","name":"Haystack","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Chicago","addressRegion":"IL","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"d002faee4b1c28521149ae4b"},"url":"https://jobsearcher.com/jobs/d002faee4b1c28521149ae4b"}}