{"schemaVersion":"jobsearcher.job.v1","id":"b7b340cfb0065a89dba7a82d","url":"https://jobsearcher.com/jobs/b7b340cfb0065a89dba7a82d","canonicalUrl":"https://jobsearcher.com/jobs/b7b340cfb0065a89dba7a82d","title":"Data Engineer","description":"Role: Data EngineerLocation: Multiple Locations in USAVisa Status: Any Visa holder except OPT'sPrimary Responsibilities:Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on AzureBuild cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta LakeCollaborate with Data Science and AI teams to operationalize ML models and embed them into production workflowsDevelop and maintain feature stores, model input pipelines, and real-time/streaming frameworksEnsure data quality, governance, and security across the full data lifecycleBuild reusable frameworks, accelerators, and automation scripts to improve engineering efficiencyOptimize performance, scalability, and reliability of data workflows and batch/streaming pipelinesParticipate in Agile development processes, including sprint planning, code reviews, and CI/CD pipelinesProvide production support and on-call coverage, ensuring system stability and rapid issue resolutionDesign, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AIRequired Qualifications:Bachelor's degree in Computer Science, Engineering, or IT related field6+ years of experience in Data Engineering with Python/PySpark6+ years of experience in building ETL/ELT pipelines using Databricks6+ years of experience working in Agile environments5+ years of strong experience in SQL / PL-SQL4+ years of experience with Azure Databricks and Delta Lake architecture4+ years of hands-on experience with CI/CD (GitHub Actions, Azure DevOps)3+ years of hands-on experience with Azure cloud services (ADF, ADLS, Databricks)2+ years of experience with Databricks Delta Live Tables (DLT)2+ years of experience with unit testing, validation, and pipeline testing frameworksPreferred Qualifications:Familiarity with medallion architecture and SCD2 implementationsAI builder: Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AIExperience building enterprise-scale data platformsStrong skills in performance tuning and debugging large-scale pipelinesExperience with real-time/streaming frameworks (Structured Streaming)Ability to work in distributed, cross-functional global teamsExposure to GenAI tools (e.g., GitHub Copilot) for engineering productivityStrong understanding of secure coding practices and vulnerability remediationProven ability to analyze logs, troubleshoot production issues, and optimize performanceDemonstrated capability to design and deploy AI-powered solutions responsibly.","company":"Sharpatoms","rawCompany":"sharpatoms","city":"New York","state":"NY","isRemote":false,"isActive":false,"createdAt":"2026-10-12T08:39:23.729Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Data Engineer","description":"Role: Data EngineerLocation: Multiple Locations in USAVisa Status: Any Visa holder except OPT'sPrimary Responsibilities:Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on AzureBuild cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta LakeCollaborate with Data Science and AI teams to operationalize ML models and embed them into production workflowsDevelop and maintain feature stores, model input pipelines, and real-time/streaming frameworksEnsure data quality, governance, and security across the full data lifecycleBuild reusable frameworks, accelerators, and automation scripts to improve engineering efficiencyOptimize performance, scalability, and reliability of data workflows and batch/streaming pipelinesParticipate in Agile development processes, including sprint planning, code reviews, and CI/CD pipelinesProvide production support and on-call coverage, ensuring system stability and rapid issue resolutionDesign, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AIRequired Qualifications:Bachelor's degree in Computer Science, Engineering, or IT related field6+ years of experience in Data Engineering with Python/PySpark6+ years of experience in building ETL/ELT pipelines using Databricks6+ years of experience working in Agile environments5+ years of strong experience in SQL / PL-SQL4+ years of experience with Azure Databricks and Delta Lake architecture4+ years of hands-on experience with CI/CD (GitHub Actions, Azure DevOps)3+ years of hands-on experience with Azure cloud services (ADF, ADLS, Databricks)2+ years of experience with Databricks Delta Live Tables (DLT)2+ years of experience with unit testing, validation, and pipeline testing frameworksPreferred Qualifications:Familiarity with medallion architecture and SCD2 implementationsAI builder: Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AIExperience building enterprise-scale data platformsStrong skills in performance tuning and debugging large-scale pipelinesExperience with real-time/streaming frameworks (Structured Streaming)Ability to work in distributed, cross-functional global teamsExposure to GenAI tools (e.g., GitHub Copilot) for engineering productivityStrong understanding of secure coding practices and vulnerability remediationProven ability to analyze logs, troubleshoot production issues, and optimize performanceDemonstrated capability to design and deploy AI-powered solutions responsibly.","datePosted":"2026-10-12T08:39:23.729Z","dateModified":"2026-10-12T08:39:23.729Z","hiringOrganization":{"@type":"Organization","name":"Sharpatoms","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"New York","addressRegion":"NY","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"b7b340cfb0065a89dba7a82d"},"url":"https://jobsearcher.com/jobs/b7b340cfb0065a89dba7a82d"}}