{"schemaVersion":"jobsearcher.job.v1","id":"7a49905b054e6e32658e329d","url":"https://jobsearcher.com/jobs/7a49905b054e6e32658e329d","canonicalUrl":"https://jobsearcher.com/jobs/7a49905b054e6e32658e329d","title":"Lead Data Engineer","description":"Job Title: Databricks Architect\nLocation: San Jose, CA (Hybrid)\nFulltime and Contract\nRequirements\nKey Responsibilities:\nWork on client projects to deliver Microsoft Azure based Data engineering & Analytics solutions.\nProvide hands on technical leadership for relevant technical skills, in multiple projects if required.\nDesign and implement high performing scalable analytics solutions on Azure following well-architected framework guidelines.\nDesign pipelines to ingest & integrate structured/unstructured data from heterogeneous sources.\nCollaborate with delivery leadership to deliver projects on time adhering to the quality standards\nMandatory Skills\nIn-depth knowledge Data processing techniques and handling large data sets.\nHands on experience using different storage services on Azure for storing data, like ADLS, SQL DB, SYnapse, Delta tables, Hive.\nExpert level hands on experience using Databricks in at least 4 projects, creating notebooks using Scala/Python.\nExpertise using Pyspark and Spark SQL for processing data for various use cases.\nExpertise in code/performance optimization, and troubleshooting PySpark notebooks.\nGood understanding of Delta lake and how it can be used in conjunction with ADLS and other storage.\nExperience working with atleast 1 big data project using map-reduce.\nExperience with Databricks scheduling features to automate Databricks jobs.\nProblem-solving skills along with good interpersonal & communication skills.\nSelf-starter who can pick up any other relevant Azure Services in the Analytics space\nGood-to-Have Skills\nDatabricks Certified Data Engineer\nKnowledge of HDInsight, and Azure DevOps is a big plus\nDP 203 - Microsoft Certified Azure Data Engineer\nKnowledge in Power BI\nExposure to Databricks on AWS\nJob Type: Contract\nPay: From $60.00 per hour\nExpected hours: 8 per week\nBenefits:\n401(k)\nDental insurance\nHealth insurance\nCompensation Package:\nYearly pay\nSchedule:\nMonday to Friday\nExperience:\nDatabricks: 10 years (Required)\nPower BI: 1 year (Required)\nAzure Devops: 1 year (Required)\nMicrosoft Certified Azure Data Engineer: 1 year (Required)\nAWS: 5 years (Required)\nADLs: 5 years (Required)\nSQL DB: 5 years (Required)\nSynapse: 5 years (Required)\nDelta Tables: 5 years (Required)\nAbility to Relocate:\nSan Jose, CA 95123: Relocate before starting work (Required)\nWork Location: Hybrid remote in San Jose, CA 95123","company":"Avistech","rawCompany":"avistech","city":"San Jose","state":"CA","isRemote":false,"isActive":false,"createdAt":"2026-08-10T13:12:44.265Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Lead Data Engineer","description":"Job Title: Databricks Architect\nLocation: San Jose, CA (Hybrid)\nFulltime and Contract\nRequirements\nKey Responsibilities:\nWork on client projects to deliver Microsoft Azure based Data engineering & Analytics solutions.\nProvide hands on technical leadership for relevant technical skills, in multiple projects if required.\nDesign and implement high performing scalable analytics solutions on Azure following well-architected framework guidelines.\nDesign pipelines to ingest & integrate structured/unstructured data from heterogeneous sources.\nCollaborate with delivery leadership to deliver projects on time adhering to the quality standards\nMandatory Skills\nIn-depth knowledge Data processing techniques and handling large data sets.\nHands on experience using different storage services on Azure for storing data, like ADLS, SQL DB, SYnapse, Delta tables, Hive.\nExpert level hands on experience using Databricks in at least 4 projects, creating notebooks using Scala/Python.\nExpertise using Pyspark and Spark SQL for processing data for various use cases.\nExpertise in code/performance optimization, and troubleshooting PySpark notebooks.\nGood understanding of Delta lake and how it can be used in conjunction with ADLS and other storage.\nExperience working with atleast 1 big data project using map-reduce.\nExperience with Databricks scheduling features to automate Databricks jobs.\nProblem-solving skills along with good interpersonal & communication skills.\nSelf-starter who can pick up any other relevant Azure Services in the Analytics space\nGood-to-Have Skills\nDatabricks Certified Data Engineer\nKnowledge of HDInsight, and Azure DevOps is a big plus\nDP 203 - Microsoft Certified Azure Data Engineer\nKnowledge in Power BI\nExposure to Databricks on AWS\nJob Type: Contract\nPay: From $60.00 per hour\nExpected hours: 8 per week\nBenefits:\n401(k)\nDental insurance\nHealth insurance\nCompensation Package:\nYearly pay\nSchedule:\nMonday to Friday\nExperience:\nDatabricks: 10 years (Required)\nPower BI: 1 year (Required)\nAzure Devops: 1 year (Required)\nMicrosoft Certified Azure Data Engineer: 1 year (Required)\nAWS: 5 years (Required)\nADLs: 5 years (Required)\nSQL DB: 5 years (Required)\nSynapse: 5 years (Required)\nDelta Tables: 5 years (Required)\nAbility to Relocate:\nSan Jose, CA 95123: Relocate before starting work (Required)\nWork Location: Hybrid remote in San Jose, CA 95123","datePosted":"2026-08-10T13:12:44.265Z","dateModified":"2026-08-10T13:12:44.265Z","hiringOrganization":{"@type":"Organization","name":"Avistech","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"San Jose","addressRegion":"CA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"7a49905b054e6e32658e329d"},"url":"https://jobsearcher.com/jobs/7a49905b054e6e32658e329d"}}