{"schemaVersion":"jobsearcher.job.v1","id":"a23479b8abf8e2dfd7378ffb","url":"https://jobsearcher.com/jobs/a23479b8abf8e2dfd7378ffb","canonicalUrl":"https://jobsearcher.com/jobs/a23479b8abf8e2dfd7378ffb","title":"Databricks Technical Lead","description":"Databricks Technical Lead\n Location: Chicago, ILDuration: 12+ Months\n We’re looking for a Databricks Technical Lead who can guide the design and build‑out of our data engineering and transformation platforms. This person will be the go‑to expert on Databricks, Delta Lake, and Spark, and will help shape how data flows across our organization — from ingestion all the way through our curated layers.\n This is a hands‑on leadership role. You won’t just review work — you’ll help solve hard problems, mentor engineers, set standards, and make architecture decisions that will influence the platform for years.\n What You’ll Do\n \n Lead the technical direction for Databricks‑based data pipelines and frameworks.\n Design and review patterns for ingesting, transforming, and publishing data (Bronze → Silver → Gold).\n Define best practices around Delta Lake, schema evolution, SCD handling, and metadata‑driven transformations.\n Provide technical oversight across multiple engineering squads.\n Work with architects, data modelers, quality engineers, and operations teams to ensure pipelines are built the right way.\n Mentor data engineers and help elevate the overall engineering capability.\n Oversee Unity Catalog governance, including RBAC, lineage, and schema enforcement.\n Help troubleshoot complex performance issues and guide teams on tuning and optimization.\n Support integration with orchestration tools and CI/CD processes.\n \n What You Bring\n \n Several years of hands‑on Spark experience and deep expertise with Databricks (Workflows, Delta Lake, Repos, Unity Catalog).\n Strong understanding of data engineering patterns, especially medallion architecture.\n Solid knowledge of AWS data services (S3, Glue, IAM, or equivalents).\n Experience with structured streaming and Auto Loader is a plus.\n Ability to lead and mentor engineers, give constructive feedback, and set engineering standards.\n Strong communication skills — able to explain complex ideas in a clear and approachable way.\n \n Recruiter Checklist — Databricks Technical Lead\n Must‑Have Skills\n \n 5+ years Databricks experience at a senior/lead level\n Strong Spark (PySpark + Spark SQL), not just SQL users\n Deep understanding of Delta Lake (OPTIMIZE, VACUUM, compaction, file layout)\n Designed or owned a medallion architecture\n Experience with schema evolution & SCD Type 1/2 handling\n Hands‑on Unity Catalog experience (permissions, lineage, governance)\n Built or maintained metadata‑driven frameworks\n Streaming experience (Auto Loader / Structured Streaming)\n Experience with Airflow, Glue, or Databricks Workflows\n Working knowledge of cloud services (AWS)\n Has led or mentored engineering teams\n Performs architecture/design reviews\n Sets standards and frameworks\n Communicates clearly with tech and non‑tech teams\n \n Red Flags\n \n Only familiar with Databricks notebooks, not platform engineering\n No Unity Catalog exposure\n Has not worked on metadata‑driven patterns\n Cannot explain SCD or schema evolution confidently\n \n Seniority Level\n \n Mid‑Senior level\n \n Employment Type\n \n Contract\n \n Job Function\n \n Information Technology\n \n Industries\n \n Staffing and Recruiting\n \n #J-18808-Ljbffr","company":"Enderit","rawCompany":"enderit","city":"Chicago","state":"IL","isRemote":false,"isActive":false,"createdAt":"2026-05-20T09:25:50.654Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"11-3021.00","title":"Computer and Information Systems Managers","slug":"computer-and-information-systems-managers"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Databricks Technical Lead","description":"Databricks Technical Lead\n Location: Chicago, ILDuration: 12+ Months\n We’re looking for a Databricks Technical Lead who can guide the design and build‑out of our data engineering and transformation platforms. This person will be the go‑to expert on Databricks, Delta Lake, and Spark, and will help shape how data flows across our organization — from ingestion all the way through our curated layers.\n This is a hands‑on leadership role. You won’t just review work — you’ll help solve hard problems, mentor engineers, set standards, and make architecture decisions that will influence the platform for years.\n What You’ll Do\n \n Lead the technical direction for Databricks‑based data pipelines and frameworks.\n Design and review patterns for ingesting, transforming, and publishing data (Bronze → Silver → Gold).\n Define best practices around Delta Lake, schema evolution, SCD handling, and metadata‑driven transformations.\n Provide technical oversight across multiple engineering squads.\n Work with architects, data modelers, quality engineers, and operations teams to ensure pipelines are built the right way.\n Mentor data engineers and help elevate the overall engineering capability.\n Oversee Unity Catalog governance, including RBAC, lineage, and schema enforcement.\n Help troubleshoot complex performance issues and guide teams on tuning and optimization.\n Support integration with orchestration tools and CI/CD processes.\n \n What You Bring\n \n Several years of hands‑on Spark experience and deep expertise with Databricks (Workflows, Delta Lake, Repos, Unity Catalog).\n Strong understanding of data engineering patterns, especially medallion architecture.\n Solid knowledge of AWS data services (S3, Glue, IAM, or equivalents).\n Experience with structured streaming and Auto Loader is a plus.\n Ability to lead and mentor engineers, give constructive feedback, and set engineering standards.\n Strong communication skills — able to explain complex ideas in a clear and approachable way.\n \n Recruiter Checklist — Databricks Technical Lead\n Must‑Have Skills\n \n 5+ years Databricks experience at a senior/lead level\n Strong Spark (PySpark + Spark SQL), not just SQL users\n Deep understanding of Delta Lake (OPTIMIZE, VACUUM, compaction, file layout)\n Designed or owned a medallion architecture\n Experience with schema evolution & SCD Type 1/2 handling\n Hands‑on Unity Catalog experience (permissions, lineage, governance)\n Built or maintained metadata‑driven frameworks\n Streaming experience (Auto Loader / Structured Streaming)\n Experience with Airflow, Glue, or Databricks Workflows\n Working knowledge of cloud services (AWS)\n Has led or mentored engineering teams\n Performs architecture/design reviews\n Sets standards and frameworks\n Communicates clearly with tech and non‑tech teams\n \n Red Flags\n \n Only familiar with Databricks notebooks, not platform engineering\n No Unity Catalog exposure\n Has not worked on metadata‑driven patterns\n Cannot explain SCD or schema evolution confidently\n \n Seniority Level\n \n Mid‑Senior level\n \n Employment Type\n \n Contract\n \n Job Function\n \n Information Technology\n \n Industries\n \n Staffing and Recruiting\n \n #J-18808-Ljbffr","datePosted":"2026-05-20T09:25:50.654Z","dateModified":"2026-05-20T09:25:50.654Z","hiringOrganization":{"@type":"Organization","name":"Enderit","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Chicago","addressRegion":"IL","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"a23479b8abf8e2dfd7378ffb"},"url":"https://jobsearcher.com/jobs/a23479b8abf8e2dfd7378ffb"}}