Lead Databricks Architect

Senior Databricks Architect (100% Remote)Company OverviewOur client is a leading provider of large-scale infrastructure and engineering services that support essential industries across North America. They are committed to investing in modern data platforms and fostering a culture of continuous learning, collaboration, and technical excellence. Role SummaryThe Intersect Group is seeking a highly skilled Senior Databricks Architect to lead the design, development, and optimization of a modern data platform leveraging the Databricks Lakehouse architecture. This role combines deep technical expertise with leadership responsibilities, driving data engineering, analytics, and AI/ML initiatives across multiple concurrent projects. You will serve as a hands‑on architect and trusted advisor, partnering with internal teams and external resources to implement scalable, high-performance data solutions. This role requires someone who can own delivery end‑to‑end, guide junior resources, and ensure consistent execution across a complex, multi‑project environment. Key ResponsibilitiesArchitect, design, and implement scalable data solutions using Databricks Lakehouse architecture. Lead hands‑on development, including building and optimizing data pipelines, notebooks, and workflows in Python and SQL. Serve as the primary technical escalation point for Databricks-related challenges across development and production environments. Collaborate with cross‑functional teams—including data modelers, engineers, and external consultants—to deliver projects on time. Oversee multiple initiatives (6–10 concurrently), ensuring alignment across varying project stages and stakeholders. Implement and optimize Databricks features such as Auto Loader, Delta Live Tables (DLT), and streaming data pipelines. Establish and enforce best practices for performance, scalability, security, and data governance (e.g., row‑level security, column masking, encryption). Mentor junior team members and contribute to overall team development through knowledge sharing and technical guidance. Provide regular updates on progress, risks, and blockers to leadership. Key Requirements5+ years of experience in data architecture, data engineering, or AI/ML solutions with ownership of technical delivery. Extensive hands‑on expertise withDatabricks , including Lakehouse architecture and Medallion design patterns. Strong experience implementing Databricks inAzure environments . Proficiency inPython and SQLfor building data pipelines and analytics solutions. Experience with large‑scale distributed systems and big data technologies (e.g., Apache Spark, Hadoop, Cassandra). Expertise in building and optimizing data ingestion frameworks, including Auto Loader, Delta Live Tables, and streaming solutions. Strong understanding of data security and governance practices within Databricks environments. Proven ability to lead projects end‑to‑end and manage multiple priorities simultaneously. Excellent communication skills and a collaborative mindset, with the ability to work across internal teams and external partners. Preferred QualificationsDatabricks certifications in data engineering or related disciplines. Experience with Azure networking or security configurations. Exposure to AI/ML workloads and advanced analytics use cases. Call to ActionIf you are a hands‑on Databricks expert who thrives in complex environments and enjoys leading high‑impact data initiatives, we encourage you to apply. Please submit your resume and contact information toThe Intersect Groupfor immediate consideration.#J-18808-Ljbffr