{"schemaVersion":"jobsearcher.job.v1","id":"c9d8f44744035a5cbe8b915b","url":"https://jobsearcher.com/jobs/c9d8f44744035a5cbe8b915b","canonicalUrl":"https://jobsearcher.com/jobs/c9d8f44744035a5cbe8b915b","title":"Azure Databricks & Agentic AI Architect","description":"Job Title: Azure Databricks & Agentic AI Architect\n\nLocation: Chicago, IL (Hybrid- 3Days a week)\n\nDuration: 12+ Months\n\nJob Description:\n\nRole Summary:\n\nWe are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.\n\nThe ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.\n\nKey Responsibilities\n\nAgentic Data Engineering Leadership\n\n· Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.\n\n· Define autonomous workflows using AI Agents for:\n\n· Data ingestion\n\n· Data mapping\n\n· Schema evolution\n\n· Data quality remediation\n\n· Metadata Enrichment\n\n· Pipeline optimization\n\n· Root cause analysis\n\n· Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.\n\nAI-Driven Data Lifecycle (AI-DLC)\n\n· Lead architecture for AI-enabled Data Development Lifecycle across:\n\n· Requirement analysis\n\n· Data modeling\n\n· Pipeline generation\n\n· Automated testing\n\n· Code review\n\n· Documentation\n\n· Deployment\n\n· Monitoring\n\n· Implement AI copilots to accelerate developer productivity.\n\n· Enable automated lineage creation and intelligent impact analysis.\n\nLakehouse & Data Platform Architecture\n\n· Design scalable Lakehouse platforms using:\n\n· Azure Databricks\n\n· Delta Lake\n\n· Unity Catalog\n\n· ADLS Gen2\n\n· Databricks Workflows\n\n· Delta Live Tables\n\nEnterprise GenAI Integration\n\n· Architect RAG-based solutions using enterprise data assets.\n\n· Design agent orchestration frameworks using:\n\n· Azure OpenAI\n\n· LangGraph\n\n· Semantic Kernel\n\n· AutoGen\n\n· MCP-enabled architectures\n\n· Build domain-specific AI agents supporting Data Engineering and Analytics teams.\n\nAI Governance & Responsible AI\n\n· Define guardrails for enterprise GenAI adoption.\n\n· Implement:\n\n· Prompt governance\n\n· Observability\n\n· Cost monitoring\n\n· Auditability\n\n· Explainability\n\n· Security controls\n\n· Establish governance models for autonomous AI agents.\n\nAI-Powered Platform Optimization\n\n· Design self-healing data pipelines.\n\n· Implement AI-driven:\n\n· Incident triage\n\n· Failure prediction\n\n· Capacity planning\n\n· Cost optimization\n\n· SLA monitoring\n\n· Enable intelligent workload placement and model routing.\n\nTechnical Skills\n\n· Data Platform\n\n· Azure Databricks\n\n· Delta Lake\n\n· Unity Catalog\n\n· Azure Data Factory\n\nAI & Agentic Frameworks\n\n· Azure OpenAI\n\n· Knowledge Graph\n\n· RAG Architecture\n\n· LangChain\n\n· LangGraph\n\n· MCP Protocol\n\n· Vector Databases\n\n· AI Agent Orchestration\n\nData Engineering\n\n· PySpark\n\n· Spark SQL\n\n· Python\n\n· SQL\n\n· ELT/ETL Modernization\n\nDevOps & AI-DLC\n\n· Azure DevOps\n\n· GitHub Actions\n\n· CI/CD\n\n· MLOps\n\n· LLMOps\n\n· Evaluation Frameworks\n\n· AI Testing Frameworks\n\nLeadership Expectations\n\n· Drive AI-First Data Engineering transformation.\n\n· Define enterprise patterns, accelerators, and reusable AI agents.\n\n· Mentor architects, data engineers, and AI engineers.\n\n· Lead executive conversations on AI adoption, ROI, and transformation roadmaps.\n\nPreferred Certifications\n\n· Databricks Certified Data Engineer Professional\n\n· Databricks Certified Solution Architect\n\n· Microsoft Azure Solution Architect (AZ-305)\n\n· Azure Data Engineer (DP-203)\n\n· Microsoft Applied Skills - Azure OpenAI\n\n· Generative AI / Agentic AI Certifications\n\nSuccess Metrics\n\n· 30-50% Data Engineering productivity improvement.\n\n· Reduction in manual pipeline development effort.\n\n· Improved data quality and governance compliance.\n\n· Measurable ROI from Agentic AI adoption.\n\n· Expansion of reusable AI agents across programs.","company":"Eapps Tech Dba Magicforce","rawCompany":"eapps tech dba magicforce","city":"Chicago","state":"IL","isRemote":false,"isActive":false,"createdAt":"2026-09-24T11:03:15.756Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Azure Databricks & Agentic AI Architect","description":"Job Title: Azure Databricks & Agentic AI Architect\n\nLocation: Chicago, IL (Hybrid- 3Days a week)\n\nDuration: 12+ Months\n\nJob Description:\n\nRole Summary:\n\nWe are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.\n\nThe ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.\n\nKey Responsibilities\n\nAgentic Data Engineering Leadership\n\n· Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.\n\n· Define autonomous workflows using AI Agents for:\n\n· Data ingestion\n\n· Data mapping\n\n· Schema evolution\n\n· Data quality remediation\n\n· Metadata Enrichment\n\n· Pipeline optimization\n\n· Root cause analysis\n\n· Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.\n\nAI-Driven Data Lifecycle (AI-DLC)\n\n· Lead architecture for AI-enabled Data Development Lifecycle across:\n\n· Requirement analysis\n\n· Data modeling\n\n· Pipeline generation\n\n· Automated testing\n\n· Code review\n\n· Documentation\n\n· Deployment\n\n· Monitoring\n\n· Implement AI copilots to accelerate developer productivity.\n\n· Enable automated lineage creation and intelligent impact analysis.\n\nLakehouse & Data Platform Architecture\n\n· Design scalable Lakehouse platforms using:\n\n· Azure Databricks\n\n· Delta Lake\n\n· Unity Catalog\n\n· ADLS Gen2\n\n· Databricks Workflows\n\n· Delta Live Tables\n\nEnterprise GenAI Integration\n\n· Architect RAG-based solutions using enterprise data assets.\n\n· Design agent orchestration frameworks using:\n\n· Azure OpenAI\n\n· LangGraph\n\n· Semantic Kernel\n\n· AutoGen\n\n· MCP-enabled architectures\n\n· Build domain-specific AI agents supporting Data Engineering and Analytics teams.\n\nAI Governance & Responsible AI\n\n· Define guardrails for enterprise GenAI adoption.\n\n· Implement:\n\n· Prompt governance\n\n· Observability\n\n· Cost monitoring\n\n· Auditability\n\n· Explainability\n\n· Security controls\n\n· Establish governance models for autonomous AI agents.\n\nAI-Powered Platform Optimization\n\n· Design self-healing data pipelines.\n\n· Implement AI-driven:\n\n· Incident triage\n\n· Failure prediction\n\n· Capacity planning\n\n· Cost optimization\n\n· SLA monitoring\n\n· Enable intelligent workload placement and model routing.\n\nTechnical Skills\n\n· Data Platform\n\n· Azure Databricks\n\n· Delta Lake\n\n· Unity Catalog\n\n· Azure Data Factory\n\nAI & Agentic Frameworks\n\n· Azure OpenAI\n\n· Knowledge Graph\n\n· RAG Architecture\n\n· LangChain\n\n· LangGraph\n\n· MCP Protocol\n\n· Vector Databases\n\n· AI Agent Orchestration\n\nData Engineering\n\n· PySpark\n\n· Spark SQL\n\n· Python\n\n· SQL\n\n· ELT/ETL Modernization\n\nDevOps & AI-DLC\n\n· Azure DevOps\n\n· GitHub Actions\n\n· CI/CD\n\n· MLOps\n\n· LLMOps\n\n· Evaluation Frameworks\n\n· AI Testing Frameworks\n\nLeadership Expectations\n\n· Drive AI-First Data Engineering transformation.\n\n· Define enterprise patterns, accelerators, and reusable AI agents.\n\n· Mentor architects, data engineers, and AI engineers.\n\n· Lead executive conversations on AI adoption, ROI, and transformation roadmaps.\n\nPreferred Certifications\n\n· Databricks Certified Data Engineer Professional\n\n· Databricks Certified Solution Architect\n\n· Microsoft Azure Solution Architect (AZ-305)\n\n· Azure Data Engineer (DP-203)\n\n· Microsoft Applied Skills - Azure OpenAI\n\n· Generative AI / Agentic AI Certifications\n\nSuccess Metrics\n\n· 30-50% Data Engineering productivity improvement.\n\n· Reduction in manual pipeline development effort.\n\n· Improved data quality and governance compliance.\n\n· Measurable ROI from Agentic AI adoption.\n\n· Expansion of reusable AI agents across programs.","datePosted":"2026-09-24T11:03:15.756Z","dateModified":"2026-09-24T11:03:15.756Z","hiringOrganization":{"@type":"Organization","name":"Eapps Tech Dba Magicforce","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Chicago","addressRegion":"IL","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"c9d8f44744035a5cbe8b915b"},"url":"https://jobsearcher.com/jobs/c9d8f44744035a5cbe8b915b"}}