JOBSEARCHER

Lead Data Engineer

ROLE: Lead Data Engineer – Hands-On
Client: MetLife
Location: Cary, NC — On-site / Hybrid
Experience: 12–18 Years
Employment Type: Full-Time The engagement involves building a centralized, AI-first enterprise Data Hubfor a global insurance and financial services organization usingAzure Databricks. The platform ingests150+ inbound data feedsand distributes data to35+ downstream systems, leveraging aMedallion Architecture — Bronze / Silver / Gold. AI is embedded across ingestion, canonical mapping, data quality, reconciliation, and business-user access. This is asenior hands-on technical leadership role. The selected candidate will own the end-to-end technical design of thedata and AI layers, build reference implementations, mentor engineering teams, and continue writing/reviewing production-gradePython, Scala, and PySpark code every week. Candidates who have NOT written or reviewed production code within the past year will NOT be considered. Data Platform Architecture & EngineeringDesign and implement enterprise-scale Lakehouse architecture. Define Bronze / Silver / Gold data contracts. Build scalable and reusable data platform components. Metadata-Driven IngestionDesign parameterized ingestion frameworks for: Batch files CDC feeds Build schema inference, data profiling, metadata and lineage capabilities. Develop production-gradePython, Scala and PySparksolutions. Establish coding and testing standards. Conduct PR/code reviews. Optimize Spark workloads, clusters and pipelines. Implement performance and cost optimization strategies. Unity Catalog Databricks Jobs & Workflows Cluster and pool management Model Serving ADLS Gen2 CI/CD & Infrastructure as CodeImplement CI/CD for Databricks and ADF usingAzure DevOps. Work withDatabricks Asset Bundles. Implement Infrastructure as Code usingTerraform. Automate data pipeline testing and deployments. Establish observability and operational monitoring. A major component of this role is designing and deliveringproduction-grade AI/LLM solutions. You will work on: AI-augmented ingestion. AI-assisted canonical mapping. Auto-generated bridge documentation. DML and canonical table generation. Source-to-canonical mapping with human review gates. Data drift and schema drift detection. Volume anomaly detection. Automated reconciliation. Synthetic privacy-preserving test data generation. Knowledge graphs. GPT-powered conversational interfaces. Text-to-SQL. Row-level and column-level security. Candidates must have 3+ years of hands-on experience designing and shipping LLM-based systems in production, including: Agentic AI workflows Tool-calling workflows Chunking strategies Embedding strategies Vector retrieval Hybrid retrieval LLM evaluation Regression test suites Accuracy tracking Hallucination monitoring Human-in-the-loop feedback Hands-on experience with at least one of: LangChain LangGraph Provider/platform experience: OpenAI Databricks Model Serving Strong enterprise security and governance experience is required across: Unity Catalog Access control PII standards Managed Identities RBAC POSIX ACLs Data privacy and protection The candidate will also participate in: AI Governance Forums 12–18 yearsof total experience in Data Engineering / Data Platform delivery Data contracts & schema design Unity Catalog Kafka / Spark Structured Streaming Spark performance tuning Enterprise Lakehouse / Medallion Architecture Azure security & governance 3+ years production LLM/GenAI engineering RAG / Agentic AI / Tool Calling LangChain / LlamaIndex / LangGraph Azure OpenAI / OpenAI / Databricks Model Serving RDF / SPARQL Neo4j Graph modelling over Lakehouse Text-to-SQL ML-based anomaly detection Finance Close Subledger LLMOps / MLOps LLM cost governance AI observability dbt Workday Workday Prism Accounting Center We are looking for ahands-on Lead Data Engineer / Data Architect / Principal Data Engineerwho can operate at botharchitecture and engineering levels. You should be comfortabledesigning the architecture, making technical decisions, writing production code, reviewing code, troubleshooting Spark workloads, mentoring engineers, and presenting architecture decisions to senior technical and business stakeholders. This is NOT a pure Data Architect, Solution Architect, Engineering Manager, or AI Strategy role. Recent hands-on production coding experience is mandatory. #Nazztec #NazztecHiring #JoinNazztec #Hiring #UrgentHiring #LeadDataEngineer #DataEngineer #SeniorDataEngineer #PrincipalDataEngineer #DataArchitect #DataEngineering #DataPlatform #DataPlatformEngineer #BigData #BigDataEngineering #Azure #MicrosoftAzure #AzureDatabricks #Databricks #DeltaLake #UnityCatalog #AzureDataFactory #ADF #ADLS #ADLSGen2 #AzureEventHubs #Kafka #Spark #ApacheSpark #PySpark #Scala #Python #SQL #DataLake #Lakehouse #LakehouseArchitecture #MedallionArchitecture #BronzeSilverGold #DataArchitecture #DataModeling #DataEngineeringJobs #AzureJobs #DatabricksJobs #AI #ArtificialIntelligence #GenerativeAI #GenAI #LLM #LLMEngineering #LLMEngineer #LLMJobs #RAG #RAGPipeline #RetrievalAugmentedGeneration #AgenticAI #AgenticWorkflows #ToolCalling #LangChain #LlamaIndex #LangGraph #AzureOpenAI #OpenAI #DatabricksModelServing #PromptEngineering #VectorDatabase #VectorSearch #HybridSearch #Embeddings #SemanticSearch #TextToSQL #SemanticLayer #KnowledgeGraph #Neo4j #RDF #SPARQL #MLOps #LLMOps #AIEngineering #AIEngineer #MachineLearning #AnomalyDetection #DataQuality #DataGovernance #DataSecurity #DataLineage #DataCatalog #Metadata #MetadataDriven #DataContracts #SchemaEvolution #DataProfiling #DataObservability #DataOps #DevOps #AzureDevOps #Terraform #InfrastructureAsCode #IaC #CICD #ContinuousIntegration #ContinuousDeployment #SparkTuning #PerformanceTuning #CloudDataEngineering #CloudArchitecture #EnterpriseData #EnterpriseArchitecture #FinancialServices #Insurance #InsuranceTechnology #FinTech #MetLife #CaryNC #NorthCarolinaJobs #USJobs #USAJobs #USCitizen #USCitizenJobs #TechJobs #ITJobs #TechnologyJobs #DataScience #CloudJobs #DataProfessionals #TechHiring #ITHiring #DataHiring #AIJobs #RemoteJobs #HybridJobs #OnsiteJobs