Lead Data Engineer
ROLE: Lead Data Engineer – Hands-On
Client: MetLife
Location: Cary, NC — On-site / Hybrid
Experience: 12–18 Years
Employment Type:
Full-Time
The engagement involves building a centralized, AI-first enterprise Data Hubfor a global insurance and financial services organization usingAzure Databricks.
The platform ingests150+ inbound data feedsand distributes data to35+ downstream systems, leveraging aMedallion Architecture — Bronze / Silver / Gold.
AI is embedded across ingestion, canonical mapping, data quality, reconciliation, and business-user access.
This is asenior hands-on technical leadership role. The selected candidate will own the end-to-end technical design of thedata and AI layers, build reference implementations, mentor engineering teams, and continue writing/reviewing production-gradePython, Scala, and PySpark code every week.
Candidates who have NOT written or reviewed production code within the past year will NOT be considered.
Data Platform Architecture & EngineeringDesign and implement enterprise-scale Lakehouse architecture.
Define Bronze / Silver / Gold data contracts.
Build scalable and reusable data platform components.
Metadata-Driven IngestionDesign parameterized ingestion frameworks for:
Batch files
CDC feeds
Build schema inference, data profiling, metadata and lineage capabilities.
Develop production-gradePython, Scala and PySparksolutions.
Establish coding and testing standards.
Conduct PR/code reviews.
Optimize Spark workloads, clusters and pipelines.
Implement performance and cost optimization strategies.
Unity Catalog
Databricks Jobs & Workflows
Cluster and pool management
Model Serving
ADLS Gen2
CI/CD & Infrastructure as CodeImplement CI/CD for Databricks and ADF usingAzure DevOps.
Work withDatabricks Asset Bundles.
Implement Infrastructure as Code usingTerraform.
Automate data pipeline testing and deployments.
Establish observability and operational monitoring.
A major component of this role is designing and deliveringproduction-grade AI/LLM solutions.
You will work on: AI-augmented ingestion.
AI-assisted canonical mapping.
Auto-generated bridge documentation.
DML and canonical table generation.
Source-to-canonical mapping with human review gates.
Data drift and schema drift detection.
Volume anomaly detection.
Automated reconciliation.
Synthetic privacy-preserving test data generation.
Knowledge graphs.
GPT-powered conversational interfaces.
Text-to-SQL.
Row-level and column-level security.
Candidates must have 3+ years of hands-on experience designing and shipping LLM-based systems in production, including:
Agentic AI workflows
Tool-calling workflows
Chunking strategies
Embedding strategies
Vector retrieval
Hybrid retrieval
LLM evaluation
Regression test suites
Accuracy tracking
Hallucination monitoring
Human-in-the-loop feedback
Hands-on experience with at least one of: LangChain
LangGraph
Provider/platform experience: OpenAI
Databricks Model Serving
Strong enterprise security and governance experience is required across: Unity Catalog
Access control
PII standards
Managed Identities
RBAC
POSIX ACLs
Data privacy and protection
The candidate will also participate in: AI Governance Forums
12–18 yearsof total experience in Data Engineering / Data Platform delivery
Data contracts & schema design
Unity Catalog
Kafka / Spark Structured Streaming
Spark performance tuning
Enterprise Lakehouse / Medallion Architecture
Azure security & governance
3+ years production LLM/GenAI engineering
RAG / Agentic AI / Tool Calling
LangChain / LlamaIndex / LangGraph
Azure OpenAI / OpenAI / Databricks Model Serving
RDF / SPARQL
Neo4j
Graph modelling over Lakehouse
Text-to-SQL
ML-based anomaly detection
Finance Close
Subledger
LLMOps / MLOps
LLM cost governance
AI observability
dbt
Workday
Workday Prism
Accounting Center
We are looking for ahands-on Lead Data Engineer / Data Architect / Principal Data Engineerwho can operate at botharchitecture and engineering levels.
You should be comfortabledesigning the architecture, making technical decisions, writing production code, reviewing code, troubleshooting Spark workloads, mentoring engineers, and presenting architecture decisions to senior technical and business stakeholders.
This is NOT a pure Data Architect, Solution Architect, Engineering Manager, or AI Strategy role.
Recent hands-on production coding experience is mandatory.
#Nazztec #NazztecHiring #JoinNazztec #Hiring #UrgentHiring #LeadDataEngineer #DataEngineer #SeniorDataEngineer #PrincipalDataEngineer #DataArchitect #DataEngineering #DataPlatform #DataPlatformEngineer #BigData #BigDataEngineering #Azure #MicrosoftAzure #AzureDatabricks #Databricks #DeltaLake #UnityCatalog #AzureDataFactory #ADF #ADLS #ADLSGen2 #AzureEventHubs #Kafka #Spark #ApacheSpark #PySpark #Scala #Python #SQL #DataLake #Lakehouse #LakehouseArchitecture #MedallionArchitecture #BronzeSilverGold #DataArchitecture #DataModeling #DataEngineeringJobs #AzureJobs #DatabricksJobs #AI #ArtificialIntelligence #GenerativeAI #GenAI #LLM #LLMEngineering #LLMEngineer #LLMJobs #RAG #RAGPipeline #RetrievalAugmentedGeneration #AgenticAI #AgenticWorkflows #ToolCalling #LangChain #LlamaIndex #LangGraph #AzureOpenAI #OpenAI #DatabricksModelServing #PromptEngineering #VectorDatabase #VectorSearch #HybridSearch #Embeddings #SemanticSearch #TextToSQL #SemanticLayer #KnowledgeGraph #Neo4j #RDF #SPARQL #MLOps #LLMOps #AIEngineering #AIEngineer #MachineLearning #AnomalyDetection #DataQuality #DataGovernance #DataSecurity #DataLineage #DataCatalog #Metadata #MetadataDriven #DataContracts #SchemaEvolution #DataProfiling #DataObservability #DataOps #DevOps #AzureDevOps #Terraform #InfrastructureAsCode #IaC #CICD #ContinuousIntegration #ContinuousDeployment #SparkTuning #PerformanceTuning #CloudDataEngineering #CloudArchitecture #EnterpriseData #EnterpriseArchitecture #FinancialServices #Insurance #InsuranceTechnology #FinTech #MetLife #CaryNC #NorthCarolinaJobs #USJobs #USAJobs #USCitizen #USCitizenJobs #TechJobs #ITJobs #TechnologyJobs #DataScience #CloudJobs #DataProfessionals #TechHiring #ITHiring #DataHiring #AIJobs #RemoteJobs #HybridJobs #OnsiteJobs