JOBSEARCHER

Lead Data Engineer

Via DiceCary, NCL6 LeadNovember 20th, 2026
Job Title: Lead Data Engineer
Location: Cary, NCMandatory Skills - Must-Have: Python, PySpark, Scala and SQL - strong production-level data engineering skills. Azure Databricks - Delta Lake, Spark performance tuning, Jobs/Workflows and Unity Catalog. Azure data services - ADLS Gen2 and Azure Data Factory (ADF). Lakehouse / Medallion Architecture - proven experience implementing Bronze, Silver and Gold layers at enterprise scale. Data Pipeline Engineering - metadata-driven ingestion frameworks, data transformation, data quality and large-scale pipeline optimization. Production AI/LLM Experience - hands-on experience designing and deploying LLM-based solutions, including RAG, embeddings, retrieval and prompt engineering. LLM Frameworks - experience with LangChain, LlamaIndex or LangGraph, plus Azure OpenAI, OpenAI or Databricks Model Serving. Azure DevOps / CI-CD - experience automating data pipeline deployments; Terraform and Databricks Asset Bundles are highly relevant. Technical Leadership - ability to design solutions, write production code, review code and guide data engineering teams. Financial services or insurance data, especially financial reconciliation, general ledger or accounting data. AI-driven data quality, anomaly detection and automated reconciliation. Knowledge graphs, ontologies, semantic layers or enterprise Text-to-SQL solutions. Azure security, data governance, PII protection and fine-grained access controls. MLOps/LLMOps, model evaluation and production AI monitoring.
Important: This is a hands-on Lead Data Engineer position, not a purely managerial or coordination role. Candidates must demonstrate recent production coding experience.