{"schemaVersion":"jobsearcher.job.v1","id":"fc8bc482abf7bf9187efc9a5","url":"https://jobsearcher.com/jobs/fc8bc482abf7bf9187efc9a5","canonicalUrl":"https://jobsearcher.com/jobs/fc8bc482abf7bf9187efc9a5","title":"Lead Data Engineer","description":"Job Title: Lead Data Engineer\r\nLocation: Cary, NCMandatory Skills - Must-Have:\r\nPython, PySpark, Scala and SQL - strong production-level data engineering skills.\r\nAzure Databricks - Delta Lake, Spark performance tuning, Jobs/Workflows and Unity Catalog.\r\nAzure data services - ADLS Gen2 and Azure Data Factory (ADF).\r\nLakehouse / Medallion Architecture - proven experience implementing Bronze, Silver and Gold layers at enterprise scale.\r\nData Pipeline Engineering - metadata-driven ingestion frameworks, data transformation, data quality and large-scale pipeline optimization.\r\nProduction AI/LLM Experience - hands-on experience designing and deploying LLM-based solutions, including RAG, embeddings, retrieval and prompt engineering.\r\nLLM Frameworks - experience with LangChain, LlamaIndex or LangGraph, plus Azure OpenAI, OpenAI or Databricks Model Serving.\r\nAzure DevOps / CI-CD - experience automating data pipeline deployments; Terraform and Databricks Asset Bundles are highly relevant.\r\nTechnical Leadership - ability to design solutions, write production code, review code and guide data engineering teams.\r\nFinancial services or insurance data, especially financial reconciliation, general ledger or accounting data.\r\nAI-driven data quality, anomaly detection and automated reconciliation.\r\nKnowledge graphs, ontologies, semantic layers or enterprise Text-to-SQL solutions.\r\nAzure security, data governance, PII protection and fine-grained access controls.\r\nMLOps/LLMOps, model evaluation and production AI monitoring.Important: This is a hands-on Lead Data Engineer position, not a purely managerial or coordination role. Candidates must demonstrate recent production coding experience.","company":"Via Dice","rawCompany":"via dice","city":"Cary","state":"NC","isRemote":false,"isActive":true,"createdAt":"2026-11-20T01:31:33.795Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Lead Data Engineer","description":"Job Title: Lead Data Engineer\r\nLocation: Cary, NCMandatory Skills - Must-Have:\r\nPython, PySpark, Scala and SQL - strong production-level data engineering skills.\r\nAzure Databricks - Delta Lake, Spark performance tuning, Jobs/Workflows and Unity Catalog.\r\nAzure data services - ADLS Gen2 and Azure Data Factory (ADF).\r\nLakehouse / Medallion Architecture - proven experience implementing Bronze, Silver and Gold layers at enterprise scale.\r\nData Pipeline Engineering - metadata-driven ingestion frameworks, data transformation, data quality and large-scale pipeline optimization.\r\nProduction AI/LLM Experience - hands-on experience designing and deploying LLM-based solutions, including RAG, embeddings, retrieval and prompt engineering.\r\nLLM Frameworks - experience with LangChain, LlamaIndex or LangGraph, plus Azure OpenAI, OpenAI or Databricks Model Serving.\r\nAzure DevOps / CI-CD - experience automating data pipeline deployments; Terraform and Databricks Asset Bundles are highly relevant.\r\nTechnical Leadership - ability to design solutions, write production code, review code and guide data engineering teams.\r\nFinancial services or insurance data, especially financial reconciliation, general ledger or accounting data.\r\nAI-driven data quality, anomaly detection and automated reconciliation.\r\nKnowledge graphs, ontologies, semantic layers or enterprise Text-to-SQL solutions.\r\nAzure security, data governance, PII protection and fine-grained access controls.\r\nMLOps/LLMOps, model evaluation and production AI monitoring.Important: This is a hands-on Lead Data Engineer position, not a purely managerial or coordination role. Candidates must demonstrate recent production coding experience.","datePosted":"2026-11-20T01:31:33.795Z","dateModified":"2026-11-20T01:31:33.795Z","hiringOrganization":{"@type":"Organization","name":"Via Dice","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Cary","addressRegion":"NC","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"fc8bc482abf7bf9187efc9a5"},"url":"https://jobsearcher.com/jobs/fc8bc482abf7bf9187efc9a5"}}