{"schemaVersion":"jobsearcher.job.v1","id":"9ea1fa3304db9282d88c64d0","url":"https://jobsearcher.com/jobs/9ea1fa3304db9282d88c64d0","canonicalUrl":"https://jobsearcher.com/jobs/9ea1fa3304db9282d88c64d0","title":"Lead Data Engineer","description":"*Role: Lead Data Engineer (Hands-On)*\n\n*Location: Cary, NC (On-site / Hybrid)*\n\n*Experience: 12-18 years*\n\n*Employment: Full-Time*\n\n*Eligibility: Citizens only*\n\n*What the Role Owns:-*\n\n* Data platform architecture and engineering: lakehouse architecture (Bronze / Silver / Gold contracts, ADLS Gen2 zone layout, Delta Lake table design, partitioning, schema evolution, retention).\n* Metadata-driven, parameterized ingestion frameworks for batch files, database extracts, CDC feeds and streaming (Azure Event Hubs / Kafka, Spark Structured Streaming).\n* Canonical PySpark and Scala Spark jobs, coding and testing standards, PR reviews, production incident debugging, Spark cluster tuning and cost guardrails.\n* CI/CD for Databricks and ADF in Azure DevOps using Databricks Asset Bundles and Terraform; observability with Azure Monitor and Log Analytics.\n* AI-augmented ingestion and canonical mapping: auto-generated bridgedocuments, DML, canonical table definitions; AI-assisted source-to-canonical mapping with human review gate.\n* AI-driven data quality, anomaly detection (data drift, schema drift, volume shifts, reconciliation breaks), automated reconciliation, and synthetic privacy-preserving test data.\n* Semantic layer and knowledge graph, plus a GPT-powered conversational interface (text-to-SQL / semantic-layer retrieval) with row- and column-level security.\n* Governance and leadership: Unity Catalog (lineage, access control, PII standards), Architecture Review Boards and AI governance forums, mentoring engineers, documentation.\n\n*Must have Skills & Experience:-*\n\n* Expert-level Python, Scala and PySpark: production-ready, modular, well-tested solutions; Spark workload troubleshooting; optimizing large-scale batch and streaming pipelines using Delta Lake.\n* Strong SQL and data modelling (dimensional and normalised), schema design, data contracts.\n* Databricks expertise: Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.\n* Azure data stack: ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks), Azure Event Hubs.\n* 3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, chunking and embedding strategy, vector and hybrid retrieval, prompt engineering.\n* Evaluation discipline: golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback.\n* Hands-on with LangChain, LlamaIndex or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).\n* Metadata-driven frameworks: schema inference, data profiling, lineage, catalogs.\n* 12-18 years of total experience in data engineering / data platform delivery.\n* Proven enterprise-scale delivery of a medallion / lakehouse architecture.\n* Azure security and governance: Entra ID, managed identities, RBAC, POSIX ACLs, Key Vault, private endpoints, PII handling.\n* CI/CD and IaC: Azure DevOps, Terraform, Databricks Asset Bundles, automated testing of data pipelines.\n* Clear technical writing and ability to present and defend designs to engineers and non-technical stakeholders.\n\n*Strongly Preferred:-*\n\n* Knowledge graphs and ontologies (RDF/SPARQL, Neo4j, graph modelling over a lakehouse).\n* Text-to-SQL or semantic-layer-backed natural-language query systems at enterprise scale.\n* ML-based anomaly detection on time-series or transactional financial data.\n* Financial services or insurance domain (finance close, GL, subledger, reconciliation, actuarial data).\n* LLMOps / MLOps: model and prompt versioning, cost governance, observability.\n* Databricks Data Engineer Professional, Azure DP-203 / DP-700, or AZ-305 certification.\n* dbt, Great Expectations or similar; Workday, Prism or Accounting Center exposure.\n\nPay: $140,000.00 - $145,000.00 per year\n\nApplication Question(s):\n* Please provide your LinkedIn URL.\n* What is your current CTC?\n* What is your expected CTC?\n* How soon you can start?\n\nExperience:\n* data engineering : 10 years (Required)\n* LLM-based systems: 3 years (Required)\n\nLicense/Certification:\n* Databricks Data Professional (Required)\n\nWork Location: In person","company":"Megthink","rawCompany":"megthink","city":"Cary","state":"NC","isRemote":false,"isActive":false,"createdAt":"2026-10-12T08:40:00.036Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Lead Data Engineer","description":"*Role: Lead Data Engineer (Hands-On)*\n\n*Location: Cary, NC (On-site / Hybrid)*\n\n*Experience: 12-18 years*\n\n*Employment: Full-Time*\n\n*Eligibility: Citizens only*\n\n*What the Role Owns:-*\n\n* Data platform architecture and engineering: lakehouse architecture (Bronze / Silver / Gold contracts, ADLS Gen2 zone layout, Delta Lake table design, partitioning, schema evolution, retention).\n* Metadata-driven, parameterized ingestion frameworks for batch files, database extracts, CDC feeds and streaming (Azure Event Hubs / Kafka, Spark Structured Streaming).\n* Canonical PySpark and Scala Spark jobs, coding and testing standards, PR reviews, production incident debugging, Spark cluster tuning and cost guardrails.\n* CI/CD for Databricks and ADF in Azure DevOps using Databricks Asset Bundles and Terraform; observability with Azure Monitor and Log Analytics.\n* AI-augmented ingestion and canonical mapping: auto-generated bridgedocuments, DML, canonical table definitions; AI-assisted source-to-canonical mapping with human review gate.\n* AI-driven data quality, anomaly detection (data drift, schema drift, volume shifts, reconciliation breaks), automated reconciliation, and synthetic privacy-preserving test data.\n* Semantic layer and knowledge graph, plus a GPT-powered conversational interface (text-to-SQL / semantic-layer retrieval) with row- and column-level security.\n* Governance and leadership: Unity Catalog (lineage, access control, PII standards), Architecture Review Boards and AI governance forums, mentoring engineers, documentation.\n\n*Must have Skills & Experience:-*\n\n* Expert-level Python, Scala and PySpark: production-ready, modular, well-tested solutions; Spark workload troubleshooting; optimizing large-scale batch and streaming pipelines using Delta Lake.\n* Strong SQL and data modelling (dimensional and normalised), schema design, data contracts.\n* Databricks expertise: Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.\n* Azure data stack: ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks), Azure Event Hubs.\n* 3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, chunking and embedding strategy, vector and hybrid retrieval, prompt engineering.\n* Evaluation discipline: golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback.\n* Hands-on with LangChain, LlamaIndex or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).\n* Metadata-driven frameworks: schema inference, data profiling, lineage, catalogs.\n* 12-18 years of total experience in data engineering / data platform delivery.\n* Proven enterprise-scale delivery of a medallion / lakehouse architecture.\n* Azure security and governance: Entra ID, managed identities, RBAC, POSIX ACLs, Key Vault, private endpoints, PII handling.\n* CI/CD and IaC: Azure DevOps, Terraform, Databricks Asset Bundles, automated testing of data pipelines.\n* Clear technical writing and ability to present and defend designs to engineers and non-technical stakeholders.\n\n*Strongly Preferred:-*\n\n* Knowledge graphs and ontologies (RDF/SPARQL, Neo4j, graph modelling over a lakehouse).\n* Text-to-SQL or semantic-layer-backed natural-language query systems at enterprise scale.\n* ML-based anomaly detection on time-series or transactional financial data.\n* Financial services or insurance domain (finance close, GL, subledger, reconciliation, actuarial data).\n* LLMOps / MLOps: model and prompt versioning, cost governance, observability.\n* Databricks Data Engineer Professional, Azure DP-203 / DP-700, or AZ-305 certification.\n* dbt, Great Expectations or similar; Workday, Prism or Accounting Center exposure.\n\nPay: $140,000.00 - $145,000.00 per year\n\nApplication Question(s):\n* Please provide your LinkedIn URL.\n* What is your current CTC?\n* What is your expected CTC?\n* How soon you can start?\n\nExperience:\n* data engineering : 10 years (Required)\n* LLM-based systems: 3 years (Required)\n\nLicense/Certification:\n* Databricks Data Professional (Required)\n\nWork Location: In person","datePosted":"2026-10-12T08:40:00.036Z","dateModified":"2026-10-12T08:40:00.036Z","hiringOrganization":{"@type":"Organization","name":"Megthink","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Cary","addressRegion":"NC","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"9ea1fa3304db9282d88c64d0"},"url":"https://jobsearcher.com/jobs/9ea1fa3304db9282d88c64d0"}}