{"schemaVersion":"jobsearcher.job.v1","id":"9b9c83fd3e87f96c24c8c212","url":"https://jobsearcher.com/jobs/9b9c83fd3e87f96c24c8c212","canonicalUrl":"https://jobsearcher.com/jobs/9b9c83fd3e87f96c24c8c212","title":"Lead Data Engineer","description":"Expert-level Python, Scala and PySpark: production-ready, modular, well-tested solutions; Spark workload troubleshooting; optimizing large-scale batch and streaming pipelines using Delta Lake.Strong SQL and data modelling (dimensional and normalised), schema design, data contracts.Databricks expertise: Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.Azure data stack: ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks), Azure Event Hubs.3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, chunking and embedding strategy, vector and hybrid retrieval, prompt engineering.Evaluation discipline: golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback.Hands-on with LangChain, LlamaIndex or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).Metadata-driven frameworks: schema inference, data profiling, lineage, catalogs.12-18 years of total experience in data engineering / data platform delivery.Proven enterprise-scale delivery of a medallion / lakehouse architecture.Azure security and governance: Entra ID, managed identities, RBAC, POSIX ACLs, Key Vault, private endpoints, PII handling.CI/CD and IaC: Azure DevOps, Terraform, Databricks Asset Bundles, automated testing of data pipelines.Clear technical writing and ability to present and defend designs to engineers and non-technical stakeholders.","company":"Riim","rawCompany":"riim","city":"Cary","state":"NC","isRemote":false,"isActive":false,"createdAt":"2026-10-09T09:01:10.345Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"},{"code":"15-2051.00","title":"Data Scientists","slug":"data-scientists"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Lead Data Engineer","description":"Expert-level Python, Scala and PySpark: production-ready, modular, well-tested solutions; Spark workload troubleshooting; optimizing large-scale batch and streaming pipelines using Delta Lake.Strong SQL and data modelling (dimensional and normalised), schema design, data contracts.Databricks expertise: Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.Azure data stack: ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks), Azure Event Hubs.3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, chunking and embedding strategy, vector and hybrid retrieval, prompt engineering.Evaluation discipline: golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback.Hands-on with LangChain, LlamaIndex or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).Metadata-driven frameworks: schema inference, data profiling, lineage, catalogs.12-18 years of total experience in data engineering / data platform delivery.Proven enterprise-scale delivery of a medallion / lakehouse architecture.Azure security and governance: Entra ID, managed identities, RBAC, POSIX ACLs, Key Vault, private endpoints, PII handling.CI/CD and IaC: Azure DevOps, Terraform, Databricks Asset Bundles, automated testing of data pipelines.Clear technical writing and ability to present and defend designs to engineers and non-technical stakeholders.","datePosted":"2026-10-09T09:01:10.345Z","dateModified":"2026-10-09T09:01:10.345Z","hiringOrganization":{"@type":"Organization","name":"Riim","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Cary","addressRegion":"NC","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"9b9c83fd3e87f96c24c8c212"},"url":"https://jobsearcher.com/jobs/9b9c83fd3e87f96c24c8c212"}}