{"schemaVersion":"jobsearcher.job.v1","id":"8aa2a0ea4ea94830a5dbaea2","url":"https://jobsearcher.com/jobs/8aa2a0ea4ea94830a5dbaea2","canonicalUrl":"https://jobsearcher.com/jobs/8aa2a0ea4ea94830a5dbaea2","title":"Data Engineer","description":"Cloud Data EngineerWe are seeking a high-caliber Senior Databricks Engineer to lead the architecture, development, and optimization of our next-generation Lakehouse platform. This is a critical role for a technical leader with 6+ years of deep data engineering expertise, specifically within the Databricks ecosystem. The ideal candidate will drive technical direction, establish robust data governance, and deliver high-impact, scalable data solutions that bridge the gap between raw data and actionable business intelligence.ResponsibilitiesData Pipeline Development & ManagementIngestion & Transformation: Design and optimize high-volume ETL/ELT pipelines using Delta Live Tables (DLT) and PySpark, ensuring data integrity across the Bronze, Silver, and Gold layers.Workflow Orchestration: Develop and maintain sophisticated pipelines using Databricks Workflows or Airflow, focusing on modularity, reusability, and automated error handling.Streaming & Real-time Integration: Implement real-time data flows utilizing Structured Streaming and Kafka/Event Hubs to enable immediate data availability for downstream consumption.Data Security & Privacy: Enforce data anonymization and fine-grained access controls to ensure compliance with global regulations (GDPR/CCPA/HIPAA).DataOps & DevOps: Implement CI/CD patterns using Databricks Asset Bundles (DABs), Terraform, and Git to automate environment parity and deployments.Data Ecosystem Management & MonitoringOpen Table Formats: Manage and optimize Delta Lake storage, utilizing advanced features like Liquid Clustering, Z-Ordering, and Change Data Feed (CDF).Compute Engine Optimization: Drive cost efficiency and performance by optimizing Spark configurations, Photon engine utilization, and Serverless SQL Warehouses.Observability & Monitoring: Integrate comprehensive monitoring and alerting (e.g., Databricks System Tables, Grafana, or Splunk) to rapidly identify bottlenecks and troubleshoot production issues.Qualifications6+ Years of hands-on, progressive experience in Data Engineering, with at least 5 years focused heavily on the Databricks platform.Architectural Understanding: Expert knowledge of Medallion Architecture, Data Vault 2.0 or Dimensional Modeling, and modern Lakehouse design patterns.Scale Expertise: Proven track record of building and managing large-scale data infrastructure (Petabyte-scale) in cloud-native environments.Industry Experience: Experience in the Insurance or Financial Services industry is preferred (focusing on claims, policy, or risk data).Technical Toolset:Cloud Environment: Azure (preferred), AWS.Databricks Stack: Unity Catalog, Delta Live Tables, Databricks SQL, MLflow.Core Languages: Expert-level SQL, Python, and PySpark.Supporting Tools: dbt (Databricks adapter), Git, and Orchestration tools","company":"EXL","rawCompany":"exl","city":"Andrews Air Force Base","state":"MD","isRemote":false,"isActive":false,"createdAt":"2026-07-22T00:55:10.064Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Data Engineer","description":"Cloud Data EngineerWe are seeking a high-caliber Senior Databricks Engineer to lead the architecture, development, and optimization of our next-generation Lakehouse platform. This is a critical role for a technical leader with 6+ years of deep data engineering expertise, specifically within the Databricks ecosystem. The ideal candidate will drive technical direction, establish robust data governance, and deliver high-impact, scalable data solutions that bridge the gap between raw data and actionable business intelligence.ResponsibilitiesData Pipeline Development & ManagementIngestion & Transformation: Design and optimize high-volume ETL/ELT pipelines using Delta Live Tables (DLT) and PySpark, ensuring data integrity across the Bronze, Silver, and Gold layers.Workflow Orchestration: Develop and maintain sophisticated pipelines using Databricks Workflows or Airflow, focusing on modularity, reusability, and automated error handling.Streaming & Real-time Integration: Implement real-time data flows utilizing Structured Streaming and Kafka/Event Hubs to enable immediate data availability for downstream consumption.Data Security & Privacy: Enforce data anonymization and fine-grained access controls to ensure compliance with global regulations (GDPR/CCPA/HIPAA).DataOps & DevOps: Implement CI/CD patterns using Databricks Asset Bundles (DABs), Terraform, and Git to automate environment parity and deployments.Data Ecosystem Management & MonitoringOpen Table Formats: Manage and optimize Delta Lake storage, utilizing advanced features like Liquid Clustering, Z-Ordering, and Change Data Feed (CDF).Compute Engine Optimization: Drive cost efficiency and performance by optimizing Spark configurations, Photon engine utilization, and Serverless SQL Warehouses.Observability & Monitoring: Integrate comprehensive monitoring and alerting (e.g., Databricks System Tables, Grafana, or Splunk) to rapidly identify bottlenecks and troubleshoot production issues.Qualifications6+ Years of hands-on, progressive experience in Data Engineering, with at least 5 years focused heavily on the Databricks platform.Architectural Understanding: Expert knowledge of Medallion Architecture, Data Vault 2.0 or Dimensional Modeling, and modern Lakehouse design patterns.Scale Expertise: Proven track record of building and managing large-scale data infrastructure (Petabyte-scale) in cloud-native environments.Industry Experience: Experience in the Insurance or Financial Services industry is preferred (focusing on claims, policy, or risk data).Technical Toolset:Cloud Environment: Azure (preferred), AWS.Databricks Stack: Unity Catalog, Delta Live Tables, Databricks SQL, MLflow.Core Languages: Expert-level SQL, Python, and PySpark.Supporting Tools: dbt (Databricks adapter), Git, and Orchestration tools","datePosted":"2026-07-22T00:55:10.064Z","dateModified":"2026-07-22T00:55:10.064Z","hiringOrganization":{"@type":"Organization","name":"EXL","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Andrews Air Force Base","addressRegion":"MD","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"8aa2a0ea4ea94830a5dbaea2"},"url":"https://jobsearcher.com/jobs/8aa2a0ea4ea94830a5dbaea2"}}