JOBSEARCHER

Sr Data Engineer (Azure Databricks)

Required Skills & Experience: 5-7+ years of Data Engineering experience building large-scale data platformsStrong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS) Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuningExperience developing and maintaining Delta Live Tables (DLT) pipelinesStrong SQL background with experience supporting legacy data warehouse environmentsExperience migrating SQL-based pipelines to modern Python/PySpark architecturesExperience with Infrastructure as Code (IaC) and Pipeline as Code methodologiesProficiency with YAML, JSON, and configuration-driven development patternsExperience automating CI/CD deployments using DevOps pipelinesStrong understanding of Medallion Architecture (Bronze, Silver, Gold) Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layersExperience troubleshooting, monitoring, and supporting production data pipelinesFamiliarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferredHealthcare industry experience preferredMedicare, payer, member enrollment, or claims data experience strongly preferred Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferredJob DescriptionDesign, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live TablesModernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutionsBuild configuration-driven data pipelines using YAML, JSON, and reusable framework patternsDevelop and maintain CI/CD deployment processes for data engineering workflowsImplement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layersPartner with upstream and downstream teams to ensure data quality, reliability, and scalabilityTroubleshoot and resolve production pipeline issues across Databricks and legacy SQL environmentsEnhance monitoring, observability, and operational support processesSupport Medicare and payer-focused initiatives involving member, claims, and operational datasetsFollow established architecture standards and engineering best practices to ensure platform consistency and maintainability