Senior Data Engineer (SQL, Databricks, CDC)

Senior Data EngineerResponsibilities:Analyze the current Provider subject area architecture, data flows, and processing patterns.Assess the feasibility of converting the Provider subject area from full-refresh processing to incremental processing.Identify source-system capabilities and determine availability of change data capture (CDC), timestamps, or other incremental processing mechanisms.Evaluate impacts to downstream applications, reporting, and analytics consumers.Develop recommended incremental processing design patterns and implementation approaches.Create reconciliation and validation strategies to compare incremental and full-refresh outputs.Collaborate with data architecture and data modeling teams to align future-state designs with enterprise standards.Identify technical debt, performance bottlenecks, and opportunities for optimization within the Provider data pipelines.Produce technical documentation, architecture diagrams, findings, and implementation recommendations.Participate in Agile ceremonies, working sessions, backlog refinement, and technical design reviews.Required Skills:Advanced SQL development and data analysis skills.Experience designing and implementing incremental processing and change data capture (CDC) solutions.Experience performing data lineage analysis and documenting source-to-target mappings.Strong understanding of ETL pipeline development and optimization.Experience working with large-scale healthcare datasets.Familiarity with provider, claims, enrollment, or healthcare master data domains.Experience with cloud data platforms such as Databricks, Spark, or equivalent distributed processing technologies.Strong troubleshooting, data profiling, and data quality analysis skills.Experience creating technical documentation, architecture diagrams, and engineering recommendations.Experience supporting healthcare analytics and reporting platforms.Experience working within Agile teams and utilizing Jira for delivery management.