Data Engineer

Senior Data Engineer Fortune 50 Healthcare Brooksource Remote OverviewOur Fortune 50 Healthcare client is seeking a Senior Data Engineer to support our mission of improving the health and well-being of our members. This role will focus on building scalable, secure, data centric solutions and compliant data platforms that power analytics, clinical insights, and business decision-making across the enterprise.The ideal candidate will have strong experience with cloud-based data platforms, Databricks, PostgreSQL, and healthcare data, with a passion for delivering high-quality, trusted data solutions in a regulated environment.Key ResponsibilitiesData Engineering & Platform DevelopmentDesign, develop, and scalable data pipeline solutions using Databricks (Spark) and cloud-native servicesBuild and optimize ETL/ELT workflows for ingesting structured and unstructured healthcare data (claims, clinical, provider, and member data)Develop and maintain data models in PostgreSQL and enterprise data warehousesSupport Lakehouse architecture leveraging Databricks, Delta Lake, and cloud storageImprove performance, reliability, and cost-efficiency of data platformsHealthcare Data & ComplianceWork with healthcare datasets, including producer/agent, broker, commission, and distribution data, ensuring proper ingestion, normalization, and optimization for analytics and reportingEnsure compliance with HIPAA, HITECH, and enterprise data governance policiesImplement data security, encryption, masking, and access controlsMaintain data lineage, auditability, and regulatory reporting readinessAdvanced Data ProcessingBuild real-time and batch pipelines for analytics and operational use casesDevelop data transformations using PySpark and SQL within DatabricksLeverage PostgreSQL for transactional and analytical workloads where applicableIntegrate data from APIs, third-party vendors, and internal systemsCollaboration & Stakeholder EngagementPartner with business stakeholders to support data-driven initiatives and member acquisition strategiesTranslate insurance distribution, agent/producer, and marketing requirements into scalable, high-quality data solutionsSupport downstream consumers, including Power BI, marketing analytics teams, and operational reporting stakeholders, by delivering curated, analytics-ready datasetsTechnical LeadershipLead design and architecture discussions for enterprise data solutionsEstablish and enforce best practices in data engineering, testing, and CI/CDContribute to enterprise data strategy and platform modernizationAI & Advanced Analytics (Databricks Genie)Leverage Databricks Genie (AI/BI capabilities) to enable natural language querying and democratize data access for business stakeholdersDesign and optimize semantic layers and governed datasets that power Genie-driven insights with trusted, high-quality dataCollaborate with stakeholders to translate business questions into AI-assisted analytics workflows using DatabricksEnsure AI outputs are accurate, explainable, and compliant with healthcare data governance and HIPAA requirementsLeverage large language models (LLMs), including Anthropic Claude, to enhance data exploration, automate insight generation, and support conversational analytics use casesIntegrate Genie capabilities with Delta Lake and curated data models to support near real-time insights and decision-makingPartner with data scientists and analytics teams to enhance AI-driven use cases, including producer performance insights, marketing attribution, and member engagement analysisRequired QualificationsBachelor’s or Master’s degree in Computer Science, Engineering, or related field5–8+ years of experience in data engineeringStrong programming in Python (PySpark) and advanced SQLHands-on experience with:Databricks (core requirement)PostgreSQLDistributed data processing frameworks (Apache Spark)Experience with cloud platforms (Azure preferred; AWS acceptable)Proficiency in building and maintaining ETL/ELT pipelinesStrong understanding of data modeling and warehousing conceptsPreferred QualificationsExperience in healthcare or insurance industry (payer experience strongly preferred)Familiarity with healthcare standards (e.g., FHIR, HL7)Experience with:Delta Lake / Lakehouse architectureOrchestration tools (Airflow, Azure Data Factory)Streaming (Kafka, Event Hubs)Knowledge of DevOps and CI/CD pipelines (Azure DevOps, GitHub Actions)Experience supporting machine learning pipelinesKey Skills & CompetenciesDeep understanding of data pipelines at scaleStrong experience with Databricks ecosystem and Spark optimizationExpertise in PostgreSQL performance tuning and schema designStrong attention to data quality, governance, and complianceExcellent communication skills, especially with non-technical stakeholdersAbility to work in a highly regulated healthcare environmentTypical Technology StackData Platform: Databricks, Delta LakeDatabase: PostgreSQL, Snowflake (optional)Cloud: Azure, Google, AWSLanguages: Python, SQLOrchestration: Airflow, Azure Data FactoryVisualization: Power BIVersion Control: GitKPIs / Success MetricsReliability and performance of Databricks pipelinesData quality and compliance adherence (HIPAA standards)Time-to-delivery for new data productsQuery performance improvements in PostgreSQL and data warehouse systemsStakeholder adoption and satisfaction