JOBSEARCHER

Data Platform Engineer

Our healthcare client is looking for a Data Engineer with 7+ years experience to own the multi-stage data pipeline layer that ingests from external sources (BigQuery direct queries, vendor file feeds, APIs) into a governed lake/warehouse. Deliver scaled event-driven integrations; managing ETL for downstream integrations, an event-bus for user engagement services, and API / query access for billions of data points. What you'll do in this role: Ship a production-ready, like-for-like ETL replacement with zero data loss and validated parity against legacy outputs. Establish lake → warehouse pipelines with schemas, contracts, and idempotent restatement for corrections. Implement rules evaluation and event context enrichment to support downstream filtering and event-response integrations. Stand up observability for pipelines and jobs (SLOs, error/retry rates, latency dashboards, lineage) and on-call runbooks. Provision reproducible infrastructure with Terraform; define CI/CD and cost/performance guardrails across environments. Develop and support agentic workflows (data access, context management, & modern AI development practices in your own work) across the organization.
Key Responsibilities: Design and implement ingestion/connectors (BigQuery direct, CSV→JSON, REST) and normalization into standardized data models. Build event-driven jobs and services (Python/Node) that enrich, dedupe, and apply rules; ensure idempotency and safe replays. Define data contracts, schema evolution, backfill strategy, and cutover plans; partner with stakeholders on acceptance criteria. Operate AWS workloads (EC2, Lambda, AppRunner, RDS, Redshift) with Terraform; secure secrets, roles, and least-privilege access. Optimize SQL for MPP systems (Redshift, Snowflake, or similar); profile queries, partition/cluster, and tune materializations. Implement observability (logs, metrics, tracing, lineage) and incident response; drive postmortems and remediation. Maintain concise documentation of architecture, workflows, standards, and governance. Qualifications7–10+ years backend/data engineering with production ownership of large event-driven data systems. Proficient in Python, Node, and similar toolsets for ETL job implementation; strong testing and reliability toolset. Deep AWS experience and Terraform-based IaC; CI/CD for data and application deployments. Expert SQL and performance tuning on Redshift, Snowflake, or equivalent. Experience delivering idempotent pipelines, restaging/reconciliation, and parity validation against legacy systems. Experience in highly-governed environments (HIPAA, GLBA, PCI, etc.).Solid security and compliance practices, supporting internal and external audits (SOC 2 ISO 27001, etc.) and remediation.
Preferred: Orchestration experience (Airflow or similar), streaming (Kinesis/Kafka/SQS), and dbt or warehouse-centric modeling. Data quality frameworks, lineage/metadata tooling, and SLA/SLO design. Exposure to real-time enrichment and rules engines; familiarity with warehouse-native features (tasks/streams).Things you should be able to do: Diagram a multi-stage lake → warehouse pipeline with cutover, backfill, restaging for corrections, and SLOs. Discuss agentic workflow support from a data/platform perspective. Demonstrate experience (positive and lessons-learned moments) for at-scale data/platform implementations.