Senior Platform Data Engineer
Senior Platform Data Engineer We are seeking a Senior Platform Data Engineer. This is a production engineering role focused on building and operating a centralized, enterprise-grade data platform that institutionalizes proprietary knowledge and delivers data as a product — not a side effect. You will design, build, and operate models, pipelines, integrations, and infrastructure across the full medallion architecture (Bronze → Silver → Gold). This role requires true end-to-end ownership: design, build, test, deploy, monitor, and continuously improve.
What You’ll Do:
- Build and maintain ingestion pipelines from raw Bronze landing through curated Gold domain models
- Develop transformation models using dbt with testing, Slim CI, and exposure tracking
- Author and manage production-grade Airflow DAGs with SLA monitoring and backfill strategies
- Design and implement streaming and event-driven pipelines (Kafka, Glue Streaming, Delta Live Tables)
- Collaborate with data scientists, platform engineers, and product teams to ship reliable data products
- Implement data quality frameworks and enforce governance standards
- Containerize workloads and deploy to Kubernetes (EKS)
- Contribute to infrastructure as code (Terraform provisioning for S3, IAM, Glue, etc.)Document lineage, metadata, ownership, and sensitivity classification
- This is not a “pipeline-only” role — it is platform ownership at scale.
- Core Technical SkillsPythonProduction-level standardsSQLAdvanced analytical queries, window functions, optimization
- Experience with Redshift, Snowflake, or Databricks SQLdbtGold-layer modeling and testing
- Slim CI, exposures, meta configurations
- OrchestrationApache Airflow (DAG authoring, task dependencies, SLAs, backfills)
- Prefect or Dagster is a plus
- Streaming & Event Pipelines
- Kafka topic consumption
- Bronze landing jobs
- Cloud Data Platforms (hands-on with at least one)
- Databricks (Delta Lake, Unity Catalog, Auto Loader, DLT)AWS (S3, Glue, Lake Formation, Redshift)
- Data Quality
- Great Expectations, dbt tests, or equivalent
- Docker & Kubernetes
- Containerized jobs
- Kubernetes Jobs and CronJobsEKS deployments
- Infrastructure as Code
- Terraform familiarity
- Data Modeling
- Dimensional modeling
- Slowly Changing Dimensions
- Medallion architecture design
- Data Platform-Specific Experience
- SaaS ingestion via Fivetran, Airbyte, or custom connectors (REST, SFTP, JDBC)
- Change Data Capture (Debezium or connector-based CDC modes)
- Handling semi-structured and unstructured data (PDF, Excel, JSON)
- Designing Bronze schemas for fidelity and downstream flexibility
- Vector embedding pipelines (chunking strategies, embeddings, pgvector/OpenSearch)
- Data lineage tooling (OpenLineage, DataHub, Unity Catalog)
- Governance standards: PII handling, sensitivity classification, column masking, partition-level access
- Ways of Working
- Agile sprint participation (planning, standups, retros)
- Treat pipelines as production software (tests, runbooks, SLAs, alerts)
- Migration-safe schema changes (expand/contract pattern)
- Proactive data quality ownership
- Rigorous peer code reviews
- Close collaboration with data science and platform engineering teams
Nice to Have:
- Migration of legacy data assets (Excel, Access, shared drives)ML feature engineering and point-in-time correct pipelines
- Open-source data tooling contributions
- Qualifications5–8+ years of experience developing data products and platforms
- Strong background in Agile software development
- Demonstrated success in prior engineering roles
- Strong analytical and problem-solving capabilities
What We’re Looking For:
- Clear communicator who can document complex data flows
- Results-driven and impact-focused
- Proactive and self-directed
- Comfortable interfacing across technical and business stakeholders
- Entrepreneurial mindset with a willingness to innovate
If you’re a production-minded data engineer who thrives on building scalable platforms and owning pipelines end-to-end, we’d love to connect.