Lead Data Platform Engineer – Python, GCP & BigQuery
We are looking for a Lead Data Platform Engineer to help build and evolve a cloud-native Data Platform for a large-scale e-commerce marketplace. The role focuses on improving developer experience, standardizing data engineering practices, and automating platform operations to enable teams to build and manage data products more efficiently. You will work across Python, GCP, BigQuery, Airflow, Terraform, GitOps, and modern data platform technologies, while also contributing to Agentic AI and AI-assisted developer capabilities.
Key Responsibilities:
Design and develop tools that standardize, automate, and simplify Data Platform operations.
Build and maintain internal CLI tools for platform and Data Product management.
Develop reusable Data Product templates following modern data engineering practices.
Build platform visualizations and operational insights using Backstage.
Analyze platform usage, audit data, adoption, and engineering metrics.
Design and implement Agentic AI workflows and developer-assistance capabilities.
Develop reusable AI skills and automation components for working with data products and assets.
Manage the distribution, lifecycle, and monitoring of AI skills.
Maintain and enhance the shared Apache Airflow / Cloud Composer platform.
Develop reusable Airflow libraries, operators, and components.
Build and maintain Infrastructure as Code assets and Terraform modules.
Contribute to GitOps adoption using technologies such as Backstage, GitHub, ArgoCD, and Crossplane.
Collaborate with platform, data engineering, analytics, and ML teams to improve engineering efficiency and platform usability.
Required Experience:
Strong commercial experience with Python development.
Hands-on experience with Google Cloud Platform (GCP).Practical experience with BigQuery and cloud-based data platforms.
Experience with Apache Airflow, ideally Cloud Composer.
Experience building data platforms, developer tooling, or internal self-service solutions.
Strong understanding of Infrastructure as Code and Terraform.
Experience with GitOps and modern software delivery practices.
Good understanding of data engineering and Data Product lifecycle management.
Experience with PySpark and/or Apache Spark.
Familiarity with FastAPI, Pydantic, and modern Python tooling.
Knowledge of CI/CD and source control best practices.
Experience working in Agile environments.
Strong problem-solving skills, autonomy, and cross-functional communication.
Nice to Have:
Experience with Vertex AI or other GenAI platforms.
Experience with GitHub Copilot, Copilot extensions, or AI-assisted development.
Experience developing Agentic AI workflows or AI automation.
Backstage plugin development.
Experience with Dataproc, dbt, Kedro, ArgoCD, or Crossplane.
Experience building observability, telemetry, or platform analytics solutions.
Experience working with large-scale data environments supporting analytics and machine learning workloads.