AWS Data Engineer – Open Data Platform

AWS Data EngineerOur client is seeking an experienced AWS Data Engineer to design and build cloud-native data pipelines on an open data platform, architecting Apache Iceberg tables and optimizing large-scale data processing across AWS and Snowflake environments.Responsibilities & QualificationsDesign and develop end-to-end data ingestion, transformation, and processing pipelines using AWS Glue, Apache Spark, and Amazon S3Build and manage Apache Iceberg tables to enable open, interoperable access across multiple compute enginesConfigure and integrate Apache Polaris with Iceberg REST Catalog for AWS and Snowflake environmentsImplement table maintenance, schema evolution, and partitioning strategies to optimize performance and costSupport platform security, access controls, and AWS networking infrastructureConduct workload benchmarking and performance analysis comparing AWS Glue versus Snowflake for cost and efficiencyProvide operational monitoring, troubleshooting, and optimization of cloud-based data platform infrastructureRequirements8–10 years of experience in data engineering and cloud-based data pipeline developmentProven expertise with Amazon S3, AWS Glue, Apache Spark, and large-scale data processingStrong hands-on experience with Apache Iceberg architecture, including table design and maintenanceDemonstrated proficiency with Apache Iceberg schema evolution and partitioning optimizationExperience configuring and deploying Apache Polaris and Iceberg REST CatalogSolid understanding of AWS IAM, networking, and security best practices in cloud environmentsExperience with performance tuning, cost optimization, and cross-platform data tool evaluation