Senior Data Engineer
Title: Senior Data EngineerSenior Data Engineer: strongest on Glue + Spark + Iceberg + streaming/batch + RedshiftDuration: 14 WeeksLocation: Remote (US/Canada)Note consultants should be able to travel with in US to Greenville site once every month.Must-have: Strong Glue, ETL, and Spark skills; experience with Iceberg; ability to work with both batch and real-time or streaming ingestion; familiarity with Kinesis and/or MSK; and understanding of Redshift and data warehouse patterns.Nice-to-have: Working knowledge of DynamoDB for metadata-related use cases, broader awareness of AWS streaming options, and prototype-building capability to support architecture validation.Position OverviewWe are seeking a Senior Data Engineer to design and build the data pipelines, data products, and integration flows. This role involves hands-on pipeline architecture, data quality validation, and building the foundational data products that enable the Industrial Data Mesh.Key ResponsibilitiesTechnical LeadershipDesign and architect batch and streaming data pipelines for industrial dataDefine data product schemas, contracts, and quality validation rulesImplement data integration patterns (CDC, event-driven, pub/sub) across OT and IT systemsDesign schema evolution strategies using Avro, Parquet, and Apache IcebergOptimize pipeline performance and cost efficiencyCustomer EngagementCollaborate with data teams to understand existing data flowsSupport data domain workshops with technical pipeline feasibility inputPresent pipeline design recommendations to customer engineering teamsSolution DevelopmentBuild production-ready data pipelines on AWS infrastructureImplement data quality validation and enrichment at ingestionDevelop automated testing for data productsCreate infrastructure-as-code for pipeline deploymentQualificationsExperience5-7 years in data engineering or ETL/ELT developmentExperience with large-scale streaming and batch data processingExperience In Manufacturing Or Industrial Data Environments PreferredTechnical Skills (AWS Services, would consider competitive alternatives)AWS Glue (ETL, Spark, Iceberg)Amazon Kinesis Data Streams / FirehoseAmazon MSK (Kafka)Amazon Athena, Amazon RedshiftAWS Step Functions, AWS LambdaAmazon S3 (partitioning, lifecycle management)Amazon DynamoDB (metadata/state)AWS CDK / CloudFormationProgramming: Python, PySpark, SQLSoft SkillsStrong problem-solving and analytical skillsAbility to work collaboratively with architects and customer teamsExperience in agile environmentsAWS Certifications (Nice to have)AWS Certified Data Analytics - SpecialtyAWS Certified Solutions Architect - Associate