Data Engineer / ETL Developer
The Data Engineer / ETL Developer designs, develops, tests, deploys, operates, and improves AWS-based data pipelines supporting modernization of Client's 30+ priority datasets. The position performs source integration, ETL/ELT, data modeling, validation, quality engineering, metadata/lineage management, orchestration, automation, testing, deployment, and performance optimization.Job RequirementAnalyze authoritative sources, ownership, schemas, update frequency, expected volume, dependencies, security considerations, and downstream uses.Develop Pipeline Design Specifications and document source-to-target mappings, ingestion, transformation, validation, orchestration, exception handling, recovery, and monitoring.Design and implement pipelines using STB-approved AWS data services, particularly Glue, Lambda, Step Functions, S3, Redshift, and RDS PostgreSQL.Support STB's Bronze / Silver / Gold Medallion/Lakehouse direction and standardized ETL/ELT patterns.Develop ingestion, cleansing, normalization, transformation, standardization, enrichment, and publication workflows using Python, SQL, and applicable engineering frameworks.Develop and maintain schemas, data models, ERDs, data dictionaries, relationships, and source-to-target structures.Implement automated schema enforcement, validation, quality rules, exception handling, and quarantine/flagging of nonconforming data.Capture metadata and lineage and integrate applicable catalog/discovery requirements into pipeline implementation.Implement logging, monitoring, error handling, automated retries, notifications, recoverability, and idempotent reprocessing.Perform required development, staging, and production testing and validate pipeline logic, data quality, performance, recovery, and scalability before release.Support CI/CD, version control, configuration management, controlled releases, and approved infrastructure-as-code practices.Maintain pipeline specifications, ERDs, data dictionaries, lineage, deployment packages, validation rules, quality procedures, and operational documentation.