Data Engineer
Data EngineerKey ResponsibilitiesData Pipeline & Architecture▸ Design, develop, and maintain large-scale batch and real-time data pipelines processing billions of transactions daily▸ Architect data lake and data warehouse solutions on cloud platforms (AWS, GCP, or Azure) to support enterprise analytics▸ Build and optimize ETL/ELT workflows using Apache Spark, Airflow, dbt, or similar frameworks▸ Ensure high availability, fault tolerance, and disaster recovery for all data systemsData Modeling & Quality▸ Develop dimensional and relational data models aligned with AMEX business domains (card services, fraud, loyalty, risk)▸ Implement data quality frameworks including validation, reconciliation, and anomaly detection▸ Enforce data lineage, cataloging, and metadata management standards using tools like Apache Atlas or AWS Glue▸ Partner with data governance teams to ensure PCI-DSS, GDPR, and SOX complianceCollaboration & Leadership▸ Collaborate with Data Scientists, ML Engineers, and Business Analysts to deliver analytical datasets and feature stores▸ Mentor junior data engineers and conduct code reviews following engineering best practices▸ Work closely with Product Managers and business stakeholders to translate requirements into technical specifications▸ Participate in Agile ceremonies; contribute to sprint planning, retrospectives, and technical roadmapsPerformance & Optimization▸ Tune query performance across distributed systems (Spark, Hive, Presto, BigQuery, Redshift)▸ Monitor pipeline SLAs and proactively resolve bottlenecks, data skew, and resource contention issues▸ Implement cost optimization strategies for cloud-based data workloadsRequired Qualifications▸ 5+ years of hands-on experience in data engineering or a related technical discipline▸ Proficiency in Python and/or Scala for data processing and pipeline development▸ Deep expertise in Apache Spark, Hadoop, or other distributed data processing frameworks▸ Experience with cloud data platforms: AWS (S3, Glue, EMR, Redshift), GCP (BigQuery, Dataflow), or Azure (ADLS, ADF, Synapse)▸ Strong SQL skills with experience in performance tuning across large datasets▸ Proficiency with workflow orchestration tools: Apache Airflow, Prefect, or Dagster▸ Experience with streaming technologies: Apache Kafka, Kinesis, or Pub/Sub▸ Solid understanding of data modeling concepts: star schema, snowflake schema, Data Vault▸ Familiarity with version control (Git), CI/CD pipelines, and DataOps practices▸ Bachelor's degree in Computer Science, Engineering, Mathematics, or a related field