Data Automation Engineer
Overview:
In this role you design and implement scalable data automation workflows that integrate AWS and Azure platforms. You will build and maintain ETL/ELT pipelines across diverse data sources, and develop batch and near real-time data processing with Spark and streaming technologies. You will prototype Generative AI solutions, enforce secure data practices, and implement CI/CD and automation using modern DevOps tools. You engage with cross-functional teams to deliver end-to-end data solutions that enable analytics and AI initiatives at scale.
Responsibilities:
- Design and implement scalable data automation workflows using AWS services with integration to Azure platforms
- Develop and maintain ETL/ELT processes to ingest, transform, and move data across DynamoDB, SQL Server, Azure SQL, and other enterprise data sources
- Build and support batch and near real-time data pipelines using Apache Spark and streaming technologies such as Kafka or Flume
- Develop scalable data processing solutions utilizing Amazon EMR, AWS Fargate, Kubernetes, and related cloud technologies
- Integrate REST APIs, external services, and Amazon Connect customer interaction data into analytics platforms
- Evaluate and prototype Generative AI solutions using Amazon Bedrock, Azure OpenAI, Hugging Face, LangChain, and related technologies
- Implement source control, CI/CD, and automation practices using GitHub, Azure DevOps, Jenkins, Docker, and Agile methodologies
- Ensure data solutions meet security and compliance requirements, including IAM, encryption, RBAC, VPC isolation, and firewall policies
Key requirements: AWS services Azure platformsETL/ELTDynamoDBSQL ServerAzure SQL