Cloud Data Engineer (AWS / Databricks)
Purple DriveCloud Data Engineer (AWS / Databricks) Contractual Plano, TX
Overview
Key Responsibilities:
- Design, build, and optimize scalable data pipelines and frameworks on AWS leveraging EMR, EKS, S3, and related services.
- Develop, deploy, and manage Databricks (DBX) workflows for big data processing, analytics, and machine learning.
- Implement infrastructure as code (IaC) solutions using Terraform for automated provisioning and environment consistency.
- Write efficient, reusable, and optimized code in Scala, Python, and Java for data transformation, streaming, and with architects, data scientists, and business stakeholders to translate requirements into technical solutions.
- Ensure performance tuning, scalability, and cost optimization across cloud-based data best practices in security, CI/CD, testing, and DevOps integration for cloud-native Skills & Experience:7–10 years of hands-on experience in data engineering and cloud frameworks.
- Proven expertise with AWS big data services (EMR, EKS, S3, Lambda, Glue).Strong development skills in Scala, Python, and Java.
- Deep knowledge of Databricks (DBX) for batch and streaming workloads.
- Solid experience with Terraform for infrastructure with CI/CD, containerization (Docker, Kubernetes), and Agile methodologies.
- Nice-to-Have Skills:
- Exposure to machine learning workflows on of Airflow or other orchestration tools.
- Experience with data lakehouse architectures.