Data Engineer
Overview:
As a Data Engineer, you will design, build, and optimize scalable data pipelines and cloud-based platforms to empower data-driven decision-making. You will work within cross-functional teams to translate data into actionable insights and robust data models. You'll tackle batch and real-time ingestion, ensure data quality, and implement CI/CD for data workflows. This role offers the chance to shape enterprise data architecture on cloud platforms and collaborate with data scientists and engineers. You will be a key contributor to scalable analytics and governance across the data stack.
Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETL/ELT workflows
- Build and optimize data processing with Python, SQL, and Apache Spark (PySpark)
- Develop batch and real-time data ingestion pipelines
- Handle structured and unstructured data from multiple sources
- Design and implement cloud-based data solutions (Azure, AWS, GCP)
- Develop and maintain data models, data lakes, and data warehouses
- Integrate data from APIs, databases, and streaming platforms
- Optimize pipeline performance and ensure data quality
- Collaborate with Data Scientists, Business Analysts, and Software Engineers
- Implement CI/CD pipelines and DevOps practices for data engineering
- Troubleshoot production issues and provide ongoing support
- Ensure compliance with data governance, security, and privacy standards
Key requirements:
- 5+ years of Data Engineer experience
- Strong Python programming
- Excellent SQL skills
- Hands-on Apache Spark / PySparkETL/ELT development experience
- Knowledge of Data Warehousing concepts
- Hadoop ecosystem experience (Hive, HDFS)
- Databricks experience
- Apache Kafka experience
- Airflow or similar workflow orchestration tool
- Linux/Unix environment experience
- Git and CI/CD familiarity
- Cloud experience with Azure, AWS or GCPcollaboration with cross-functional teamsproblem-solving mindseteffective communication
- PythonSQLApache Spark / PySpark