Data Engineer
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
Overview:
In this role, you will design and build scalable Spark-based ETL pipelines to transform large volumes of financial data. You will work closely with cross-functional teams to ensure timely data delivery and robust validation. You will implement data quality checks and validation frameworks that support clean, query-ready datasets for business use. The role combines pipeline development, data validation, and orchestration to enable data-driven decisions at scale.
Responsibilities:
Design and develop scalable Spark ETL pipelinesTransform large volumes of financial data within the project lifecycleDevelop and maintain data validation frameworksOrchestrate data workflows with Apache AirFlow, managing task dependencies and SLAsPerform data quality checks and monitoring to ensure data integrityProvide SQL-based data analysis and deliver clean datasets for business consumptionLeverage Unix/Linux shell scripting for data processing and automation
Key requirements:
Expertise in designing and implementing Spark-based ETL solutionsStrong command of Unix/Linux, shell scripting (Bash, Korn shell, or Perl) and file processing with awkProficiency in SQL querying and handling complex joinsExperience implementing Spark-based data validation frameworksExperience with Spark SQL for data analysis and validationExpertise in building complex data workflows with Apache AirFlowAbility to ensure timely data delivery through automated validation controlsStrong analytical skills and focus on data qualityAnalytical thinkingAttention to detailProblem solvingApache SparkSpark SQLApache AirFlow