Data Engineer-SQL, Python, ETL- Canada
Data Engineer-SQL, Python, ETL
Location: Remote/CANADA
Duration: 6+ Months
Job Description:
We are looking for Senior Data Engineering Candidates based out of the US/Canada. Expertise in SQL, Python and ETL flows is a must have. Medium expertise/working knowledge in Scala is also needed, as our data pipelines are written in both Scala and Hive/Spark SQL, with new pipelines being written in Scala and legacy ones on SQL (which require ongoing maintenance).
Minimum Requirements:
- 5 years experience building scalable Spark data pipelines (preferably using Scala)3-5 years experience in high level programming languages such as Java, Scala, or Python
- Proficiency in Spark/MapReduce development and expertise with data processing (ETL) technologies to build and deploy production-quality ETL pipelines
- Good understanding of distributed storage and compute (S3, Hive, Spark)
- Experience using ETL framework (ex: Airflow, Flume, Oozie etc.) to build and deploy production-quality ETL pipelines
- Demonstrated ability to analyze large data sets to identify gaps and inconsistencies, provide data insights, and advance effective product solutions
- Working knowledge of relational databases and expertise in query authoring (SQL) on large datasets
- Experienced with big data technologies such as Hadoop, Spark, Hive, etc.
- Experience working with Git and Jira (or other source control and task management tools)