Data Engineer-(DE)

divh2Data Engineer-(DE)/h2pLocation: Scottsdale AZ (day 1 onsite)/ppDuration: CTH and Fulltime/ppJob Description:/ppMust have skill set: Spark, S3, Glue, AWS Redshift and infrastructure, AWS Data Lake Formation and Glue components, data security, SQL, and Python/pp6-8 years of IT experience focusing on enterprise data architecture and management./pp Experience in Conceptual/Logical/Physical Data Modelling expertise in Relational and Dimensional Data Modelling/pp Experience with Databricks on Prem, Structured Streaming, Delta Lake concepts, and Delta Live Tables required/pp Experience with Spark scala/pp Data Lake concepts such as time travel and schema evolution and optimization/pp Structured Streaming and Delta Live Tables with Databricks a bonus/pp Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support/pp Advanced level understanding of streaming data pipelines and how they differ from batch systems/pp Formalize concepts of how to handle late data, defining windows, and data freshness/pp Advanced understanding of ETL and ELT and ETL/ELT tools such as Data Migration Service etc/pp Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermark, etc./pp Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus/pp Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness/pp Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design performance optimization)/pp Indexing and partitioning strategy experience/pp Debug, troubleshoot, design and implement solutions to complex technical issues/pp Experience with large-scale, high-performance enterprise big data application deployment and solution/pp Architecture experience in AWS environment a bonus/pp Familiarity working with Lambda specifically with how to push and pull data, how to use AWS tools to view data for processing massive data at scale a bonus/pp Experience with Gitlabs and CloudWatch and ability to write and maintain gitlabs for supporting CI/CD pipelines/pp Experience working with AWS Lambdas for configuration and optimization and experience with S3/pp Familiarity with Schema Registry, message formats such as Avro, ORC, etc./p/div