Lead Data Engineer

Lead Data EngineerLocation: Tarrytown NY 10591 (Hybrid role – 3 days onsite 2 days WFH)Duration: 6-12+ MonthsJob Description:Must have AWS, Apache Airflow, Pyspark, RedshiftCandidate should have 12+ years of experience in Data EngineeringDesigning, creating, testing and maintaining the complete data management & processing systems.Working closely with the stakeholders & solution architect.Ensuring architecture meets the business requirements.Building highly scalable, robust & fault-tolerant systems.Taking care of the complete ETL process.Knowledge of Hadoop ecosystem and different frameworks inside it – HDFS, YARN, MapReduce, Apache Pig, Hive, Flume, Sqoop, ZooKeeper, Oozie, Impala and KafkaMust have knowledge and working experience in Real-time processing Framework (Apache Spark), PySpark and in AWS RedshiftMust have experience on SQL-based technologies (e.g. MySQL/ Oracle DB) and NoSQL technologies (e.g. Cassandra and MongoDB)Should have Python/Scala/Java Programming skillsDiscovering data acquisitions opportunitiesFinding ways & methods to find value out of existing data.Improving data quality, reliability & efficiency of the individual components & the complete system.Creating a complete solution by integrating a variety of programming languages & tools together.Creating data models to reduce system complexities and hence increase efficiency & reduce cost.Introducing new data management tools & technologies into the existing system to make it more efficient.Setting & achieving individual as well as the team goal.