Lead Data Engineer

Hello Everyone,No GC,Role: Senior Lead Data EngineerLocation: San Jose, CA 95125 Hybrid(Locals)Key: Hadoop, Spark SQL strong• Key Responsibilities:• Architect, construct , test and maintain ELT pipelines using Hadoop and Spark SQL• - Manage and optimize the infrastructure ensuring reliability and performance• -Design and implement data models and schemas for enterprise data warehouse and data marts• -Write and maintain scripts in Python for various applications• -Develop, optimize, troubleshoot complex SQL queries and database structures for relational and non-relational databases• - Work closely with analysts and product managers to understand data requirements and deliver high-quality data• -Provide technical guidance and mentorship to other team members fostering best practices in data engineering• -Monitor system performance, identify bottlenecks, implement solutions to ensure high data availabilityRequired Skills:-High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark, Scala for large scale data processing-Good knowledge in Python for data manipulation and scripting-Mastery of advanced SQL and deep experience working with relational databases and data warehouses-Hands-on experience with workflow management tools like Apache Airflow-Familiarity with Unix shell scripting for automation and system management