Sr. Data Engineer

Are you an experienced Data Engineer looking for the chance to make a real impact? We're hiring! We help organizations create robust technical architecture, develop complex systems and build out their infrastructures. Check out our opening here. Role and Responsibilities: Design and Develop scalable industry standard ETL pipelines using big data technologies. Loading from disparate data sets by leveraging various big data technologies EMR, Hive, Spark. Presto etc. Ingest, Extract, Move, Transform, Cleanse, and Load massive structured and unstructured data in Hadoop environment both in batch and real-time. Build and maintain robust data pipelines for multiple use cases, and ensure high quality data i.e., accurate, complete and timely. High-speed querying using in-memory technologies such as Spark. Design and implement data modeling Translate complex functional and technical requirements into detailed design Code and query Optimization. Skills & Qualifications 3-5 years of experience in Hadoop (Hive and Impala) & Spark (Spark/Scala) Proficient in a scripting language and object-oriented language – Python/Scala preferred Proficient with a public cloud (AWS, Azure, GCP) Expert in SQL development Proficient in using Big Data stack (Hadoop, Hive, Spark, Kafka, Kerberos, OOZIE, impala, etc.) Adept in multi-threading and concurrency concepts. Job Type: Full-time Pay: $100,000.00 - $151,000.00 per year Schedule: Monday to Friday Ability to commute/relocate: San Francisco Bay Area, CA: Reliably commute or planning to relocate before starting work (Required) Experience: Scripting: 2 years (Required) Hadoop: 2 years (Preferred) Python: 2 years (Required) Work Location: In person