Data Engineer
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
Will be responsible for designing, developing, integrating, and maintaining enterprise-level Big Data Systems with both batch and streaming datasets. Should have a very strong understanding of MPP databases, Shared Nothing, Shared, and other Modern Big Data tech stacks like Spark, Hadoop, AWS Redshift, Confluent Kafka, AWS Kinesis, and other Streaming technologies. This individual is expected to design Big Data systems based on Industry best practices and architectural guidance of Big Data systems, with an understanding of integration with other data sources and tools like Markit, Business Objects, Informatica, MS SQL, PL/SQL, etc. Enhance/Maintain/support existing applications. Collaborate with other developers in designing.Technical Skills:Partner with business leadership to identify problems, and opportunities for technology innovation with a focus on Big Data implementations.Help establish a clear, consistent technology vision through collaboration, influence, and enablement.Research, recommend, design, and develop Big Data systems with a sound understanding of the Big Data application architecture and Integration.Identify and assess the organizational impact of enterprise architecture and standards, including changes in skills, processes, and structures with an emphasis on the data warehouse.Should have developed ETL pipeline using Python & Spark or PySparkExtensive experience in SQL query tuning In educational qualification – the candidate must have someone with Computer Science degree/diplomaIn experience – Focus on 10+ years of total experience with 4+ years of experience in Databricks and PySparkCreating robust and extensible data pipelines for production systems Creating secure, performant, and well-modeled data storesMust be fluent in any one of the scripting languages such as Python/JavaExperience working in an onsite client technical consulting environment preferred.Source code version control management using tools like Git/GitHubExperiences working within Agile Frameworks, such as Scrum or Kanban