Azure Data Engineer

Azure Data EngineerHybrid onsite in Hopkins MNContract through 5/30/2025Project Summary:This project involves building AC360 data pipelines in Azure using Spark technologies. As part of this initiative, the development of Zafin data pipelines in Azure is also included.Key Responsibilities:1. Identify Requirements and Develop Solutions• Identify requirements for RMS replacement and develop solutions in Azure.• Re-engineer solutions to retrofit the rules with AC360 in Azure.2. Data Pipeline Management• Register data pipelines in Azure, onboard datasets, and enable data processing.• Build Airflow DAGs and ensure scheduling is integrated.3. DevOps and Production Management• Apply a DevOps mindset, take ownership of production success, and optimize operations via automation, active alerting, and self-healing.• Lead the resolution of production issues to ensure high availability and performance.4. Code Development and Review• Develop high-quality code and define best engineering practices.• Perform peer code reviews to ensure successful deliverables with engineering excellence.5. Documentation• Document engineering artifacts such as technical design documents, flowcharts, system context diagrams, and code comments.6. Collaboration• Collaborate cross-functionally with product owners, data scientists, business users, project managers, and other engineers to achieve elegant solutions.• Work with other engineers on the team to elevate technology and consistently apply best practices.7. End-to-End Development and Testing• Execute true end-to-end development and testing of software products in an agile lifecycle.• Partner with key technical and business stakeholders to ensure the quality of the final product through manual and automated tests.• Support the new software as it is moved to production.Required Skills/Technologies:• Mainframe COBOL: Good knowledge to understand the rules in Strata rules engine.• Spark-Scala or Spark-SQL, Spark-Py: Minimum 5 years of experience.• Airflow, Elastic, Azure: 3 years or higher.• Big Data Management: 10 years or higher.• Microservices-Based Applications: 5+ years.• Azure, Kubernetes, PostGres: Experience required.• Maven, Docker, Kubernetes, Kibana, ELK, Jenkins, Spring Boot: 4+ years.• CI/CD Processes (Maven, Git, Jenkins): 4+ years.• Cloud Technologies: 4+ years.• Server-Side Languages (Python, Java, GoLang): 3+ years.• Communication Skills: Excellent verbal and written communication skills, 10 years or higher.• Technical, Logical, Analytical, and Problem-Solving Skills: 10 years or higher.Preferred Skills/Attributes:• Experience with Spark, Scala, Python, Azure stack.Interview Process:• Rounds: 1• Format: Video• Technical Level: Real-time coding sessionRequired Skills : • Mainframe COBOL: Good knowledge to understand the rules in Strata rules engine. • Spark-Scala or Spark-SQL, Spark-Py: Minimum 5 years of experience. • Airflow, Elastic, Azure: 3 years or higher. • Big Data Management: 10 years or higher. • Microservices-Based Applications: 5+ years. • Azure, Kubernetes, PostGres: Experience required. • Maven, Docker, Kubernetes, Kibana, ELK, Jenkins, Spring Boot: 4+ years. • CI/CD Processes (Maven, Git, Jenkins): 4+ years. • Cloud Technologies: 4+ years. • Server-Side Languages (Python, Java, GoLang): 3+ years. • Communication Skills: Excellent verbal and written communication skills, 10 years or higher. • Technical, Logical, Analytical, and Problem-Solving Skills: 10 years or higher.Basic Qualification :Additional Skills :Background Check : YesDrug Screen : YesNotes :Selling points for candidate :Project Verification Info :MSA: Restricted Client Letter: Will Provide"Candidate must be your W2 Employee :YesExclusive to Apex :NoFace to face interview required :NoCandidate must be local :YesCandidate must be authorized to work without sponsorship ::NoInterview times set :YesType of project :Master Job Title :