Sr OpenShift API Developer - Big Data Skills
Job Title: Sr OpenShift API Developer - Big Data SkillsLocation: Charlotte, NC – 3 days/week onsitePosition type: Contract-W2/1099Job Description:Platform Management: Deploy, configure, and maintain containerized Spark Applications on OpenShift or Google CloudData Pipeline Development: Design and implement large-scale data processing workflows using Apache Spark and Apache AirflowOptimization: Tune Spark jobs for performance, leveraging OpenShift's resource management capabilities(e.g., Kubernetes orchestration, auto-scaling).Integration: Integrate spark with other data sources (e.g., Kafka, s3, cloud storage) and sinks (e.g., databases, data lakes)CI/CD Implementation: Build and maintain CI/CD pipelines for deploying Spark application in OpenShift using tools like GitHub actions, Sonar, Harness.Monitoring & Troubleshooting: Monitor cluster health, Spark job performance, and resource utilization using OpenShift tools(e.g., Prometheus, Grafana) and resolve issues proactively Security: Ensure compliance with security standards, Implementing role-based access control(RBAC) and encryption for data in transit and at rest.Collaboration: Work with cross-functional teams to define requirements, Architect solutions, and support production deployments.Technical Skills:Proficiency in Spark frameworks(Python/PySpark, Scala, or Java)Hands on experience with micro services architecture solutions (eg, API design, development, and testing)Proficiency in creating and maintaining conda environments and dependenciesWorking knowledge with Docker and Kubernetes solutions (e.g., pods, deployments, services and images)Knowledge of distributed systems, cloud platforms (AWS, GP, Azure), and data storage solutions (e.g., S3, HDFS)Programing: Strong coding skills in Python, Scala, or Java; experience with shell scripting is a plus.Tools: Experience with Git Actions, Helm, Harness, and CI/CD tools.Problem-Solving: Ability to debug complex issues across distributed systems and optimize resource usage.