Databricks Data Engineer(Azure + Python)
Role : Databricks Data Engineer(Azure + Python)Location : RemoteContractJob DescriptionMust Have Technical/Functional Skills:Exp. in Apache SPARK development, Kubenetes, CI-CD Pipeline, Jenkins, Dockers, Kubernetes, PL SQL, PythonWriting SQL queries and proceduresWriting Python code to automate and develop small functionalitiesCreating CI/CD pipelinesWriting Jenkins jobsManaging applications in Kubernetes environments, including deployment, configuration, and triagingHands-on experience with Apache SparkRoles & Responsibilities:The candidate will provide technical leadership for the team(s) they are associated with and participate in key technical decisions. They will engage with customers on escalations and ensure that there is continuous improvement in all areas. Participate in technical discussions within the team and with other groups within Business Units associated with specified projectsYou design, develop, and maintain our real time data processing, data Lakehouse infrastructure.You have experience with Python writing data pipelines and data processing layers.You develop and maintain Ansible playbooks for infrastructure configuration and managementYou develop and maintain Kubernetes manifests, Helm charts, and other deployment artifactsYou have hands-on experience on Docker and containerization and how to manage/prune the images in private registries.You have hands-on experience on access control in K8S clusterYou have hands-on experience on SPARK and maintaining SPARK CLUSTERYou monitor and troubleshoot issues related to Kubernetes clusters and containerized applicationsYou drive initiatives to containerize standalone apps to be containerized i n Kubernetes.You develop and maintain infrastructure as code (IaC) and collaborate with other teams to ensure consistent infrastructure management across the organizationYou use observability tools to do "capacity management" of our services and infrastructure resources.You are for guiding the development and testing activities of other engineers that involve several inter-dependenciesExperience in AWS ECS and EKS is added advantageExperience in Dremio is added advantageExperience in Dynatrace or any tracing, infrastructure, or real time monitoring tool is added advantage