Apache Spark Platform Engineer

Apache Spark Platform Engineer -Only W2 REMOTE Long Term Must have skills:- Data Mesh Virtualization Metadata Responsibilities : Apache Spark: Strong understanding and hands-on experience with Apache Spark, including Spark SQL, Spark Streaming. Databricks: Proficiency in using Azure Databricks, including setting up and managing clusters, notebooks, and jobs. Python and SQL: Strong programming skills in either Python and SQL as these are the primary languages used for writing Spark applications. Data Processing and Analysis: Experience in designing and implementing data processing and analysis pipelines using Spark and Databricks. Distributed Computing: Knowledge of distributed computing concepts and experience with distributed computing frameworks like Spark. Cloud Platform: Familiarity with Azure cloud platform and its services, including Azure Data Factory, Azure access, Azure Storage, Azure Data Lake, and Azure SQL Database. Data Engineering: Understanding of data engineering concepts and experience with ETL processes, data modeling, and data warehousing. Performance Optimization: Ability to optimize Spark applications for performance and scalability, including tuning Spark configurations and leveraging Spark optimizations. Data Security and Governance: Understanding of data security and governance practices, including data encryption, access controls, and compliance regulations. Monitoring and Troubleshooting: Proficiency in monitoring and troubleshooting Spark applications, identifying and resolving performance bottlenecks or issues. Collaboration and Communication: Excellent communication skills and the ability to collaborate effectively with cross-functional teams, including data scientists, data engineers, and business stakeholders. Desired skills: •Must know Spark, Databricks, Azure, and Linux OS experience. •Experience in Spark platform performance improvements and Capacity planning. •Knowledge of cloud platform-based database troubleshooting. •Experience with CI/CD pipelines, version control systems (e.g., Git), and automated deployment of Spark applications. Job Type: Contract Pay: $43.00 - $45.00 per hour Schedule: 8 hour shift Experience: Data Mesh: 5 years (Required) Virtualization: 5 years (Required) Metadata: 9 years (Required) Apache Spark:: 8 years (Required) Databricks:: 6 years (Required) Python and SQL:: 8 years (Required) Data Processing and Analysis:: 1 year (Required) Azure Data Factory, Azure access, Azure Storage: 6 years (Required) Azure Data Lake, and Azure SQL Database.: 6 years (Required) Work Location: Remote