JOBSEARCHER

Apache Spark Platform Engineer

ARCHIVED

We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.

Apache Spark Platform Engineer -Only W2 REMOTE Long Term Must have skills:- Data Mesh Virtualization Metadata
Responsibilities: Apache Spark: Strong understanding and hands-on experience with Apache Spark, including Spark SQL, Spark Streaming. Databricks: Proficiency in using Azure Databricks, including setting up and managing clusters, notebooks, and jobs. Python and SQL: Strong programming skills in either Python and SQL as these are the primary languages used for writing Spark applications. Data Processing and Analysis: Experience in designing and implementing data processing and analysis pipelines using Spark and Databricks. Distributed Computing: Knowledge of distributed computing concepts and experience with distributed computing frameworks like Spark. Cloud Platform: Familiarity with Azure cloud platform and its services, including Azure Data Factory, Azure access, Azure Storage, Azure Data Lake, and Azure SQL Database. Data Engineering: Understanding of data engineering concepts and experience with ETL processes, data modeling, and data warehousing. Performance Optimization: Ability to optimize Spark applications for performance and scalability, including tuning Spark configurations and leveraging Spark optimizations. Data Security and Governance: Understanding of data security and governance practices, including data encryption, access controls, and compliance regulations. Monitoring and Troubleshooting: Proficiency in monitoring and troubleshooting Spark applications, identifying and resolving performance bottlenecks or issues. Collaboration and Communication: Excellent communication skills and the ability to collaborate effectively with cross-functional teams, including data scientists, data engineers, and business stakeholders. Desired skills: •Must know Spark, Databricks, Azure, and Linux OS experience. •Experience in Spark platform performance improvements and Capacity planning. •Knowledge of cloud platform-based database troubleshooting. •Experience with CI/CD pipelines, version control systems (e.g., Git), and automated deployment of Spark applications.
Job Type: Contract
Pay: $43.00 - $45.00 per hour
Schedule: 8 hour shift
Experience: Data Mesh: 5 years (Required) Virtualization: 5 years (Required) Metadata: 9 years (Required) Apache Spark:: 8 years (Required) Databricks:: 6 years (Required) Python and SQL:: 8 years (Required) Data Processing and Analysis:: 1 year (Required) Azure Data Factory, Azure access, Azure Storage: 6 years (Required) Azure Data Lake, and Azure SQL Database.: 6 years (Required)
Work Location: Remote