Senior Data Engineer

Role descriptionJOB DESCRIPTIONDATA ENGINEER - Sparx, Hadoop, OzoneCH, FlinkPrimary Skill*Secondary Skill*Sparx, Hadoop, OzoneCHETL, Scala, Java, or PythonRole Overview:We are seeking a highly skilled Big Data Engineer with strong experience in Apache Spark, Hadoop ecosystem, and Apache Ozone. The ideal candidate will design, develop, and optimize large-scale data processing systems, ensuring high performance, scalability, and reliability for enterprise-level applications.Key Responsibilities:• Design and implement distributed data processing solutions using Apache Spark, Hadoop, Flink• Develop and maintain Spark applications for data transformation, aggregation, and ETL processes using Scala, Java, or Python• Utilize Apache Ozone for storing large-scale datasets, ensuring efficient data access and management in a distributed environment• Manage and optimize HDFS and Apache Ozone, Kafka for scalable and fault-tolerant storage.• Develop ETL pipelines for batch and real-time data ingestion and transformation.• Implement and ensure data validation, data security, integrity, and compliance across big data platforms.• Monitor and troubleshoot performance issues in large-scale clusters.• Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions.• Automate workflows and improve operational efficiency using scripting and orchestration tools.Required Skills & Qualifications:• Strong expertise in Apache Spark (Core, SQL, Streaming).• Hands-on experience with Hadoop ecosystem (HDFS, YARN, MapReduce).• Proficiency in Apache Ozone for object storage and integration with Hadoop.• Solid programming skills in Java , Scala , or Python.• Experience with Hive, HBase , and Kafka is a plus.• Knowledge of cluster management and resource optimization.• Familiarity with Linux/Unix environments and shell scripting.• Understanding of data security, governance, and compliance standards.• Experience with cloud-based big data platforms• Exposure to containerization (Docker, Kubernetes) for big data workloads.• Knowledge of CI/CD pipelines for data engineering projects.Behavioral Skills:• Good Communication skills• 5 days Work from Office at Berkley Heights, NJ• Team Player• Ability to work in a changing environment• Strong problem solving and analytical skills• Ability to work independently or within a team• Manage day-to-day challenges and communicate developmental risks with the technical teamQualifications:• Bachelor’s degree in computer science, Software Engineering, or a related field.• Proficiency in business process modeling and documentation tools.• Product implementation experience is preferredOther detailsSalary range $63000 to $134500