Python Developer

Key Responsibilities Develop, optimize, and maintain ETL pipelines using Apache Spark and Python. Implement batch and streaming data pipelines to process structured and unstructured data from various sources. Collaborate with data scientists and analysts to create robust, scalable, and reusable data processing workflows. Write clean, scalable, and maintainable code while following best practices in software development. Perform code reviews, testing, and continuous integration to ensure high-quality data solutions. Optimize Spark applications for performance and scalability, focusing on reducing execution time and improving resource utilization. Work with large datasets and distributed systems to ensure high availability and fault tolerance. Design and implement data storage solutions such as HDFS, Hive, and other big data technologies. Troubleshoot and resolve issues related to Spark jobs and data pipelines in production environments. Participate in architectural design decisions and provide input on data engineering best practices.Skills And Qualifications Proficient in Python with strong knowledge of Python libraries (e.g., Pandas, NumPy, PySpark). Strong experience with Apache Spark (2+ years). Experience in building, maintaining, and optimizing data pipelines using Spark. Familiarity with big data technologies such as Hadoop, Hive, HDFS, Kafka, and HBase. Experience with SQL and working with large relational and non-relational databases. Understanding of distributed computing principles and optimization techniques. Knowledge of data processing tools and frameworks such as Airflow or Luigi. Experience with cloud platforms such as AWS, Azure, or Google Cloud (e.g., EMR, Databricks, etc.). Solid understanding of CI/CD pipelines and automated testing. Familiarity with version control systems like Git. Strong problem-solving skills and ability to work in a fast-paced, collaborative environment.