Python Pyspark Developer

Overview In this role you will develop and optimize data-centered applications using Python and PySpark, ensuring scalable and reliable cloud-based workflows. You will collaborate with cross-functional teams to implement data pipelines and analytics solutions on AWS, while applying CI/CD practices. The position emphasizes building robust services with Django or Flask and maintaining testable, high-quality code. This is a hands-on, impact-driven role at the intersection of data engineering and software development. ResponsibilitiesDevelop and maintain Python and PySpark applications, including Spark SQL and related APIsDesign and implement scalable data pipelines and data processing workflowsDevelop and test using Python frameworks and testing tools (e.g., Pytest, PyUnit)Collaborate with cross-functional teams to deploy services on AWS (S3, Databricks, Data Lake Storage)Implement CI/CD pipelines and automate deployment processesBuild and maintain applications using Django or FlaskLeverage data pipeline tools such as Airflow, Kafka, and Jenkins to orchestrate workflows Key requirementsExpert proficiency in PythonExpert proficiency in PySpark, Spark SQL, and Spark APIsExperience with Python testing frameworks (Pytest, PyUnit)In-depth knowledge of Python frameworks such as Django or FlaskExperience with AWS cloud platforms and services (S3, Databricks, Data Lake Storage)Experience with CI/CD pipelines and toolsExperience with data pipeline tools (Airflow, Kafka, Jenkins)Ability to design scalable applicationsEffective communicationTeam collaborationStrong problem-solving and analytical abilitiesPythonPySparkSpark SQL