Data Engineer

Job Title: Data EngineerLocation: [Your Location or "Remote"]About the Role:We are seeking a highly skilled Data Engineer with strong expertise in Databricks and PySpark. The ideal candidate will also have hands-on experience with Airflow for orchestration and Kafka for streaming data pipelines. You will play a key role in building, optimizing, and maintaining our data infrastructure to support data-driven decision-making across the organization.Responsibilities:Design, build, and maintain scalable data pipelines using Databricks and PySpark.Develop and manage ETL workflows and data integration processes.Implement data orchestration and workflow automation using Apache Airflow.Build real-time and batch data streaming solutions leveraging Apache Kafka.Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver high-quality solutions.Optimize data workflows for performance, scalability, and reliability.Ensure data quality, integrity, and security across all pipelines.Monitor and troubleshoot data pipelines, ensuring minimal downtime and data loss.Requirements:Bachelor's or Master's degree in Computer Science, Engineering, or related field.3+ years of experience in data engineering or a similar role.Strong hands-on experience with Databricks and PySpark.Proven experience in building and maintaining data pipelines with Apache Airflow.Solid understanding of Apache Kafka, including producing and consuming data streams.Proficiency in SQL and working with large-scale data processing.Experience with cloud platforms (AWS, Azure, or GCP) is a plus.Familiarity with data modeling, warehousing, and governance best practices.Excellent problem-solving and communication skills.Nice to Have:Experience with Delta Lake, Spark Streaming, or other big data technologies.Knowledge of CI/CD tools and DevOps practices for data workflows.Exposure to monitoring tools like Datadog, Grafana, or similar.