Data Engineer with Databricks

Data Engineer With DatabricksWe are seeking an experienced Data Engineer with 4–6 years of experience and strong hands-on expertise in Databricks and Apache Spark. The ideal candidate will design, develop, and optimize scalable data pipelines to support analytics and data-driven business candidate will work closely with data scientists, analysts, business stakeholders, and cross-functional teams to ensure data is reliable, secure, high-quality, and readily available.Key ResponsibilitiesDesign, develop, and maintain scalable ETL/ELT data pipelines using Databricks and Apache Spark.Build and optimize batch and real-time/streaming data pipelines for large volumes of structured and unstructured data engineering solutions using Databricks, Spark, and related data technologies.Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.Monitor, troubleshoot, and optimize data processing workflows to ensure reliability and performance.Optimize Databricks clusters for performance, scalability, and cost efficiency.Implement data quality, governance, security, and compliance standards across data pipelines.Perform data validation and troubleshooting to identify and resolve pipeline issues.Maintain technical documentation related to data architecture, data pipelines, workflows, and processes.Support continuous improvement of data engineering processes and platform capabilities.Explore and integrate AI/Agentic AI capabilities into data engineering workflows where applicable.Required Skills4–6 years of experience in Data Engineering.Strong hands-on experience with experience with Apache Spark.Experience developing ETL/ELT pipelines.Experience with both batch and streaming data processing.Experience working with large volumes of structured and unstructured understanding of data engineering and data processing concepts.Experience with data pipeline monitoring, troubleshooting, and performance of data governance, security, and compliance.Experience optimizing Databricks clusters for cost and performance.Strong problem-solving and analytical skills.Good communication and collaboration skills.Preferred SkillsExperience with AI Agents / Agentic AI.Experience integrating AI capabilities into data platforms or data workflows.Knowledge of cloud data platforms such as AWS, Azure, or GCP.Experience with streaming technologies such as Kafka.Knowledge of data lake/lakehouse architecture.