Lead Databricks Developer

Lead Databricks DeveloperResponsibilitiesLead the design and development of scalable data pipelines using Databricks and Azure/AWS/GCP cloud platforms.Architect data solutions integrating structured, semi-structured, and unstructured data sources.Build, optimize, and manage ETL/ELT workflows leveraging PySpark, Delta Lake, and Databricks SQL.Collaborate with data engineers, analysts, and business stakeholders to define requirements and deliver data-driven solutions.Implement best practices for performance tuning, cost optimization, and workload management in Databricks.Ensure data quality, security, and compliance across pipelines and storage layers.Mentor junior developers, conduct code reviews, and enforce development standards.Integrate Databricks with enterprise data lakes, warehouses, and reporting platforms.Drive automation, CI/CD, and monitoring for data pipelines.Stay updated on Databricks features, Spark ecosystem advancements, and industry best practices.Required SkillsStrong expertise in Databricks with hands-on experience in PySpark, Delta Lake, Databricks SQL.Proficiency in at least one programming language (Python, Scala, or Java).Cloud platform experience (Azure Data Services, AWS Glue, or GCP BigQuery).Solid understanding of ETL/ELT, data warehousing, and data lakehouse architecture.Experience with CI/CD pipelines, Git, DevOps practices.Knowledge of data governance, security, and compliance frameworks.Strong problem-solving, analytical, and leadership skills.Preferred QualificationsExperience with MLflow, Databricks Machine Learning, or AI/ML integration.Familiarity with streaming data (Kafka, Event Hubs, Kinesis).Background in data modeling and SQL performance tuning.Prior experience leading offshore/onshore development teams.