Azure Databricks Data Engineer

Azure Databricks Data EngineerAbout the RoleWe're building a bench of strong Azure Databricks Data Engineers for an upcoming engagement with a data and AI consulting firm. The role centers on designing and building data pipelines, transforming large datasets, and enabling analytics and machine learning workflows within the Databricks ecosystem on Azure.This isn't a backfill. The client is establishing new capabilities and wants to bring on sharp engineers who can own their work from day one.What You'll DoDesign, build, and optimize data pipelines using Azure Databricks and Apache SparkDevelop ETL/ELT workflows that ingest, transform, and serve data across the organizationWork within the Databricks Lakehouse architecture, leveraging Delta Lake for reliable data storageWrite clean, performant PySpark and/or Scala code for large-scale data processingIntegrate Databricks workflows with Azure Data Factory, Azure Data Lake Storage, and other Azure data servicesImplement data quality checks, monitoring, and alerting to ensure pipeline reliabilityPartner with data scientists and analysts to prepare and deliver datasets that power downstream models and dashboardsQualifications4+ years of data engineering experience, with at least 2 years working in Azure DatabricksStrong proficiency in PySpark or Spark with ScalaHands-on experience with Delta Lake and the Lakehouse architecture patternExperience building production-grade ETL/ELT pipelines at scaleSolid understanding of Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, and Azure SQLAbility to optimize Spark jobs for performance, cost, and reliabilityPreferred SkillsExperience with Databricks Unity Catalog for data governanceFamiliarity with MLflow for experiment tracking and model managementKnowledge of streaming data processing with Structured Streaming or KafkaPrior experience supporting data science or ML teams with feature engineering and data preparationTech StackAzure Databricks, Apache Spark, PySpark, Delta Lake, Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), Azure SQL, Databricks Notebooks, SQL