JOBSEARCHER

Azure Data Engineer

We are seeking an experienced Azure Data Engineer with strong hands-on expertise in Azure Databricks to design, develop, and optimize scalable data engineering solutions on the Microsoft Azure platform. The ideal candidate will have strong experience building data pipelines, ETL/ELT frameworks, data lakehouse solutions, and large-scale data processing platforms using Azure Databricks, Apache Spark, Python/PySpark, SQL, and Azure data services.
Key Responsibilities: Design, develop, and maintain scalable data pipelines and data engineering solutions on Microsoft Azure. Build and optimize complex ETL/ELT pipelines using Azure Databricks. Develop large-scale data processing solutions using PySpark, Spark SQL, Python, and SQL.Design and implement Databricks Lakehouse architecture for structured and unstructured data. Work extensively with Delta Lake / Delta Tables for reliable and high-performance data processing. Develop and manage Databricks notebooks, workflows, jobs, clusters, and pipelines. Implement data ingestion and transformation pipelines using Azure Data Factory (ADF) and Azure Databricks. Integrate data from multiple sources including databases, APIs, files, cloud storage, and enterprise applications. Work with Azure Data Lake Storage (ADLS Gen2) for enterprise-scale data storage and processing. Design efficient data models and transformation frameworks to support analytics and reporting requirements. Perform Spark and Databricks performance tuning, including query optimization, partitioning, caching, cluster optimization, and workload management. Implement data quality, validation, monitoring, logging, and error-handling frameworks. Support batch and real-time/streaming data processing requirements. Implement security and access controls across Azure data platforms. Collaborate with Data Architects, Data Scientists, BI teams, and business stakeholders to deliver enterprise data solutions. Participate in code reviews, testing, deployment, troubleshooting, and production support. Implement CI/CD and DevOps practices for Azure data engineering solutions.
Required Skills: Strong hands-on experience with Azure Databricks – Mandatory. Strong experience with Apache Spark and PySpark. Strong programming experience in Python. Advanced SQL and data transformation skills. Hands-on experience with Azure Data Factory (ADF).Experience with Azure Data Lake Storage (ADLS Gen2).Strong understanding of Delta Lake and Lakehouse Architecture. Experience designing and developing ETL/ELT and enterprise data pipelines. Strong understanding of data warehousing, data modeling, and distributed data-processing concepts. Experience optimizing Databricks/Spark workloads for performance and scalability. Knowledge of Azure security, identity, monitoring, and data governance concepts. Experience with Git and CI/CD deployment processes.
Preferred Skills: Experience with Databricks Unity Catalog and data governance. Experience with Delta Live Tables / Lakeflow Declarative Pipelines. Knowledge of Azure Synapse Analytics and other Azure data services. Experience with streaming technologies such as Kafka or Azure Event Hubs. Experience working with Power BI or downstream analytics platforms. Familiarity with Terraform / Infrastructure as Code. Relevant Microsoft Azure or Databricks certifications are preferred.
Qualifications: Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related discipline. Strong professional experience in Azure Data Engineering and Databricks-based data platforms. Proven experience delivering scalable, production-grade data pipelines and enterprise data solutions.