Azure Databricks Data Engineer
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
Overview:
As a Azure Databricks Data Engineer, you will design and maintain scalable data pipelines within Databricks, leveraging Delta Lake, Unity Catalog, and CI/CD practices. You will collaborate with cross-functional teams to translate data requirements into robust processing solutions, optimize workloads, and ensure governance and security. This role focuses on building scalable, reliable data platforms that enable data-driven insights at scale. You will work hands-on with PySpark, Python, and Delta Lake features while contributing to the SDLC and production deployments.
Responsibilities:
Design, develop, and maintain scalable ETL/ELT pipelines and notebooks using DatabricksBuild and optimize data processing solutions with PySpark and PythonImplement and manage data governance, security, and access controls using Unity CatalogLeverage Delta Lake features (Delta Sharing, Time Travel, CDF) and data optimization techniquesDevelop and maintain CI/CD pipelines for Databricks deployments following DevOps practicesPerform performance tuning and optimization of Databricks workloadsCollaborate with architects, business analysts, and stakeholders to define data requirementsSupport production deployments, troubleshooting, monitoring, and performance improvementsAdhere to coding standards, documentation, and SDLC best practices
Key requirements:
Deep Databricks experience with architecture and ecosystemHands-on Delta Lake and Delta SharingUnity Catalog, Structured Streaming, and workflow orchestrationPerformance tuning and optimizationData governance and access managementExperience with Medallion Architecture (Bronze/Silver/Gold) Strong Python coding skills and PySpark for large-scale processingSQL proficiency and query optimizationDevOps and CI/CD for Databricks workflowsGit-based source control and deployment automationExcellent verbal and written communicationStrong problem-solving and analytical abilitiesAbility to work in a large global teamDatabricksDelta LakeDelta Sharing