Azure Databricks Data Engineer Python & AI
Job Summary:
We are seeking an experienced Azure Data Engineer / Databricks Engineer with strong expertise in Azure, Databricks, Python, and modern data engineering. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data platforms while leveraging AI/ML capabilities to support data-driven business solutions. Experience in Wealth Management, Wealth Advisory, or Financial Services is highly desirable.
Key Responsibilities:
Design, develop, and maintain scalable data pipelines using Azure Databricks.
Develop robust data processing and transformation solutions using Python, PySpark, and SQL.Build and optimize ETL/ELT pipelines using Azure data services.
Work with Azure Data Lake Storage (ADLS) and cloud-based data platforms.
Develop Databricks notebooks, workflows, jobs, and Delta Lake solutions.
Implement data quality, validation, monitoring, and performance optimization.
Collaborate with data scientists, architects, analysts, and business stakeholders to deliver data solutions.
Integrate data from multiple structured and unstructured sources.
Support AI/ML and Generative AI initiatives through high-quality, well-governed data pipelines and platforms.
Implement CI/CD and DevOps practices for data engineering workloads.
Troubleshoot production issues and continuously improve pipeline reliability and performance.
Ensure adherence to data security, governance, and compliance standards.
Required Qualifications:
Strong experience in Azure Data Engineering.
Hands-on experience with Azure Databricks.
Strong programming skills in Python and PySpark.
Strong SQL skills.
Experience with Azure Data Lake Storage (ADLS Gen2).Experience with Delta Lake and data lakehouse architecture.
Experience developing and optimizing ETL/ELT pipelines.
Knowledge of Azure services such as Azure Data Factory, Azure Synapse, Azure Key Vault, and related data services.
Experience with Git and CI/CD pipelines.
Strong understanding of data modeling, data warehousing, and distributed data processing.
Experience preparing and managing data for AI/ML workloads.
Preferred Qualifications:
Exposure to Machine Learning, Generative AI, or LLM-based solutions.
Knowledge of Azure AI or Azure OpenAI.Experience with vector databases, RAG pipelines, or AI-enabled data applications.
Experience in Wealth Management, Wealth Advisory, Asset Management, Investment Management, or Financial Services.
Familiarity with financial products, portfolios, client data, investment analytics, or advisory platforms.