Senior Data Engineer
We're Hiring: Senior Data Engineer – Dynamics 365 Analytics (Databricks & Microsoft Fabric)
About the Role:
Our client is building a modern analytics platform that turns our Dynamics 365 Finance & Operations data into trusted insights for the business. We're looking for a Senior Data Engineer to lead the build on Databricks and Microsoft Fabric, using AI-assisted tools like Copilot to deliver faster and more efficiently.
Key Responsibilities:
- Build and maintain ingestion pipelines that bring D365 F&O and other source data into the lakehouse in near real time
- Develop incremental load and change data capture (CDC) logic, including MERGE/upsert patterns and handling of deleted and late-arriving records
- Orchestrate and schedule pipelines using Databricks Workflows and Fabric Data Factory, with dependency management, retries, and alerting
- Optimize Delta Lake tables and Spark jobs for performance and cost (partitioning, Z-ordering, file compaction, cluster sizing)
- Build automated data quality checks and reconciliation against source system totals
- Publish curated tables to Microsoft Fabric and support BI developers building Power BI semantic models
- Apply CI/CD practices: version control, code reviews, automated testing, and deployments across dev, test, and production
- Monitor pipeline health, troubleshoot failures, and own production support for the data platform
- Use GitHub Copilot, Databricks Assistant, and Copilot in Fabric to accelerate coding, testing, and documentation, while reviewing all AI-generated code
- Document pipelines, data lineage, and table definitions for the wider team
Required Skills:
- 5+ years of experience in data engineering, including 2+ years on Azure data platforms
- Azure Databricks: PySpark, Spark SQL, Delta Lake, Unity Catalog, Databricks Workflows, Delta Live Tables, Databricks SQL Warehouse
- Microsoft Fabric: Lakehouse, Data Warehouse, OneLake (shortcuts and mirroring), Data Factory pipelines, Dataflows Gen2, Fabric notebooks
- Power BI: semantic modeling, DAX, Direct Lake and Import modes, row-level security, Power BI Service administration
- Dynamics 365 F&O data integration: Azure Synapse Link for Dataverse, Fabric Link, Dataverse, F&O data entities and table structures
- Azure services: ADLS Gen2, Azure Data Factory, Azure Key Vault, Microsoft Entra ID (managed identities, service principals), Azure Monitor
- AI and Copilot: GitHub Copilot, Copilot in Fabric and Power BI, Databricks Assistant, Fabric data agents; Azure OpenAI is a plus
- Languages: advanced SQL (T-SQL / Spark SQL), Python
- Data modeling: dimensional modeling (Kimball star schema), medallion architecture (bronze/silver/gold)
- DevOps: Git, Azure DevOps or GitHub Actions, Fabric and Databricks Git integration, Databricks Asset Bundles, Terraform or Bicep
- Data quality and governance: data validation frameworks, Microsoft Purview, lineage and access control
Nice to Have:
- Microsoft (DP-600 / DP-700) or Databricks certifications
- Exposure to machine learning or generative AI solutions
- Finance or supply chain domain knowledge