Senior Azure Data Migration & Dataverse Integration Engineer

Role: Senior Azure Data Migration & Dataverse Integration EngineerCompany: Arch SystemsLocation: RemoteType: Full-timePosition SummarySeeking a Senior Azure Data Migration & Dataverse Integration Engineer to design, develop, implement, and support enterprise data migration and synchronization capabilities between Microsoft SQL Server and Microsoft Dataverse.The position requires a senior hands-on engineer capable of owning the complete data pipeline lifecycle, including source analysis, data mapping, extraction, transformation, migration, synchronization, validation, exception handling, monitoring, performance optimization, and automated deployment.The engineer will develop and maintain repeatable pipelines supporting initial bulk migration, incremental migration, Change Data Capture (CDC), ongoing synchronization, and transition/cutover activities between legacy SQL Server applications and modern Microsoft Power Platform/Dataverse solutions.Primary ResponsibilitiesThe Data Migration Engineer will:Design and implement end-to-end SQL Server-to-Dataverse data migration pipelines.Develop automated pipelines using Azure Data Factory (ADF) and supporting Azure services.Analyze legacy SQL Server databases, tables, views, relationships, stored procedures, and data dependencies.Map legacy SQL Server data structures to Dataverse tables, columns, relationships, choices, lookups, and alternate keys.Develop repeatable processes for:Initial/bulk data migrationIncremental data migrationChange Data CaptureDelta processingScheduled synchronizationNear-real-time synchronizationLegacy-to-Dataverse synchronizationDataverse-to-legacy synchronization where requiredCutover and final synchronizationDevelop and maintain ADF pipelines, datasets, linked services, integration runtimes, triggers, parameters, variables, and reusable pipeline components.Develop transformation and data-cleansing logic required to convert legacy data into Dataverse-compatible structures.Implement controlled sequencing to maintain parent/child relationships and referential integrity.Develop lookup resolution and cross-reference processes for Dataverse GUIDs and legacy SQL identifiers.Implement restartable and recoverable migration processes.Develop pipeline logging, audit, exception handling, retry, and reconciliation capabilities.Validate migrated data for completeness, accuracy, integrity, and consistency.Perform source-to-target record-count and field-level reconciliation.Troubleshoot failed records, pipeline failures, API limitations, data-quality issues, and performance problems.Develop automated deployment processes for data pipeline components across DEV, TEST, UAT, and PROD environments.Azure Data Factory / Pipeline ResponsibilitiesThe candidate must have advanced hands-on experience designing and developing Azure Data Factory solutions.Responsibilities include:ADF pipeline architecture and developmentCopy ActivitiesLookup ActivitiesStored Procedure ActivitiesForEach and Until processingConditional processingPipeline dependenciesParameterized pipelinesDynamic expressionsMetadata-driven pipeline developmentLinked ServicesDatasetsIntegration Runtime configurationManaged Identity authenticationAzure Key Vault integrationPipeline triggers and schedulingIncremental load processingWatermark processingCDC patternsRetry and failure handlingPipeline logging and auditingPerformance tuning and parallelizationEnvironment-specific configurationInfrastructure and pipeline deployment automationThe engineer should be capable of developing reusable, metadata-driven pipeline patterns rather than creating independent hard-coded pipelines for every table.SQL Server ResponsibilitiesStrong SQL Server development and data engineering skills are required, including:Complex T-SQL developmentStored proceduresViewsCommon Table ExpressionsTemporary and staging tablesData profilingData cleansingData transformationJoins and relationship analysisPrimary and foreign key analysisIndexing and query optimizationSQL Server Change Data CaptureChange TrackingWatermark-based incremental extractionData validation queriesReconciliation queriesDuplicate detectionReferential-integrity validationThe engineer must be capable of analyzing legacy databases with limited documentation and determining the technical relationships necessary to migrate data successfully.Microsoft Dataverse ResponsibilitiesAdvanced Dataverse knowledge is required, including:Dataverse tables and columnsStandard and custom tablesTable relationshipsOne-to-many and many-to-many relationshipsLookupsChoice columnsAlternate keysGUID managementOwnership and security considerationsDataverse Web APIDataverse APIs and connectorsUpsert patternsBatch processingDuplicate detectionDataverse service protection/API limitsApplication users and service principalsEnvironment configurationSolutions and ALMThe candidate must understand how Dataverse differs from a traditional relational SQL database and design migration processes accordingly.Data SynchronizationThe engineer will design synchronization processes supporting coexistence between legacy and modernized applications during phased transition.This includes:SQL Server → Dataverse synchronizationDataverse → SQL Server synchronization where requiredUnidirectional and bidirectional synchronization patternsIncremental synchronizationCDC and watermark processingEvent-driven integration where applicableConflict detection and resolutionSource-of-record determinationDuplicate preventionIdempotent processingFailed-record reprocessingSynchronization audit historyData latency monitoringReconciliation between source and target systemsThe engineer must understand the risks associated with bidirectional synchronization, including update conflicts, synchronization loops, transaction ordering, and determination of the authoritative system of record.Data Migration ArchitectureThe candidate should be capable of implementing a controlled migration architecture such as:Legacy SQL Server → Extraction → Staging/Bronze → Validation/Transformation → Silver → Business Rules/Mapping → Gold/Target-Ready Data → ADF Load Pipeline → Dataverse → ReconciliationThe architecture must support traceability from the original source record through transformation and final Dataverse record creation.Pipeline Monitoring and OperationsDevelop operational capabilities to monitor migration and synchronization processing, including:Pipeline execution statusRecords extractedRecords transformedRecords successfully loadedRecords rejectedWarning and error countsSource and target record countsProcessing durationSynchronization latencyRetry statusFailed-record queuesReconciliation statusImplement monitoring using applicable Microsoft technologies such as:Azure MonitorApplication InsightsLog AnalyticsADF monitoringAzure SQL loggingPower BI or operational dashboardsDevOps and CI/CDThe engineer will establish controlled deployment and configuration management for all data migration components.Responsibilities include:Azure DevOps ReposBranching and source-control practicesPull requests and code reviewsAzure DevOps PipelinesAutomated DEV/TEST/UAT/PROD deploymentsEnvironment-specific parametersInfrastructure as CodeBicep and/or ARM templatesADF deployment automationSQL database deploymentDataverse configuration coordinationKey Vault and secret-management integrationRelease validationRollback proceduresAll pipelines, scripts, SQL objects, infrastructure definitions, configuration, and supporting code must be maintained under source control.Security RequirementsThe engineer will implement data migration capabilities consistent with enterprise security requirements, including:Microsoft Entra IDManaged IdentitiesService PrincipalsDataverse Application UsersRole-Based Access ControlLeast-privilege accessAzure Key VaultSecure connection managementEncryption in transit and at restAudit loggingEnvironment separationControlled production accessExperience supporting Federal Government, DoD, RMF, NIST 800-53, or comparable regulated environments is preferred.Required Technical Skills7+ years of data engineering, database development, integration, or data migration experience.4+ years of Microsoft Azure data engineering experience.Advanced Azure Data Factory development experience.Advanced Microsoft SQL Server and T-SQL experience.Strong Microsoft Dataverse experience.Experience migrating enterprise SQL data into Dataverse.Experience with CDC, incremental loads, delta processing, and synchronization.Experience developing ETL/ELT pipelines.Experience with REST APIs and Dataverse Web API.Experience with Azure SQL.Experience with Azure Key Vault and Managed Identities.Experience with Azure DevOps source control and pipelines.Experience implementing automated data validation and reconciliation.Preferred Technical ExperienceExperience with several of the following is highly desirable:Azure Data Lake Storage Gen2Azure SQL DatabaseAzure SQL Managed InstanceMicrosoft FabricAzure FunctionsAzure Service BusAzure Event GridLogic AppsPower AutomatePower AppsPower BISynapse AnalyticsC#/.NETPythonPowerShellBicepGitAzure DevOpsPreferred CertificationsOne or more of the following Microsoft certifications is preferred:DP-203 – Azure Data Engineer Associate or current Microsoft equivalentPL-400 – Power Platform Developer AssociatePL-600 – Power Platform Solution Architect ExpertMicrosoft Azure Administrator, Developer, or Solutions Architect certificationDesired Candidate ProfileThis position requires a hands-on senior data engineer, not solely a data architect or Power Platform administrator.The successful candidate must be able to independently take a legacy SQL data requirement through the complete technical lifecycle:Legacy SQL Analysis → Data Profiling → Source-to-Target Mapping → Pipeline Design → ADF Development → Transformation → Dataverse Load → CDC/Synchronization → Validation → Reconciliation → CI/CD → Production OperationsThe engineer must be capable of working directly with application developers, Power Platform developers, database administrators, solution architects, business analysts, and system owners to determine how legacy data will be migrated, synchronized, validated, and ultimately transitioned from the legacy environment. Key DeliverablesThe position will be responsible for producing and maintaining:Source-to-target data mappingsData migration pipelinesData synchronization pipelinesADF pipeline frameworkSQL extraction and transformation scriptsStaging database structuresCDC/incremental processing frameworkDataverse loading processesData-quality rulesException and retry processesMigration audit recordsData reconciliation reportsPipeline monitoringCI/CD deployment pipelinesInfrastructure-as-Code definitionsMigration runbooksCutover proceduresProduction support documentationTechnical architecture and data-flow diagrams