Data Engineer – Azure Databricks

Data Engineer – Azure Databricks Location: Seattle, WA (Preferred) Also Open To: St. Louis, MO | Dallas/Plano, TX | Charleston, SC | Ridley Park, PA Duration: 6+ Months (Extension Possible) Employment Type: W2 Only Work Authorization: US Citizen or Green Card Holders Only Position Overview We are seeking an experienced Data Engineer with strong Azure Databricks expertise to support a large-scale enterprise data migration and modernization initiative. This role requires a well-rounded engineer with hands-on experience across the full software development lifecycle, including architecture, development, migration, deployment, production support, and ongoing optimization of cloud-based data platforms. The ideal candidate will possess deep expertise in Python, PySpark, Azure Databricks, SQL, Azure services, CI/CD, Infrastructure as Code, and production operations. This position involves close collaboration with business stakeholders, application teams, infrastructure teams, and data consumers to deliver scalable, secure, and high-performing data solutions. Work Breakdown 60% Development & Engineering 20% Business & Stakeholder Collaboration 20% Production Support & Operations Required Technical SkillsData Engineering & Development Strong programming experience in Python and PySpark Advanced SQL skills for analytics, ETL, and data transformation workloads Experience designing, developing, and optimizing large-scale ETL/ELT pipelines Strong understanding of data modeling, data integration, and data governance principles Experience processing large volumes of structured and unstructured data Azure Databricks Hands-on experience with Azure Databricks Cluster configuration, optimization, and management Job orchestration and scheduling Workspace administration and governance Performance tuning and workload optimization Monitoring and troubleshooting production workloads Azure Cloud Services Strong experience with: Azure Storage Accounts Azure Synapse Analytics Azure Key Vault Virtual Machine Scale Sets (VMSS) Azure Function Apps Azure Web Apps Azure Log Analytics Workspace Service Principals Managed Identities Azure Container Apps (ACA) Azure Container Instances Infrastructure & DevOps Experience creating and managing Azure infrastructure using ARM Templates Strong experience with GitLab and Azure DevOps CI/CD pipeline development and release management Source control and branching strategies Infrastructure as Code (IaC) best practices Security & Networking Experience implementing secure access using: Azure Active Directory Managed Identities Service Principals Understanding of Azure networking concepts Experience integrating secure networking solutions within cloud data platforms Soft Skills Excellent communication and stakeholder management skills Strong analytical and problem-solving abilities Ability to work effectively in Agile/Scrum environments Experience collaborating with cross-functional teams and business users Key Responsibilities Design, develop, and maintain end-to-end data pipelines and ETL/ELT solutions using Python and PySpark. Build, optimize, and support large-scale data processing workloads within Azure Databricks. Configure and manage Databricks clusters, job scheduling, governance controls, and monitoring frameworks. Develop scalable and secure cloud-based data integration solutions using Azure services. Collaborate with business stakeholders and application teams to gather requirements and translate them into technical solutions. Implement secure authentication and authorization mechanisms using Azure Active Directory, Managed Identities, and Service Principals. Create and maintain Infrastructure as Code (IaC) solutions using ARM Templates. Configure and support Azure services including Storage Accounts, Synapse, Key Vault, VMSS, Function Apps, Web Apps, Log Analytics Workspace, and Container Services. Partner with networking and security teams to design secure cloud-based data architectures. Implement monitoring, alerting, logging, and cost optimization strategies across data workloads. Build and maintain CI/CD pipelines using GitLab and Azure DevOps. Support production deployments, incident management, troubleshooting, and operational activities. Participate in Agile ceremonies including sprint planning, daily stand-ups, backlog grooming, and retrospectives. Ensure compliance with enterprise data governance, lineage, security, and regulatory requirements. Provide on-call and production support for critical business applications and data platforms. Work closely with both legacy and modernization teams to support enterprise transformation initiatives. Ideal Candidate Profile The ideal candidate will have: 8+ years of Data Engineering experience Strong Azure Databricks implementation experience Proven experience in large-scale migration and modernization projects Experience owning solutions from development through production deployment and support Strong cloud engineering, automation, and operational support background Ability to thrive in a highly collaborative enterprise environment Pay: $55.00 - $60.00 per hour Work Location: In person