AI Data Engineer – Azure Databricks | Python, PySpark & SQL

Location: Cary, NCWork Arrangement: Hybrid — 2–3 days onsite per weekEmployment Type: Full-time, Direct HireAbout the RoleWe are seeking an experienced AI Data Engineer to design, build, and optimize scalable data pipelines and modern data solutions using Microsoft Azure, Databricks, Python, PySpark, and SQL.In this role, you will work on production-grade data engineering solutions, improve data processing performance, implement reliable data workflows, and collaborate with engineering teams and business stakeholders.This is a direct full-time opportunity based in Cary, North Carolina, with a hybrid schedule requiring 2–3 days onsite per week.Work Authorization: This position has a U.S. citizenship requirement based on the applicable hiring requirements. Candidates must meet the confirmed eligibility criteria for the role.Key ResponsibilitiesDesign, develop, and maintain scalable batch and streaming data pipelines using Python, PySpark, and Azure Databricks.Build and optimize data processing solutions using Apache Spark and Delta Lake.Develop and maintain data integration workflows using Azure Data Factory and Azure Data Lake Storage Gen2.Implement data modeling, schema design, data quality checks, and performance optimization.Work with Databricks Unity Catalog, Jobs & Workflows, cluster management, and performance tuning.Develop CI/CD pipelines and infrastructure automation using Azure DevOps and Terraform.Implement automated testing and monitoring to ensure reliable data pipelines.Collaborate with engineers and stakeholders to document technical solutions and communicate design decisions.Required QualificationsStrong hands-on experience with Python, PySpark, and SQL.Production experience with Azure Databricks and Delta Lake.Experience designing and developing data pipelines for large-scale data processing.Hands-on experience with Azure Data Factory and Azure Data Lake Storage Gen2.Knowledge of data modeling, dimensional and normalized schemas, and data contracts.Experience troubleshooting and optimizing Apache Spark workloads.Understanding of CI/CD practices and automated testing.Strong problem-solving, technical documentation, and communication skills.Ability to work onsite in Cary, NC, 2–3 days per week.Must meet the confirmed work-authorization requirements for this position.Preferred QualificationsExperience with Databricks Unity Catalog, Jobs & Workflows, and Model Serving.Experience with Databricks Asset Bundles.Knowledge of Scala and advanced Spark optimization techniques.Experience with Terraform and infrastructure as code.Experience with automated testing of data pipelines and Azure DevOps.Work ArrangementThis is a hybrid, direct-hire position based in Cary, North Carolina. Candidates must be able to commute to the office for 2–3 days per week. Please apply only if the location and onsite schedule work for you.