Senior Azure Data Engineer
Job title: Senior Azure Data EngineerLocation: Texas (Onsite)Employment Type: Full-time (W2 only, No C2C)Experience: 10+ YearsJob Summary: We are seeking an experienced Azure Data Engineer with 10+ years of experience in designing, building, and optimizing large-scale data processing systems and data pipelines on the Microsoft Azure cloud platform. The ideal candidate will have strong expertise in data engineering, ETL/ELT development, big data processing, data warehousing, and cloud-based data solutions. The candidate will work closely with data architects, analysts, and business stakeholders to design and implement scalable data solutions that support enterprise analytics and reporting.Key Responsibilities:Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using Azure Data Factory (ADF), Azure Databricks, and Azure Synapse Analytics to process and transform large volumes of structured and unstructured data.Build and optimize big data processing solutions using Apache Spark, PySpark, Scala, or SQL within Azure Databricks, ensuring high performance and efficient data transformation.Design and implement modern data warehouse and lakehouse architectures using Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), and Azure SQL Database to support enterprise-level analytics and reporting.Develop and maintain data ingestion frameworks to integrate data from multiple sources such as databases, APIs, flat files, streaming platforms, and third-party applications into Azure-based data platforms.Implement data modeling techniques including star schema, snowflake schema, and dimensional modeling to support business intelligence and analytical workloads.Develop SQL-based data transformation and data quality validation processes using T-SQL, Spark SQL, and stored procedures to ensure accuracy, consistency, and reliability of data.Integrate data engineering solutions with business intelligence and analytics tools such as Power BI, Tableau, or Azure Analysis Services for reporting and data visualization.Implement real-time and streaming data processing solutions using Azure Event Hub, Azure Stream Analytics, or Apache Kafka to support near real-time analytics.Optimize data storage, partitioning, indexing, and query performance across Azure-based data platforms to ensure efficient data retrieval and processing.Implement data governance, security, and compliance using Azure Active Directory (AAD), Role-Based Access Control (RBAC), data encryption, and data masking techniques.Automate deployment and data pipeline orchestration using CI/CD pipelines with Azure DevOps, GitHub Actions, or Jenkins, ensuring reliable and consistent deployment processes.Monitor and troubleshoot data pipelines, workflows, and cloud resources using Azure Monitor, Log Analytics, and Application Insights to ensure system reliability and performance.Collaborate with data architects, DevOps engineers, and cross-functional Agile teams to design scalable data engineering solutions aligned with enterprise data strategies.Mentor junior data engineers and contribute to best practices in data engineering, cloud architecture, and performance optimization.Technical Skills:Cloud Platform: Microsoft AzureAzure Data Services: Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), Azure SQL DatabaseBig Data Technologies: Apache Spark, PySpark, Scala, Spark SQLProgramming Languages: Python, SQL, ScalaData Warehousing: Azure Synapse Analytics, Azure SQL Data Warehouse, Snowflake Concepts, Dimensional ModelingETL/ELT Tools: Azure Data Factory, Databricks WorkflowsStreaming Technologies: Azure Event Hub, Azure Stream Analytics, Apache KafkaDatabases: SQL Server, Azure SQL Database, PostgreSQL, MySQL, NoSQLData Visualization Tools: Power BI, TableauVersion Control: Git, GitHub, Bitbucket, GitLabCI/CD Tools: Azure DevOps, Jenkins, GitHub ActionsContainerization: Docker, KubernetesOperating Systems: Windows, LinuxMethodologies: Agile, Scrum, DevOps, DataOps