Data Engineering Architect

Data Platform ManagerPittsburgh, PA – 3 days/week on-siteAbout eNGINEeNGINE builds Technical Teams. We are a Solutions and Placement firm shaped by decades of interaction with Technical professionals. Our inspiration is continuous learning and engagement with the markets we serve, the talent we represent, and the teams we build. Our Consulting Workforce is encouraged to enjoy career fulfillment in the form of challenging projects, schedule flexibility, and paid training/certifications. Successful outcomes start and finish with eNGINE.Position OvervieweNGINE is seeking a Data Platform Manager to lead the design, evolution, and optimization of a modern cloud-based data ecosystem built on Microsoft Azure and Databricks. This individual will serve as both a technical leader and hands-on contributor, driving enterprise data initiatives that support analytics, reporting, machine learning, and business intelligence across the organization.This role combines architecture, engineering leadership, and platform ownership. The ideal candidate brings deep expertise in Databricks and Azure data services, along with a passion for building scalable solutions, mentoring technical teams, and delivering reliable data products that enable informed business decisions.Working closely with senior leadership, this individual will help shape the organization's data strategy while ensuring the platform remains secure, performant, and ready to support future growth.ResponsibilitiesData Platform LeadershipProvide technical leadership and mentorship to a team of data engineers and platform specialists.Establish engineering standards, development best practices, and code review processes.Partner with business stakeholders, analysts, and data science teams to translate business requirements into scalable technical solutions.Support production operations by helping resolve data platform incidents and identifying opportunities for continuous improvement.Cloud Data EngineeringDesign and deliver enterprise-grade data pipelines within Azure Databricks and the broader Azure ecosystem.Build efficient data ingestion frameworks capable of processing both batch and real-time data sources.Develop scalable ELT/ETL solutions that ensure data quality, availability, and consistency across multiple systems.Enable seamless integration of data originating from APIs, databases, event streams, and third-party platforms.Data Architecture & Lakehouse StrategyDefine and maintain a Lakehouse architecture leveraging Delta Lake and Databricks best practices.Implement and optimize Bronze, Silver, and Gold data layers to support analytics and operational workloads.Develop logical and physical data models that support reporting, advanced analytics, and future scalability.Drive architectural decisions around storage, partitioning, performance, and long-term maintainability.Platform Performance & OptimizationContinuously improve Spark workloads, Databricks clusters, and query performance to maximize efficiency and reduce cloud spend.Identify opportunities for workload optimization through partitioning, indexing strategies, and execution plan analysis.Monitor platform health, scalability, and reliability while implementing proactive performance improvements.Establish governance around cluster configuration, auto-scaling policies, and resource utilization.Governance, Security & ComplianceImplement enterprise-grade data governance utilizing Unity Catalog and Azure-native security services.Ensure appropriate role-based access controls, data lineage tracking, and data protection measures are in place.Promote data quality through validation frameworks, monitoring processes, and standardized governance practices.Collaborate with security and compliance teams to align data management practices with organizational requirements.Advanced Analytics & Machine Learning EnablementPartner with data scientists to operationalize machine learning solutions and deploy models into production environments.Support MLOps initiatives using modern tooling and automation practices.Deliver data products that empower business intelligence, predictive analytics, and data-driven decision making.Required QualificationsBachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline.5+ years of experience designing and building enterprise data platforms in cloud environments.2+ years of hands-on experience with Azure Databricks and large-scale Spark-based data processing.Previous experience leading, mentoring, or managing data engineering teams.Strong understanding of modern data architecture patterns, including Lakehouse, Data Mesh, and Master Data Management concepts.Demonstrated ability to work directly with business stakeholders and translate business needs into technical solutions.Technical ExpertiseAzure & Cloud ServicesExperience with:Azure DatabricksAzure Data Factory (ADF)Azure Data Lake Storage (ADLS Gen2)Azure SQL DatabaseAzure Synapse AnalyticsAzure Networking (VNETs)Azure Key VaultAzure Identity and Access ManagementProgramming & DevelopmentStrong hands-on experience with:Python (PySpark)SQLScalaPowerShellJavaData Engineering & Big DataExperience building and supporting solutions utilizing:Apache SparkApache KafkaAirflowdbtMLflowDistributed data processing frameworksBatch and streaming data architecturesDatabases & StorageKnowledge of:DatabricksSQL ServerTeradataBigQueryRelational and NoSQL database technologiesParquet, Avro, and ORC file formatsData compression and storage optimization techniquesPreferred CertificationsMicrosoft Certified: Azure Data Engineer Associate (DP-203)Databricks Certified Data Engineer ProfessionalMicrosoft Certified: Azure Solutions Architect ExpertNext StepsNo C2C, relocation, referral, or sponsorship candidates for this role. For finer details on how eNGINE can impact your career, apply today!