Data Engineer (Mid Level)-Orbit

Data Engineer – Insights (AI/ML) Irth Solutions is a leading provider of cloud-based SaaS software for damage prevention, asset integrity, stakeholder engagement and land management, helping energy, utility, telecom, and infrastructure companies protect their critical network infrastructure. With nearly three decades of industry experience, Irth serves customers across North America and continues to expand its platform with new data-driven and AI-powered capabilities.

Implement Bronze → Silver → Gold medallion architecture patterns for ingestion, transformation, cleansing, and standardization. Implement Change Data Capture (CDC) and Slowly Changing Dimensions (SCD Type 1 and Type 2).Handle schema evolution and changing source-system structures. Implement data validation, reconciliation, and quality rules as part of pipeline processing. Build reusable and maintainable pipeline components following established engineering standards. Configure and maintain Delta Lake storage structures, tables, schemas, partitions, and optimization routines. Apply Delta Lake performance and maintenance practices, including: OPTIMIZEZ-ORDERVACUUMAppropriate partitioning and file-management strategies Assist with implementation of metadata, cataloging, and lineage standards using Unity Catalog. Support integration between cloud storage platforms and Databricks, including: Amazon S3 → Databricks Azure Storage → Databricks Google Cloud Storage → Databricks Assist with implementation of scalable storage and processing patterns defined by the Data Architect. Implement automated data-quality checks, profiling, validation, and monitoring in accordance with enterprise governance standards. Apply data-quality rules at appropriate stages of the Bronze, Silver, and Gold layers. Implement RBAC policies, security controls, and data-classification tags defined by the enterprise governance model. Support implementation of metadata and lineage mapping across Unity Catalog and Microsoft Purview. Help ensure datasets are properly documented, classified, governed, and discoverable. Support remediation of data-quality and governance issues identified through monitoring or reviews. Build, schedule, monitor, and maintain production workflows using: Databricks Workflows Delta Live TablesAzure Data Factory (ADF) Other approved orchestration tools Contribute to CI/CD pipelines for data-engineering code, including source control, automated testing, deployment, and environment management. Support DEV → QA → PROD promotion processes. Monitor production pipelines and respond to failures and data-quality issues. Troubleshoot failed jobs, investigate root causes, and support pipeline recovery. Perform performance tuning across Spark jobs, SQL workloads, Delta tables, and data pipelines. Participate in operational improvements that increase pipeline reliability, scalability, and cost efficiency. Work directly with the Senior Data Architect to translate architecture designs and technical standards into actionable implementation tasks. Participate in architecture reviews, technical design discussions, coding reviews, and engineering standards meetings. Collaborate with Data Scientists, ML Engineers, Analysts, Product teams, and other engineering stakeholders to understand data requirements. Document: Data pipelines Data flows Data dictionaries Transformation logic Data-quality rules Test cases Job schedules Operational procedures Maintain clear and accurate technical documentation to support platform adoption, troubleshooting, and future development. Provide implementation feedback to the Data Architect and identify opportunities to improve platform patterns, tooling, and developer experience. Role ScopeThis is primarily an implementation-focused Data Engineering role. The Senior Data Architect will establish the overall platform architecture, standards, and design patterns; the Data Engineer will translate those patterns into reliable, production-ready pipelines and platform capabilities. The role provides an opportunity to gain deeper hands-on experience with Databricks, Spark, Delta Lake, Unity Catalog, cloud data platforms, data governance, and multi-cloud lakehouse engineering while contributing to a strategic enterprise data platform.