Enterprise Data Engineer (Databricks)

Job Title: Enterprise Data Engineer (Databricks)Location: Hybrid – Austin, TXDuration: 5 MonthsPay Rate: $65/hr on 1099Role SummaryThe Enterprise Data Engineer will design, build, and operate scalable data pipelines within an Azure-based Databricks Lakehouse architecture. The primary focus is to deliver and maintain a software-driven data model for analytics and data consumption.This is a hands-on, execution-focused role responsible for engineering reliable data ingestion from multiple data sources, performing transformations, implementing data quality checks, and delivering curated datasets integrated with ServiceNow (ITSM/ITSLM) and ApptioOne (ITFM).The role involves close collaboration with data architects, platform teams, providers, and stakeholders to translate architectural designs into scalable, governed, and production-ready data solutions. The work follows Agile software engineering practices, including GitHub-based workflows and CI/CD-driven SDLC processes.Key ResponsibilitiesDesign, build, and maintain data models supporting data consumption, integration, semantic analytics, reporting, and executive dashboards. Develop scalable data ingestion and transformation pipelines using Azure PaaS services, Databricks, Delta Lake, Python, and Spark SQL. Implement integrations for ServiceNow operational data (SLA, incidents, CMDB) and ApptioOne financial and cost allocation data. Develop and enforce data quality checks, validation rules, and monitoring mechanisms for end-to-end pipeline reliability. Apply Unity Catalog governance including data access control, lineage management, and schema enforcement as per architectural standards. Optimize Databricks Lakehouse performance including pipeline efficiency, storage layout, and query optimization. Support CI/CD pipelines and DevOps automation for data engineering workflows using Azure DevOps and GitHub Actions. Collaborate with architects, stakeholders, Capgemini teams, and service providers to deliver reporting and analytics solutions. Troubleshoot production data issues and ensure operational stability of analytics and reporting systems. Maintain documentation, runbooks, and operational standards for Databricks data pipelines. Required Skills & Experience5+ years of experience in Data Engineering or Analytics Engineering roles. Hands-on experience with Databricks, Delta Lake, and Spark-based data pipelines. Strong understanding of Medallion Architecture, especially Gold/Platinum layer implementation. Proficiency in Python, SQL, and Spark (PySpark or Spark SQL). Experience integrating enterprise systems such as ServiceNow (SLA, incident, CMDB data). Experience working with financial or cost management platforms (e.g., ApptioOne or similar ITFM tools). Strong understanding of data modeling techniques and methodologies. Familiarity with Unity Catalog for data governance and access control. Experience with Power BI or similar BI tools consuming Lakehouse datasets. Experience with Azure data services (e.g., ADLS Gen2, orchestration tools, integration patterns), Azure DevOps, and GitHub-based CI/CD pipelines. Preferred QualificationsExperience supporting public sector data initiatives. Familiarity with ITIL 4 / ITIL 5 frameworks and SLA-based reporting. Experience with financial systems, SLA analytics, operational KPIs, or cost transparency dashboards. Exposure to MLflow, Feature Store, or AI/ML pipelines (implementation support role, not architecture ownership).