Systems Analyst III / Databricks Data Engineer
Systems Analyst III / Databricks Data Engineer Client: Texas Health and Human Services Commission (HHSC)Location: Austin, TX (Hybrid - 2 Days Onsite, 3 Days Remote)Duration: 12+ Months (High possibility of extension)Job Type: Contract (W2/1099/C2C)Interview: Microsoft Teams / In-PersonLocal Candidates Only Job Summary Texas Health and Human Services Commission (HHSC) is seeking an experienced Systems Analyst III / Databricks Data Engineer to design, develop, and optimize enterprise-scale data solutions using the Databricks platform. The ideal candidate will have extensive experience building scalable ETL/ELT pipelines, implementing modern Lakehouse architecture, and developing analytical dashboards that enable business decision-making. This role requires deep expertise in Databricks, Apache Spark, Delta Lake, Lakeflow Declarative Pipelines (DLT), SQL, Python/Scala, data warehousing, and cloud-based data engineering practices. The selected consultant will collaborate with business stakeholders, architects, and technical teams to deliver secure, scalable, and high-performance data solutions. Key Responsibilities Design, develop, and maintain scalable enterprise data solutions using the Databricks platform. Develop and optimize ETL/ELT pipelines using Apache Spark (PySpark or Scala). Build and maintain Lakehouse architectures utilizing Delta Lake and Medallion Architecture (Bronze, Silver, Gold). Implement Lakeflow Declarative Pipelines (formerly Delta Live Tables) for production-grade data processing. Develop and schedule workflows using Lakeflow Jobs (formerly Databricks Workflows) or equivalent orchestration tools such as Apache Airflow. Design and maintain dimensional data models including Star and Snowflake schemas. Develop high-performance SQL queries for analytical workloads. Create interactive dashboards and applications using Databricks SQL Dashboards and Databricks Apps. Implement enterprise data governance, metadata management, data quality validation, and security best practices. Monitor pipeline performance and optimize workloads for scalability, reliability, and cost efficiency. Analyze business requirements and translate them into technical solutions. Perform system analysis, troubleshooting, and production support activities. Collaborate with business users, project managers, developers, and architects throughout the software development lifecycle. Document technical designs, workflows, data models, and operational procedures. Present technical findings and business insights to both technical and non-technical stakeholders. Required Qualifications Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience). Minimum 8 years of IT experience supporting the design, development, deployment, or delivery of technology solutions. Minimum 8 years of experience with Databricks. Strong expertise in Apache Spark using PySpark or Scala. Extensive experience building enterprise ETL/ELT pipelines. Strong knowledge of Delta Lake. Experience implementing Lakeflow Declarative Pipelines (Delta Live Tables). Experience with Medallion Architecture (Bronze, Silver, Gold). Expertise in Lakehouse architecture. Strong experience with SQL. Strong programming skills in Python (preferred) or Scala. Experience designing dimensional data models (Star/Snowflake schema). Experience developing Databricks SQL Dashboards and Databricks Apps. Experience implementing data governance, data quality, security, and validation frameworks. Experience scheduling and orchestrating jobs using Lakeflow Jobs or Apache Airflow. Excellent verbal and written communication skills. Ability to work independently while collaborating across cross-functional teams. Preferred Qualifications Databricks Certified Data Engineer (Associate or Professional). Experience working with Texas State agencies or other public sector organizations. Experience with Git-based version control. Experience implementing CI/CD pipelines for data engineering projects. Familiarity with DevOps best practices. Experience working in Agile/Scrum environments. Technical Skills Data Engineering Databricks Apache Spark PySpark Scala Delta Lake Delta Live Tables (DLT) Lakeflow Declarative Pipelines Lakeflow Jobs Databricks Workflows Data Lakehouse Medallion Architecture Programming Python SQL Scala Data Warehousing Dimensional Modeling Star Schema Snowflake Schema ETL ELT Reporting & Analytics Databricks SQL Databricks Dashboards Databricks Apps Data Management Data Governance Data Quality Data Validation Data Security Metadata Management DevOps & Tools Git CI/CD Azure DevOps (preferred) Apache Airflow Soft Skills Strong analytical and problem-solving skills Excellent communication and presentation abilities Ability to interact with business and technical stakeholders Strong documentation skills Self-motivated and detail-oriented Ability to manage multiple priorities in a fast-paced environment