JOBSEARCHER

Databricks Engineer

TranseuPlano, TXL6 LeadOctober 24th, 2026
Overview: We are seeking an experienced Databricks Engineer with strong expertise in modern data engineering, large-scale data migration, and Databricks Lakehouse architecture. The ideal candidate will have hands-on experience migrating PySpark and Hive workloads from AWS EMR or legacy Hadoop platforms to Databricks while ensuring data quality, governance, security, scalability, and performance. The candidate should possess strong experience with PySpark, Spark SQL, Delta Lake, Unity Catalog, Spark Declarative Pipelines (SDP), and enterprise-scale data engineering best practices.
Key Responsibilities: Design, develop, and maintain scalable data pipelines using Databricks. Develop modern ETL/ELT solutions using PySpark and Spark SQL.Lead migration initiatives from AWS EMR and legacy Hadoop environments to Databricks Lakehouse. Migrate PySpark workloads from AWS EMR to Databricks. Convert Hive-based ETL jobs to Databricks using Delta Lake. Modernize legacy data warehouse and Hadoop ecosystems into Databricks Lakehouse Architecture. Convert and optimize Hive SQL, Spark SQL, and PySpark workloads. Build scalable and reusable data transformation frameworks. Develop and optimize Spark Declarative Pipelines (SDP).Perform end-to-end data validation, reconciliation, and quality assurance. Design and implement enterprise data quality frameworks. Optimize Databricks jobs for performance, scalability, reliability, and cost efficiency. Work closely with architects, data engineers, analysts, and business stakeholders. Manage CI/CD pipelines and deployment automation for Databricks workloads. Ensure compliance with enterprise governance, security, and data management standards.
Required Technical Skills:
  • DatabricksStrong experience with Databricks Lakehouse Platform
  • Delta Lake
  • Databricks Workflows
  • Databricks SQLDatabricks Runtime
  • Performance tuning and optimization
  • Data Engineering
  • PySparkSpark SQLApache SparkLarge-scale distributed data processingETL / ELT development
  • Data pipeline development
  • Performance tuning
  • Unity Catalog
  • Hands-on experience with:
  • Unity Catalog implementation
  • Setup and administration
  • Fine-grained RBACData governance
  • Data lineage
  • Audit capabilities
  • Data security implementation
  • Spark Declarative Pipelines (SDP)
  • Experience with:
  • Designing and implementing Spark Declarative Pipelines
  • Pipeline orchestration
  • Monitoring and optimization
  • Building reusable and scalable transformation frameworks
  • Data Validation & Data QualityData validation framework design
  • Source-to-target reconciliation
  • Validation of migrated datasets
  • Data quality monitoring
  • Data observability
  • Data accuracy and completeness validation
  • Migration Experience
  • Proven experience migrating:
  • PySpark workloads from AWS EMR to Databricks
  • Hive ETL workloads to DatabricksLegacy Hadoop ecosystems to DatabricksLegacy Data Warehouses to Databricks Lakehouse
  • Experience with:
  • Code conversion
  • TestingPerformance optimization
  • Cutover planning and execution
  • Cloud Technologies
  • Strong experience with AWS services:
  • AWS EMRAmazon S3AWS GlueIAMAWS Lambda
  • Exposure to Azure Databricks is a plus.
  • DevOps & Automation
  • Experience with:
  • Azure DevOpsGitHub Actions
  • JenkinsCI/CD implementation
  • Infrastructure as Code (Terraform preferred)
Required Qualifications: Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field.5+ years of Data Engineering experience.3+ years of hands-on Databricks experience. Strong programming skills in Python and SQL.Strong understanding of Spark architecture and distributed computing. Experience working in Agile/Scrum environments.
Preferred Qualifications:
  • Databricks Certified Data Engineer (Associate or Professional)
  • Experience with enterprise data governance and cataloging
  • Knowledge of Lakehouse Architecture
  • Experience with multi-cloud environments
  • Strong understanding of enterprise security best practices
Nice to Have:
  • Apache Airflow
  • KafkaStructured Streaming
  • Data Mesh Architecture
  • Data Observability platforms
  • Lakehouse Architecture patterns
  • Azure Databricks
  • TerraformRequired Skills
  • DatabricksPySparkApache SparkSpark SQLDelta LakeUnity Catalog
  • Spark Declarative Pipelines (SDP)AWS EMRHiveData EngineeringETLData Migration
  • Data Validation
  • Data QualityData Governance
  • Performance TuningCI/CDAzure DevOpsGitHub Actions
TerraformPythonSQL
Skills: Digital : Databricks