{"schemaVersion":"jobsearcher.job.v1","id":"3839e4c60056fbcec74c9f79","url":"https://jobsearcher.com/jobs/3839e4c60056fbcec74c9f79","canonicalUrl":"https://jobsearcher.com/jobs/3839e4c60056fbcec74c9f79","title":"Databricks Engineer","description":"Overview:We are seeking an experienced Databricks Engineer with strong expertise in modern data engineering, large-scale data migration, and Databricks Lakehouse architecture. The ideal candidate will have hands-on experience migrating PySpark and Hive workloads from AWS EMR or legacy Hadoop platforms to Databricks while ensuring data quality, governance, security, scalability, and performance.The candidate should possess strong experience with PySpark, Spark SQL, Delta Lake, Unity Catalog, Spark Declarative Pipelines (SDP), and enterprise-scale data engineering best practices.Key ResponsibilitiesDesign, develop, and maintain scalable data pipelines using Databricks.Develop modern ETL/ELT solutions using PySpark and Spark SQL.Lead migration initiatives from AWS EMR and legacy Hadoop environments to Databricks Lakehouse.Migrate PySpark workloads from AWS EMR to Databricks.Convert Hive-based ETL jobs to Databricks using Delta Lake.Modernize legacy data warehouse and Hadoop ecosystems into Databricks Lakehouse Architecture.Convert and optimize Hive SQL, Spark SQL, and PySpark workloads.Build scalable and reusable data transformation frameworks.Develop and optimize Spark Declarative Pipelines (SDP).Perform end-to-end data validation, reconciliation, and quality assurance.Design and implement enterprise data quality frameworks.Optimize Databricks jobs for performance, scalability, reliability, and cost efficiency.Work closely with architects, data engineers, analysts, and business stakeholders.Manage CI/CD pipelines and deployment automation for Databricks workloads.Ensure compliance with enterprise governance, security, and data management standards.Required Technical SkillsDatabricksStrong experience with Databricks Lakehouse PlatformDelta LakeDatabricks WorkflowsDatabricks SQLDatabricks RuntimePerformance tuning and optimizationData EngineeringPySparkSpark SQLApache SparkLarge-scale distributed data processingETL / ELT developmentData pipeline developmentPerformance tuningUnity CatalogHands-on experience with:Unity Catalog implementationSetup and administrationFine-grained RBACData governanceData lineageAudit capabilitiesData security implementationSpark Declarative Pipelines (SDP)Experience with:Designing and implementing Spark Declarative PipelinesPipeline orchestrationMonitoring and optimizationBuilding reusable and scalable transformation frameworksData Validation & Data QualityData validation framework designSource-to-target reconciliationValidation of migrated datasetsData quality monitoringData observabilityData accuracy and completeness validationMigration ExperienceProven experience migrating:PySpark workloads from AWS EMR to DatabricksHive ETL workloads to DatabricksLegacy Hadoop ecosystems to DatabricksLegacy Data Warehouses to Databricks LakehouseExperience with:Code conversionTestingPerformance optimizationCutover planning and executionCloud TechnologiesStrong experience with AWS services:AWS EMRAmazon S3AWS GlueIAMAWS LambdaExposure to Azure Databricks is a plus.DevOps & AutomationExperience with:Azure DevOpsGitHub ActionsJenkinsCI/CD implementationInfrastructure as Code (Terraform preferred)Required QualificationsBachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field.5+ years of Data Engineering experience.3+ years of hands-on Databricks experience.Strong programming skills in Python and SQL.Strong understanding of Spark architecture and distributed computing.Experience working in Agile/Scrum environments.Preferred QualificationsDatabricks Certified Data Engineer (Associate or Professional)Experience with enterprise data governance and catalogingKnowledge of Lakehouse ArchitectureExperience with multi-cloud environmentsStrong understanding of enterprise security best practicesNice to HaveApache AirflowKafkaStructured StreamingData Mesh ArchitectureData Observability platformsLakehouse Architecture patternsAzure DatabricksTerraformRequired SkillsDatabricksPySparkApache SparkSpark SQLDelta LakeUnity CatalogSpark Declarative Pipelines (SDP)AWS EMRHiveData EngineeringETLData MigrationData ValidationData QualityData GovernancePerformance TuningCI/CDAzure DevOpsGitHub ActionsTerraformPythonSQLSkills:Digital : Databricks","company":"Transeu","rawCompany":"transeu","city":"Plano","state":"TX","isRemote":false,"isActive":false,"createdAt":"2026-10-24T13:35:24.397Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Databricks Engineer","description":"Overview:We are seeking an experienced Databricks Engineer with strong expertise in modern data engineering, large-scale data migration, and Databricks Lakehouse architecture. The ideal candidate will have hands-on experience migrating PySpark and Hive workloads from AWS EMR or legacy Hadoop platforms to Databricks while ensuring data quality, governance, security, scalability, and performance.The candidate should possess strong experience with PySpark, Spark SQL, Delta Lake, Unity Catalog, Spark Declarative Pipelines (SDP), and enterprise-scale data engineering best practices.Key ResponsibilitiesDesign, develop, and maintain scalable data pipelines using Databricks.Develop modern ETL/ELT solutions using PySpark and Spark SQL.Lead migration initiatives from AWS EMR and legacy Hadoop environments to Databricks Lakehouse.Migrate PySpark workloads from AWS EMR to Databricks.Convert Hive-based ETL jobs to Databricks using Delta Lake.Modernize legacy data warehouse and Hadoop ecosystems into Databricks Lakehouse Architecture.Convert and optimize Hive SQL, Spark SQL, and PySpark workloads.Build scalable and reusable data transformation frameworks.Develop and optimize Spark Declarative Pipelines (SDP).Perform end-to-end data validation, reconciliation, and quality assurance.Design and implement enterprise data quality frameworks.Optimize Databricks jobs for performance, scalability, reliability, and cost efficiency.Work closely with architects, data engineers, analysts, and business stakeholders.Manage CI/CD pipelines and deployment automation for Databricks workloads.Ensure compliance with enterprise governance, security, and data management standards.Required Technical SkillsDatabricksStrong experience with Databricks Lakehouse PlatformDelta LakeDatabricks WorkflowsDatabricks SQLDatabricks RuntimePerformance tuning and optimizationData EngineeringPySparkSpark SQLApache SparkLarge-scale distributed data processingETL / ELT developmentData pipeline developmentPerformance tuningUnity CatalogHands-on experience with:Unity Catalog implementationSetup and administrationFine-grained RBACData governanceData lineageAudit capabilitiesData security implementationSpark Declarative Pipelines (SDP)Experience with:Designing and implementing Spark Declarative PipelinesPipeline orchestrationMonitoring and optimizationBuilding reusable and scalable transformation frameworksData Validation & Data QualityData validation framework designSource-to-target reconciliationValidation of migrated datasetsData quality monitoringData observabilityData accuracy and completeness validationMigration ExperienceProven experience migrating:PySpark workloads from AWS EMR to DatabricksHive ETL workloads to DatabricksLegacy Hadoop ecosystems to DatabricksLegacy Data Warehouses to Databricks LakehouseExperience with:Code conversionTestingPerformance optimizationCutover planning and executionCloud TechnologiesStrong experience with AWS services:AWS EMRAmazon S3AWS GlueIAMAWS LambdaExposure to Azure Databricks is a plus.DevOps & AutomationExperience with:Azure DevOpsGitHub ActionsJenkinsCI/CD implementationInfrastructure as Code (Terraform preferred)Required QualificationsBachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field.5+ years of Data Engineering experience.3+ years of hands-on Databricks experience.Strong programming skills in Python and SQL.Strong understanding of Spark architecture and distributed computing.Experience working in Agile/Scrum environments.Preferred QualificationsDatabricks Certified Data Engineer (Associate or Professional)Experience with enterprise data governance and catalogingKnowledge of Lakehouse ArchitectureExperience with multi-cloud environmentsStrong understanding of enterprise security best practicesNice to HaveApache AirflowKafkaStructured StreamingData Mesh ArchitectureData Observability platformsLakehouse Architecture patternsAzure DatabricksTerraformRequired SkillsDatabricksPySparkApache SparkSpark SQLDelta LakeUnity CatalogSpark Declarative Pipelines (SDP)AWS EMRHiveData EngineeringETLData MigrationData ValidationData QualityData GovernancePerformance TuningCI/CDAzure DevOpsGitHub ActionsTerraformPythonSQLSkills:Digital : Databricks","datePosted":"2026-10-24T13:35:24.397Z","dateModified":"2026-10-24T13:35:24.397Z","hiringOrganization":{"@type":"Organization","name":"Transeu","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Plano","addressRegion":"TX","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"3839e4c60056fbcec74c9f79"},"url":"https://jobsearcher.com/jobs/3839e4c60056fbcec74c9f79"}}