{"schemaVersion":"jobsearcher.job.v1","id":"d0b264cea3546d7052f3f4be","url":"https://jobsearcher.com/jobs/d0b264cea3546d7052f3f4be","canonicalUrl":"https://jobsearcher.com/jobs/d0b264cea3546d7052f3f4be","title":"Data Bricks Migration and Support engineer","description":"Job Title: Data Bricks Migration and Support engineerLocation: Seattle, WA / Bellevue, WA / Everett, WA / Renton, WA / Richardson, TX / Plano, TX / Dallas, TX / St. Louis, MO / Charleston, SC / Arlington, VA (Onsite)FulltimeMust Have Technical/Functional SkillsSuccessfully executed a data migration or modernization to Data Bricks, preferably IBM Data Stage to Data Bricks on AWSShould have Experience in handling Large Migrations to Data Bricks.Should have good analytical skills to compare the legacy and modern data platform end to end right from source to target.Good understanding of DataBricks implementation of Medallion layer architecture.Independently Lead and Managed large Data Bricks migrations.CI/CD Integration: Implement version control (e.g., Git) and automated deployment processes for Databricks assetsTechnical and architectural skills required are below.Core Data Engineering LanguagesExperience in Advanced SQL for building modular analytics workflows, utilizing advanced Common Table Expressions (CTEs), and writing high-performance queries inside Data Bricks SQL Analytics.Experience in Python or Scala to build, optimize, and debug complex data transformation scripts, custom functions, and machine learning pipelines.Big Data & Architecture CoreExperience in Apache Spark Ecosystem for understanding cluster execution flow, memory allocation, driver/worker nodes, and handling data frames.Experience in Delta Lake Architecture to understand ACID transactions on object storage, data skipping, partition strategies, and automated data compaction.Databricks Platform ExpertiseExperience in Delta Live Tables (DLT) & Workflows for constructing and orchestrating production-ready, declarative streaming, and batch ETL pipelines.Experience in Unity Catalog for setting up data governance, column/row-level access control, and tracking end-to-end data lineage across workspaces.Experience in Auto Loader for implementing modern, incremental data ingestion patterns from cloud blob storage into the lakehouse.Code Translation & RefactoringPipeline Conversion: Translate visual DataStage Parallel Jobs and Sequences into Python/PySpark scripts or Data bricks NotebooksLegacy Refactoring: Modernize legacy logic rather than applying \"lift and shift\" anti-patterns; adapt workflows to think in distributed DataFrames rather than DataStage stages.Logic Mapping: Map DataStage components-such as Aggregators, Joiners, Transformers, and Sort stages-to equivalent Spark operationsTesting & ReconciliationValidation & Reconciliation: Build automated reconciliation frameworks to compare row counts, checksums, and aggregate sums between legacy DataStage outputs and new Databricks outputData Cleansing: Identify and resolve data type discrepancies, null-handling differences, and encoding issues during the extraction and loading phasesPlatform Orchestration & GovernanceOrchestration: Replace DataStage sequence jobs with Databricks workflows ( or external orchestrators like Azure Data Factory/Airflow) to schedule and manage dependenciesData Governance: Enforce data lineage, security, and cataloging using Unity Catalog to ensure compliance in the new Lakehouse environment.GOOD TO Cloud Infrastructure & CI/CDCloud Providers (AWS): Understanding underlying cloud object storage , identity access management (IAM), and network security configurations.DevOps & Bundles: Familiarity with Databricks Asset Bundles (DABs) and CI/CD tools to automate the deployment of workspaces and pipeline assets.Legacy Assessment & Migration MechanicsCode Conversion & Translation: The ability to parse legacy code structures and refactor them into Databricks-native code.AI-Assisted Migration: Skills in using AI coding assistants and open framework agent tools to analyze application interdependencies, automate schema mapping, and accelerate lift-and-shift workloadsCode Conversion & Translation: The ability to parse legacy code structures from ETL pipelines, Informatica, data Stage preferredExperience working in Agile teams and understanding of data governance frameworks.ResponsibilitiesSupport post-migration environment from IBM DataStage to DatabricksIncident & Lifecycle ManagementCI/CD Deployment: Support code deployments across Development, Test, and Production environments using Databricks Repos and REST APIsMonitoring & Alerting: Set up monitoring via Databricks System Tables and observability tools to catch job failures, data anomalies, or latency spikes earlyPipeline Maintenance & OrchestrationWorkflow Management: Transition from DataStage job sequences to native data bricks workflows for scheduling, dependency tracking, and alertsETL Refactoring: Troubleshoot and fix issues in generated PySpark or Spark SQL code that replaced legacy DataStage Transformer or Lookup stagesStreaming & Batch Integration: Support ongoing data ingestion using data bricks autoloader to process files continuously from cloud storagePerformance Tuning & Cost OptimizationCompute Management: Monitor and configure serverless or classic clusters to prevent over-provisioningQuery Optimization: Analyze Spark execution plans. Replace inefficient row-by-row processing logic (a common DataStage carryover) with vectorized operations and native Spark functionsStorage Optimization: Maintain Delta Lake tables by enforcing layout optimization ((ZORDER)Data Governance & SecurityAccess Control: Implement granular permissions, column-masking, and row-level filters using Data bricks unity catalog to replace DataStage's legacy security p oliciesData Quality: Utilize Delta Live Tables (DLT) to build pipelines with built-in, declarative data quality expectations and monitoringAdditional SkillsExcellent communicationSkillsAbility to collaborate with Legacy and Modernize application teams and stake holders","company":"AceStack","rawCompany":"acestack","city":"Seattle","state":"WA","isRemote":false,"isActive":false,"createdAt":"2026-09-11T13:00:16.223Z","occupations":[{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"},{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-1243.00","title":"Database Architects","slug":"database-architects"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"518210","title":"Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services","slug":"computing-infrastructure-providers-data-processing-web-hosting-and-related-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Data Bricks Migration and Support engineer","description":"Job Title: Data Bricks Migration and Support engineerLocation: Seattle, WA / Bellevue, WA / Everett, WA / Renton, WA / Richardson, TX / Plano, TX / Dallas, TX / St. Louis, MO / Charleston, SC / Arlington, VA (Onsite)FulltimeMust Have Technical/Functional SkillsSuccessfully executed a data migration or modernization to Data Bricks, preferably IBM Data Stage to Data Bricks on AWSShould have Experience in handling Large Migrations to Data Bricks.Should have good analytical skills to compare the legacy and modern data platform end to end right from source to target.Good understanding of DataBricks implementation of Medallion layer architecture.Independently Lead and Managed large Data Bricks migrations.CI/CD Integration: Implement version control (e.g., Git) and automated deployment processes for Databricks assetsTechnical and architectural skills required are below.Core Data Engineering LanguagesExperience in Advanced SQL for building modular analytics workflows, utilizing advanced Common Table Expressions (CTEs), and writing high-performance queries inside Data Bricks SQL Analytics.Experience in Python or Scala to build, optimize, and debug complex data transformation scripts, custom functions, and machine learning pipelines.Big Data & Architecture CoreExperience in Apache Spark Ecosystem for understanding cluster execution flow, memory allocation, driver/worker nodes, and handling data frames.Experience in Delta Lake Architecture to understand ACID transactions on object storage, data skipping, partition strategies, and automated data compaction.Databricks Platform ExpertiseExperience in Delta Live Tables (DLT) & Workflows for constructing and orchestrating production-ready, declarative streaming, and batch ETL pipelines.Experience in Unity Catalog for setting up data governance, column/row-level access control, and tracking end-to-end data lineage across workspaces.Experience in Auto Loader for implementing modern, incremental data ingestion patterns from cloud blob storage into the lakehouse.Code Translation & RefactoringPipeline Conversion: Translate visual DataStage Parallel Jobs and Sequences into Python/PySpark scripts or Data bricks NotebooksLegacy Refactoring: Modernize legacy logic rather than applying \"lift and shift\" anti-patterns; adapt workflows to think in distributed DataFrames rather than DataStage stages.Logic Mapping: Map DataStage components-such as Aggregators, Joiners, Transformers, and Sort stages-to equivalent Spark operationsTesting & ReconciliationValidation & Reconciliation: Build automated reconciliation frameworks to compare row counts, checksums, and aggregate sums between legacy DataStage outputs and new Databricks outputData Cleansing: Identify and resolve data type discrepancies, null-handling differences, and encoding issues during the extraction and loading phasesPlatform Orchestration & GovernanceOrchestration: Replace DataStage sequence jobs with Databricks workflows ( or external orchestrators like Azure Data Factory/Airflow) to schedule and manage dependenciesData Governance: Enforce data lineage, security, and cataloging using Unity Catalog to ensure compliance in the new Lakehouse environment.GOOD TO Cloud Infrastructure & CI/CDCloud Providers (AWS): Understanding underlying cloud object storage , identity access management (IAM), and network security configurations.DevOps & Bundles: Familiarity with Databricks Asset Bundles (DABs) and CI/CD tools to automate the deployment of workspaces and pipeline assets.Legacy Assessment & Migration MechanicsCode Conversion & Translation: The ability to parse legacy code structures and refactor them into Databricks-native code.AI-Assisted Migration: Skills in using AI coding assistants and open framework agent tools to analyze application interdependencies, automate schema mapping, and accelerate lift-and-shift workloadsCode Conversion & Translation: The ability to parse legacy code structures from ETL pipelines, Informatica, data Stage preferredExperience working in Agile teams and understanding of data governance frameworks.ResponsibilitiesSupport post-migration environment from IBM DataStage to DatabricksIncident & Lifecycle ManagementCI/CD Deployment: Support code deployments across Development, Test, and Production environments using Databricks Repos and REST APIsMonitoring & Alerting: Set up monitoring via Databricks System Tables and observability tools to catch job failures, data anomalies, or latency spikes earlyPipeline Maintenance & OrchestrationWorkflow Management: Transition from DataStage job sequences to native data bricks workflows for scheduling, dependency tracking, and alertsETL Refactoring: Troubleshoot and fix issues in generated PySpark or Spark SQL code that replaced legacy DataStage Transformer or Lookup stagesStreaming & Batch Integration: Support ongoing data ingestion using data bricks autoloader to process files continuously from cloud storagePerformance Tuning & Cost OptimizationCompute Management: Monitor and configure serverless or classic clusters to prevent over-provisioningQuery Optimization: Analyze Spark execution plans. Replace inefficient row-by-row processing logic (a common DataStage carryover) with vectorized operations and native Spark functionsStorage Optimization: Maintain Delta Lake tables by enforcing layout optimization ((ZORDER)Data Governance & SecurityAccess Control: Implement granular permissions, column-masking, and row-level filters using Data bricks unity catalog to replace DataStage's legacy security p oliciesData Quality: Utilize Delta Live Tables (DLT) to build pipelines with built-in, declarative data quality expectations and monitoringAdditional SkillsExcellent communicationSkillsAbility to collaborate with Legacy and Modernize application teams and stake holders","datePosted":"2026-09-11T13:00:16.223Z","dateModified":"2026-09-11T13:00:16.223Z","hiringOrganization":{"@type":"Organization","name":"AceStack","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Seattle","addressRegion":"WA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"d0b264cea3546d7052f3f4be"},"url":"https://jobsearcher.com/jobs/d0b264cea3546d7052f3f4be"}}