Data Bricks Migration

Role: Data Bricks Migration and Support engineer Location: Seattle, WA / Bellevue, WA / Everett, WA / Renton, WA / Richardson, TX / Plano, TX / Dallas, TX / St. Louis, MO / Charleston, SC / Arlington, VAResponsibilitiesSupport post-migration environment from IBM DataStage to DatabricksIncident & Lifecycle ManagementCI/CD Deployment: Support code deployments across Development, Test, and Production environments using Databricks Repos and REST APIsMonitoring & Alerting: Set up monitoring via Databricks System Tables and observability tools to catch job failures, data anomalies, or latency spikes earlyPipeline Maintenance & OrchestrationWorkflow Management: Transition from DataStage job sequences to native data bricks workflows for scheduling, dependency tracking, and alertsETL Refactoring: Troubleshoot and fix issues in generated PySpark or Spark SQL code that replaced legacy DataStage Transformer or Lookup stagesStreaming & Batch Integration: Support ongoing data ingestion using data bricks autoloader to process files continuously from cloud storagePerformance Tuning & Cost OptimizationCompute Management: Monitor and configure serverless or classic clusters to prevent over-provisioningQuery Optimization: Analyze Spark execution plans. Replace inefficient row-by-row processing logic (a common DataStage carryover) with vectorized operations and native Spark functionsStorage Optimization: Maintain Delta Lake tables by enforcing layout optimization (\(ZORDER\)Data Governance & SecurityAccess Control: Implement granular permissions, column-masking, and row-level filters using Data bricks unity catalog to replace DataStage's legacy security policiesData Quality: Utilize Delta Live Tables (DLT) to build pipelines with built-in, declarative data quality expectations and monitoring