Senior Databricks Developer (IBM DataStage Migration)

We are looking for a highly skilled Databricks Developer with strong expertise in IBM DataStage and hands-on experience migrating enterprise ETL workloads from DataStage to Databricks. The ideal candidate should possess excellent analytical and logical thinking skills, be capable of solving complex data engineering challenges, and communicate effectively with business and technical stakeholders.Key ResponsibilitiesDesign, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.Lead and execute migration projects from IBM DataStage to Databricks, ensuring data integrity, performance, and reliability.Analyze existing DataStage jobs and redesign them into optimized Databricks solutions following best practices.Develop ETL/ELT processes for large-scale enterprise data integration.Optimize Spark jobs and SQL queries for performance and scalability.Perform code reviews, debugging, troubleshooting, and production support.Collaborate with solution architects, business analysts, QA teams, and other developers to deliver high-quality solutions.Implement data validation, reconciliation, and quality checks throughout migration activities.Document technical designs, migration strategies, and operational procedures.Participate in Agile ceremonies and contribute to continuous improvement initiatives.Required Skills7+ years of experience in Data Engineering or ETL development.Strong hands-on experience with IBM DataStage.Extensive experience developing solutions using Azure Databricks.Strong proficiency in PySpark, Spark SQL, and SQL.Proven experience in DataStage to Databricks migration projects.Experience working with Delta Lake, Azure Data Lake Storage (ADLS), and cloud-based data platforms.Strong understanding of ETL/ELT concepts, data warehousing, and data modeling.Experience with Git, Azure DevOps, or similar version control and CI/CD tools.Ability to troubleshoot complex production issues and optimize data processing workflows.Required CompetenciesExcellent logical thinking and analytical problem-solving skills with the ability to design efficient and scalable solutions.Strong critical thinking skills to evaluate migration approaches, identify risks, and recommend optimal solutions.Excellent verbal and written communication skills with the ability to interact confidently with business users, architects, and cross-functional teams.Ability to work independently while managing multiple priorities in a fast-paced environment.Strong attention to detail and commitment to delivering high-quality solutions.