Senior Data Engineer (DataStage & Databricks)
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
We are seeking an experienced Data Engineer with strong expertise in IBM DataStage and Databricks to design, develop, and optimize enterprise data integration and analytics solutions. The ideal candidate should have extensive experience building scalable ETL pipelines, migrating legacy DataStage workloads to modern cloud platforms, and working with large-scale data processing technologies.Required Skills8+ years of experience in Data Engineering and ETL development.Strong hands-on experience with IBM DataStage (Parallel Jobs, Server Jobs, Sequence Jobs).Expertise in Databricks using PySpark, Spark SQL, and Delta Lake.Experience with ETL/ELT pipeline development and optimization.Strong SQL development skills with databases such as Oracle, SQL Server, DB2, Snowflake, or PostgreSQL.Experience with cloud platforms such as AWS, Azure, or GCP.Hands-on experience with data migration from legacy ETL platforms to Databricks.Knowledge of data warehousing concepts, dimensional modeling, and star/snowflake schemas.Experience working with Git, CI/CD pipelines, and DevOps practices.Familiarity with workflow orchestration tools such as Airflow or Azure Data Factory.Strong analytical and troubleshooting skills.Excellent verbal and written communication skills.ResponsibilitiesDesign, develop, and maintain ETL processes using IBM DataStage.Build scalable data pipelines using Databricks, PySpark, and Spark SQL.Migrate existing DataStage jobs to Databricks while ensuring performance and data quality.Develop and maintain Delta Lake architectures for batch and streaming workloads.Optimize Spark jobs for high performance and scalability.Work closely with Data Architects, Business Analysts, and Application teams to understand business requirements.Implement data quality, validation, and reconciliation processes.Monitor production jobs, troubleshoot failures, and resolve performance issues.Ensure data governance, security, and compliance standards are followed.Participate in code reviews, design discussions, and Agile ceremonies.