Zortech Solutions

Optimize Databricks jobs for performance and scalability to handle big data workloads.. Implement best practices for data management, security, and governance within the Databricks environment.. Azure Synapse/Bigquery/Redshift is good to have.. 5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform.. Azure Synapse/Bigquery/Redshift experience is good to have

databricks

Role: Lead Databricks Engineer with Pyspark & Python Exp Location: NYC, NY OR Iselin, NJ (Hybrid – 3 days’ work from office) Duration: 6+ Months Job Description Inperson interview is must (Financial background is must) Major Responsibilities <ul><li>Work on Finance data related to Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo</li><li>Design, develop, and deploy Databricks jobs to process and analyze large volumes of data. </li><li>Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines. </li><li>Optimize Databricks jobs for performance and scalability to handle big data workloads. </li><li>Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks. </li><li>Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.</li><li>Demonstrated proficiency with Data Analytics, Data Insights</li><li>Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process</li><li>Azure Synapse/Bigquery/Redshift is good to have.</li><li>Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards. </li></ul>Skills <ul><li>5+ years’ – Strong experience in Finance / Banking industry – Capital markets, investment banking - Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo</li><li>10+ years - Enterprise Data Management</li><li>10+ years - SQL Server based development of large datasets</li><li>5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding.</li><li>Azure Synapse/Bigquery/Redshift experience is good to have</li><li>3+ years Python(numpy, pandas) coding experience</li><li>Experience with Snowflake utilities such as SnowSQL and SnowPipe - good to have</li><li>Experience in Data warehousing - OLTP, OLAP, Dimensions, Facts, and Data modeling.</li><li>Previous experience leading an enterprise-wide Cloud Data Platform migration with strong architectural and design skills</li><li>Capable of discussing enterprise level services independent of technology stack</li><li>Experience with Cloud based data architectures, messaging, and analytics</li><li>Superior communication skills</li><li>Cloud certification(s)</li><li>Any experience with Regulatory Reporting is a Plus </li></ul>Education <ul><li>Minimally a BA degree within an engineering and/or computer science discipline</li><li>Master’s degree strongly preferred</li></ul>

START NEW SEARCH