Lead Databricks Engineer with Pyspark & Python
Lead Databricks Engineer
Location: NYC, NY OR Iselin, NJ (Hybrid – 3 days' work from office)
Duration: 6+ Months
Major Responsibilities:
- Work on Finance data related to Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo
- Design, develop, and deploy Databricks jobs to process and analyze large volumes of data.
- Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines.
- Optimize Databricks jobs for performance and scalability to handle big data workloads.
- Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks.
- Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.
- Demonstrated proficiency with Data Analytics, Data Insights
- Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process
- Azure Synapse/Bigquery/Redshift is good to have.
- Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards.
Skills:
5+ years' – Strong experience in Finance / Banking industry – Capital markets, investment banking - Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo 10+ years - Enterprise Data Management 10+ years - SQL Server based development of large datasets 5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding.