Lead Databricks Engineer with Pyspark & Python Exp - US
ARCHIVED
We can't find an active application page for this role right now. It may reopen or be listed elsewhere. Use Next Steps to search for an active apply link and similar live jobs.
Lead Databricks EngineerLocation: NYC, NY OR Iselin, NJ (Hybrid – 3 days' work from office)Duration: 6+ MonthsJob Description:Inperson interview is must(Financial background is must)Major Responsibilities:Work on Finance data related to Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repoDesign, develop, and deploy Databricks jobs to process and analyze large volumes of data.Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines.Optimize Databricks jobs for performance and scalability to handle big data workloads.Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks.Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.Demonstrated proficiency with Data Analytics, Data InsightsProficient writing SQL queries and programming including stored procedures and reverse engineering existing processAzure Synapse/Bigquery/Redshift is good to have.Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards.Skills:5+ years' – Strong experience in Finance / Banking industry – Capital markets, investment banking - Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo10+ years - Enterprise Data Management10+ years - SQL Server based development of large datasets5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding.Azure Synapse/Bigquery/Redshift experience is good to have3+ years Python(numpy, pandas) coding experienceExperience with Snowflake utilities such as SnowSQL and SnowPipe - good to haveExperience in Data warehousing - OLTP, OLAP, Dimensions, Facts, and Data modeling.Previous experience leading an enterprise-wide Cloud Data Platform migration with strong architectural and design skillsCapable of discussing enterprise level services independent of technology stackExperience with Cloud based data architectures, messaging, and analyticsSuperior communication skillsCloud certification(s)Any experience with Regulatory Reporting is a PlusEducation:Minimally a BA degree within an engineering and/or computer science discipline