Senior Data Engineer

About the RoleWe are looking for an experienced Lead Data Engineer with deep expertise in Databricks, PySpark, and AWS. The ideal candidate will have a strong understanding of media measurement datasets (Nielsen) and a proven ability to lead multiple concurrent projects across onsite and offshore teams. This role requires a senior technical leader to define architecture, manage delivery, and mentor teams within the fast-paced New York media industry.Key Responsibilities- Technical Leadership: Architect and deliver enterprise-scale data platforms using Databricks, PySpark, Delta Lake, and AWS. Lead the development of media analytics pipelines, including the ingestion and modeling of Nielsen data.- Project & Delivery Management: Manage scope, timelines, and risks for multiple simultaneous projects. Collaborate with New York-based stakeholders and drive sprint delivery.- Onsite–Offshore Coordination: Act as the primary technical lead onsite, coordinating with offshore teams to ensure alignment, quality, and continuity.- Team Mentorship: Mentor junior and mid-level engineers through code reviews and architectural guidance.Required Skills- Databricks: Delta Lake, Workflows, Unity Catalog, Notebooks, and SQL.- Data Processing: Strong PySpark and distributed computing experience.- AWS: S3, EMR, Glue, Lambda, Step Functions, Kinesis, and IAM.- Domain Expertise: Proven experience handling Nielsen datasets (National/Local TV ratings, Digital Ad ratings, Panels).- Leadership: Experience leading large data engineering programs and managing senior stakeholders.Good to Have- Knowledge of Snowflake, Airflow, or Dagster.- Infrastructure as Code (Terraform / CloudFormation).- Exposure to ad-tech ecosystems (Comscore, Kantar, IAS, Moat) or ML feature pipelines.Soft Skills- Strong decision-making and ownership mindset.- Exceptional communication skills for interacting with senior business stakeholders.- Strategic, problem-solving, and delivery-focused approach.