Senior Data Engineer
Occupations:
Data Warehousing SpecialistsData ScientistsDatabase ArchitectsComputer Systems Engineers/ArchitectsSoftware DevelopersIndustries:
Management, Scientific, and Technical Consulting ServicesEducational Support ServicesWeb Search Portals, Libraries, Archives, and Other Information ServicesContinuing Care Retirement Communities and Assisted Living Facilities for the ElderlyBusiness Support ServicesAbout the RoleWe are looking for an experienced Lead Data Engineer with deep expertise in Databricks, PySpark, and AWS. The ideal candidate will have a strong understanding of media measurement datasets (Nielsen) and a proven ability to lead multiple concurrent projects across onsite and offshore teams. This role requires a senior technical leader to define architecture, manage delivery, and mentor teams within the fast-paced New York media industry.Key Responsibilities- Technical Leadership: Architect and deliver enterprise-scale data platforms using Databricks, PySpark, Delta Lake, and AWS. Lead the development of media analytics pipelines, including the ingestion and modeling of Nielsen data.- Project & Delivery Management: Manage scope, timelines, and risks for multiple simultaneous projects. Collaborate with New York-based stakeholders and drive sprint delivery.- Onsite–Offshore Coordination: Act as the primary technical lead onsite, coordinating with offshore teams to ensure alignment, quality, and continuity.- Team Mentorship: Mentor junior and mid-level engineers through code reviews and architectural guidance.Required Skills- Databricks: Delta Lake, Workflows, Unity Catalog, Notebooks, and SQL.- Data Processing: Strong PySpark and distributed computing experience.- AWS: S3, EMR, Glue, Lambda, Step Functions, Kinesis, and IAM.- Domain Expertise: Proven experience handling Nielsen datasets (National/Local TV ratings, Digital Ad ratings, Panels).- Leadership: Experience leading large data engineering programs and managing senior stakeholders.Good to Have- Knowledge of Snowflake, Airflow, or Dagster.- Infrastructure as Code (Terraform / CloudFormation).- Exposure to ad-tech ecosystems (Comscore, Kantar, IAS, Moat) or ML feature pipelines.Soft Skills- Strong decision-making and ownership mindset.- Exceptional communication skills for interacting with senior business stakeholders.- Strategic, problem-solving, and delivery-focused approach.