Data Engineer

Position: Data EngineerLocation:Manhattan NY (Hybrid 3 days a week to office)Long TermKey SkillsSQL | Python | Spark | Airflow / ADF | Cloud data services | CI/CDRequired Qualifications8-10+ years in data engineeringStrong SQL and Python (or Scala) skillsExperience with orchestration tools such as Airflow or Azure Data FactoryHands-on experience with cloud data services and big data frameworks such as SparkUnderstanding of data warehousing and lakehouse conceptsFamiliarity with Git, CI/CD and infrastructure-as-codeExperience supporting ML/AI data needs is a plusKey ResponsibilitiesDesign, build and optimize batch and streaming data pipelinesImplement data models and transformations defined by the Data ArchitectIngest and integrate data from multiple source systems and APIsImplement data quality checks, monitoring and alertingOptimize performance and cost of data workloadsCollaborate with AI Developers and analysts to provision datasetsMaintain documentation, version control and CI/CD for data codeWork with offshore data engineers on knowledge transfer and delivery