Data Engineer

Senior Data EngineerAs a Senior Data Engineer, you will be part of a cross-functional development team that is focused on creating a forecasting platform at National Grid available to our business teams. Using the agile framework, you will build end-to-end pipelines based on rigorous engineering standards and coding practices to deliver data that is accessible and of the highest quality. A Senior Data Engineer will also contribute to the modernization of our architecture and tools to help increase our output, scalability, and speed. A Senior Data Engineer will design and develop highly scalable and extensible data pipelines which enable collection, storage, distribution, modeling, and analysis of large data sets from many channels.This position requires an innovative software engineer who is passionate about data & data quality. The ideal candidate will have strong data warehousing and API integration experience and the ability to develop scalable data pipelines that make data management and analytics/reporting faster, more insightful, and more efficient.Develop, test, document and support scalable data pipelinesBuild out and evolve data integrations including APIs to support continuing increases in data volume and complexityEstablish and follow data governance processes and guidelines to ensure data availability, usability, consistency, integrity, and securityBuild, implement, and maintain scalable solutions that align to our data governance standards and architectural road maps for data integrations, data storage, reporting, and analytic solutionsCollaborate with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility, and fostering data-driven decision making across the organizationDesign and develop data integrations and a data quality framework. Write unit/integration/functional tests and document workDesign, implement, and automate deployment of our distributed system for collecting and processing streaming events from multiple sourcesPerform data analysis needed to troubleshoot data-related issues and aid in the resolution of data issuesGuide and mentor junior engineers on coding best practices and optimizationQualifications:Education: 4-year college degree or equivalent combination of education and experience. Prefer an academic background in Computer Science, Mathematics, Statistics, or related technical field5 years of relevant work experience in analytics, data engineering, business intelligence or related fieldSkilled in object-oriented programming (Python in particular)Strong Experience in Python, PySpark and SQLStrong Experience in DatabricksExperience developing integrations across multiple systems and APIsExperience with or knowledge of Agile software development methodologiesExperience with cloud-based databases, specifically Azure technologies (e.g., Azure data lake, ADF, Azure DevOps and Azure Functions)Experience using SQL queries as well as writing and perfecting SQL queries in a business environment with large-scale, complex datasetsExperience with data warehouse technologies. Experience creating ETL and/or ELT jobsExcellent problem solving and troubleshooting skillsProcess oriented with great documentation skillsExperience designing data schemas and operating SQL/NoSQL database systems is a plusExperience in Kafka, Flink,Fivetran and Matillion is nice to haveExperience in Data Science and Machine Learning is nice to haveNice to have:Experience in software engineeringExperience in Snowflake