Senior Data Engineer – Streaming, Analytics & Visualization

CompanyenGenJob Summary***CANDIDATE MUST BE US Citizen (due to contractual/access requirements)***As a Senior Data Engineer, you will design and operate real‐time and analytical data pipelines that power enterprise reporting and decision‐making, including security and operational telemetry data. You will be responsible for ingesting, modeling, and optimizing high‐volume streaming and batch data in Azure Data Explorer (ADX), enabling fast, reliable analytics at scale.This role blends deep data engineering with analytics enablement. You will structure and transform data using modern cloud and streaming technologies (Kafka, Cribl, Azure Data Factory), applying medallion architecture principles to ensure data quality, performance, and governance across the full data lifecycle.A core responsibility is translating engineered data into business‐ready insights by developing curated datasets, analytics, and Power BI (and Tableau) dashboards used by internal teams and leadership. Success in this role requires strong proficiency in KQL, SQL, and Python, experience optimizing ADX for analytical workloads, and the ability to partner with stakeholders—including security and risk teams—to turn business questions into scalable, high‐impact data products.Essential ResponsibilitiesDesign, develop, and maintain robust data processes and solutions to ensure the efficient movement and transformation of data across multiple systemsDevelop and maintain data models, databases, and data warehouses to support business intelligence and analytics needsCollaborate with stakeholders across IT, product, analytics, and business teams to gather requirements and provide data solutions that meet organizational needsMonitor work against production schedule, provide progress updates, and report any issues or technical difficulties to lead developers regularlyImplement and manage data governance practices, ensuring data quality, integrity, and compliance with relevant regulations.Collaborate on the design and implementation of data security measures, including access controls, encryption, and data maskingMentor other associate and intermediate data engineers as neededPerform data analysis and provide insights to support decision‐making across various departmentsStay current with industry trends and emerging technologies in data engineering, recommending new tools and best practices as neededOther duties as assigned or requested.Experience Required5 years of experience in design and analysis of algorithms, data structures, and design patterns in the building and deploying of scalable, highly available systems5 years of experience in a data engineering, ETL development, or data management role.5 years of experience in SQL and experience with database technologies (e.g., MySQL, PostgreSQL, MongoDB).5 years of experience in data warehousing concepts and experience with data warehouse solutions (e.g., Snowflake, Redshift, BigQuery)Experience Preferred7 years of experience with data pipeline and workflow management tools (e.g., Apache Airflow, AWS Glue, Azure Data Factory).7 years of experience defining system architectures and exploring technical feasibility trade‐offs for optimizing short term execution while planning for long term technical capabilities7 years of experience working with a variety of technology systems, designing solutions or developing data solutions in healthcare7 years of experience with cloud platforms (AWS, Azure, GCP) and their respective data services7 years of experience in data governance, data quality, and data security best practices7 years of experience translating requirements, design mockups, prototypes or user stories into technical designs7 years of experience in producing data‐related code that is fault‐tolerant, efficient, and maintainable7 years of experience structuring and transforming data from multiple source systems.7 years of experience creating data visualizations and dashboards using Power BI or Tableau for internal and external audiences.7 years of experience using Cribl as a streaming and transformation solution.7 years of experience using Azure Data Factory as a transformation solution.5 years of experience using Azure Data ExplorerSkillsDemonstrated ability to achieve stretch goals in a highly innovative and fast‐paced environmentAdaptability: Strong ability to take on diverse tasks and projects, adapting to the evolving needs of the organizationAnalytical Thinking: Strong analytical skills with a focus on detail and accuracyInterest and ability to learn other data development technologies/languages as neededTechnical Proficiency: Comfortable with a range of data tools and technologies, with a willingness to learn new skills as neededStrong track record in designing and implementing large‐scale data sourcesStrong sense of ownership, urgency, and driveDemonstrated passion for user experience and improving usabilityTeam Collaboration: A team player who can work effectively in cross‐functional environmentsExperience and willingness to mentor junior data engineers and help develop their skills and leadershipEducation RequiredBachelor's degree in Computer Science, Information Systems, Data Science, Computer Engineering or related fieldEducation PreferredMaster's degree in Computer Science, Information Systems, Data Science, Computer Engineering or related fieldLicenses or Certifications RequiredNoneLicenses or Certifications PreferredNoneLanguage (Other than English)NoneTravel Requirement0% - 25%Pay Range Minimum$102,700.00Pay Range Maximum$164,600.00EEO StatementHighmark Health and its affiliates prohibit discrimination against qualified individuals based on their status as protected veterans or individuals with disabilities and prohibit discrimination against all individuals based on any category protected by applicable federal, state, or local law.Accommodation ContactFor accommodation requests, please contact HR Services Online at HRServices@highmarkhealth.orgCalifornia Consumer Privacy Act NoticeCalifornia Consumer Privacy Act Employees, Contractors, and Applicants Notice#J-18808-Ljbffr