Reliability Engineer
Job Description
Company:
Lawrence Berkeley National Laboratory( _Through Bay Systems Consulting Inc._ )
1 year full-time Contract.
Extension is based on company budget and performance
National Energy Research Scientific Computing Center (NERSC) - Site Reliability Engineer.
As a Site Reliability Engineer in the Operations Technology Group, you will be a member of a 24x7 team that helps ensure NERSC is accessible, reliable, and secure for our scientific users. This position leverages advanced data collection and monitoring systems to proactively manage the health of our environment.
DUTIES & RESPONSIBILITIES:
Onsite 5-day weekly schedule consisting of Owl (midnight–8 am) shifts to monitor the NERSC HPC Facility.
Review and respond to alerts from computer systems, storage, network, and other data
center/facility-related systems by triaging or calling the appropriate on-call staff.
Create appropriate solutions to improve processes, prevent issue recurrence, and automate
responses to all routine service conditions.
Identify issues and propose solutions that will improve monitoring capabilities or provide better
automation for triage.
Possess expertise in ServiceNow and its usage to develop and implement customized service
management solutions.
Respond to alerts from multiple systems to ensure that data collection continues 24/7, providing
real-time information for diagnoses.
Develop and maintain tools within the monitoring pipeline in collaboration with the Operations
Team.
Create new software to provide alerts and notifications from HPC system APIs into the
monitoring pipeline.
Builds and maintains application/tool configurations to ensure software runs reliably as data and
user demands grow.
Collaborate with other groups at NERSC to ensure that communication and workflows are clearly
understood.
Work closely with other NERSC groups to coordinate center-wide maintenance activities and
manage diagnostic and notification software during maintenance periods.
Perform regular physical and logical walkthroughs of the data center floor to monitor
environmental health, power distribution units, and cooling infrastructure to ensure peak
operational efficiency.
Provide accurate information in the trouble ticketing system for outages, maintenance updates,
and other incidents so that workflows and protocols can be appropriately tracked by others.
Work on and resolve problems of diverse scope where data analysis requires the evaluation of
identifiable factors.
Demonstrate good judgment in selecting methods and techniques for obtaining solutions.
Work on and resolve complex issues where the analysis of situations or data requires an in-depth
evaluation of variable factors.
REQUIREMENTS:
Experience in or willingness to work within a 24/7 onsite team environment to support
large-scale data centers or critical installations.
Experience on Linux shell and working in a command-line (e.g. SSH) environment.
Experience with developing tools using various programming languages such as C, C++, Perl,
Java, or Python or a scripting language with knowledge of standard software development
practices.
Motivated, self-starter who can learn technologies that improve data center management in
areas like Kubernetes, Prometheus/VictoriaMetrics, Alertmanager, building management
software, evaporative
cooling, and power utilization.
Experience with network security: configuring/maintaining ACLs, knowledge of firewalls
Experience collaborating across technical teams to resolve operational bottlenecks and ensure
system reliability and alignment with service-level objectives.
Good to Have:
Practical experience in developing and deploying Agentic AI or autonomous
automation tools to streamline technical tasks.
Experience with ServiceNow implementation is a plus
Familiarity with ITSM best practices and an understanding of how to align service lifecycles with
business goals is preferred.
SKILLSStrong hands-on knowledge of the Linux shell and working in a command-line (e.g. SSH)
environment.
Strong hands-on knowledge on various programming languages such as C, C++, Perl, Java, or Python
or a scripting language with knowledge of standard software development practices.
Knowledge of and ability to work on large data communications networks/ Network Protocols and IT
infrastructure supporting highly available systems and applications.
Strong communication skills and ability to work effectively across multiple technical teams.
Ability to build, and deploy Agentic AI solutions that utilize autonomous agents to
automate decision-making, optimize complex workflows, and enhance proactive system monitoring.