{"schemaVersion":"jobsearcher.job.v1","id":"6df3c69636c25a6aa75e0116","url":"https://jobsearcher.com/jobs/6df3c69636c25a6aa75e0116","canonicalUrl":"https://jobsearcher.com/jobs/6df3c69636c25a6aa75e0116","title":"Splunk Observability Engineer","description":"Role: Splunk Observability EngineerDuration: 6 monthsLocation: Chicago, IL - onsiteSynopsisTo design, implement, and optimize a full-stack observability strategy using the Splunk Observability Cloud (formerly SignalFx) and Splunk Enterprise/Cloud. You will ensure that engineering teams have 360-degree visibility into system health, moving the organization from reactive \"firefighting\" to proactive \"pattern-based\" incident prevention.Key Responsibilities· Data Orchestration: Architect the ingestion of the \"Three Pillars\" (Metrics, Logs, Traces) using OpenTelemetry (OTel) collectors.· Aggregation Strategy: Develop logic to aggregate high-cardinality data to reduce \"noise\" while maintaining \"signal\" for troubleshooting.· Analytical Modeling: Use SPL (Search Processing Language) and SignalFlow to perform pattern analysis, detecting anomalies before they trigger traditional threshold alerts.· Visual Storytelling: Build executive and technical dashboards that correlate disparate data points (e.g., showing how a spike in 500-errors in Logs relates to a specific span in a Trace).Required Hands on Technical Skills1. Telemetry & Data Specialization· Logs: Proficiency in \"Logging-in-Context.\" You must be able to link logs directly to trace IDs so developers can jump from a failing trace to the specific line of code in the logs.· Metrics: Expertise in SignalFlow (Splunk’s background streaming analytics language). You should know how to calculate percentiles ($P95, P99$), rates of change, and historical averages.· Traces: Deep understanding of Distributed Tracing. You must know how to instrument applications (Java, Python, Go) to capture spans and identify bottlenecks in microservices.2. Pattern Analysis & Aggregation· Anomaly Detection: Ability to configure Metric Finder and MDetector using standard deviations or \"Mean Absolute Deviation\" to find outliers.· Data Scrubbing: Skills in using Splunk Ingest Actions or Edge Processors to filter, mask, or aggregate data at the edge to save on license costs and improve search speed.· Pattern Discovery: Using Splunk’s machine learning commands (e.g., findkeywords, cluster) to group millions of log events into a few dozen \"patterns\" for faster root cause analysis.3. Hands on - Dashboards & Visualization· High-Cardinality Handling: Designing dashboards that don’t \"break\" when viewing thousands of containers.· Contextual Drill-downs: Building \"Glass Tables\" (in ITSI) or Unified Dashboards that allow a user to click a metric and immediately see the associated logs.· Frameworks: Familiarity with the Dashboard Studio and JSON-based dashboard definitions for version control (GitOps).Preferred Qualifications & Certifications· DevOps & IAC skills· Splunk Cloud Certified Metrics User: Focuses on the metrics and alerting side.· Splunk Core Certified Power User: Essential for mastering complex SPL for log analysis.· OpenTelemetry Expert: Knowledge of the OTel Collector configuration (receivers, processors, exporters) is currently the most \"in-demand\" skill for this role","company":"Syncredible Solutions","rawCompany":"syncredible solutions","city":"Chicago","state":"IL","isRemote":false,"isActive":false,"createdAt":"2026-09-06T08:22:51.577Z","occupations":[{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1243.01","title":"Data Warehousing Specialists","slug":"data-warehousing-specialists"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Splunk Observability Engineer","description":"Role: Splunk Observability EngineerDuration: 6 monthsLocation: Chicago, IL - onsiteSynopsisTo design, implement, and optimize a full-stack observability strategy using the Splunk Observability Cloud (formerly SignalFx) and Splunk Enterprise/Cloud. You will ensure that engineering teams have 360-degree visibility into system health, moving the organization from reactive \"firefighting\" to proactive \"pattern-based\" incident prevention.Key Responsibilities· Data Orchestration: Architect the ingestion of the \"Three Pillars\" (Metrics, Logs, Traces) using OpenTelemetry (OTel) collectors.· Aggregation Strategy: Develop logic to aggregate high-cardinality data to reduce \"noise\" while maintaining \"signal\" for troubleshooting.· Analytical Modeling: Use SPL (Search Processing Language) and SignalFlow to perform pattern analysis, detecting anomalies before they trigger traditional threshold alerts.· Visual Storytelling: Build executive and technical dashboards that correlate disparate data points (e.g., showing how a spike in 500-errors in Logs relates to a specific span in a Trace).Required Hands on Technical Skills1. Telemetry & Data Specialization· Logs: Proficiency in \"Logging-in-Context.\" You must be able to link logs directly to trace IDs so developers can jump from a failing trace to the specific line of code in the logs.· Metrics: Expertise in SignalFlow (Splunk’s background streaming analytics language). You should know how to calculate percentiles ($P95, P99$), rates of change, and historical averages.· Traces: Deep understanding of Distributed Tracing. You must know how to instrument applications (Java, Python, Go) to capture spans and identify bottlenecks in microservices.2. Pattern Analysis & Aggregation· Anomaly Detection: Ability to configure Metric Finder and MDetector using standard deviations or \"Mean Absolute Deviation\" to find outliers.· Data Scrubbing: Skills in using Splunk Ingest Actions or Edge Processors to filter, mask, or aggregate data at the edge to save on license costs and improve search speed.· Pattern Discovery: Using Splunk’s machine learning commands (e.g., findkeywords, cluster) to group millions of log events into a few dozen \"patterns\" for faster root cause analysis.3. Hands on - Dashboards & Visualization· High-Cardinality Handling: Designing dashboards that don’t \"break\" when viewing thousands of containers.· Contextual Drill-downs: Building \"Glass Tables\" (in ITSI) or Unified Dashboards that allow a user to click a metric and immediately see the associated logs.· Frameworks: Familiarity with the Dashboard Studio and JSON-based dashboard definitions for version control (GitOps).Preferred Qualifications & Certifications· DevOps & IAC skills· Splunk Cloud Certified Metrics User: Focuses on the metrics and alerting side.· Splunk Core Certified Power User: Essential for mastering complex SPL for log analysis.· OpenTelemetry Expert: Knowledge of the OTel Collector configuration (receivers, processors, exporters) is currently the most \"in-demand\" skill for this role","datePosted":"2026-09-06T08:22:51.577Z","dateModified":"2026-09-06T08:22:51.577Z","hiringOrganization":{"@type":"Organization","name":"Syncredible Solutions","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"Chicago","addressRegion":"IL","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"6df3c69636c25a6aa75e0116"},"url":"https://jobsearcher.com/jobs/6df3c69636c25a6aa75e0116"}}