Splunk Observability Engineer

About the Company:At Syncredible Solutions, we believe in the transformative power of technology. we specialize in Digital transformation services, Enterprise solutions, Power BI Analytics solutions and staffing services.Synopsis:To design, implement, and optimize a full-stack observability strategy using the Splunk Observability Cloud (formerly SignalFx) and Splunk Enterprise/Cloud. You will ensure that engineering teams have 360-degree visibility into system health, moving the organization from reactive "firefighting" to proactive "pattern-based" incident prevention.Key ResponsibilitiesData Orchestration: Architect the ingestion of the "Three Pillars" (Metrics, Logs, Traces) using OpenTelemetry (OTel) collectors.Aggregation Strategy: Develop logic to aggregate high-cardinality data to reduce "noise" while maintaining "signal" for troubleshooting.Analytical Modeling: Use SPL (Search Processing Language) and SignalFlow to perform pattern analysis, detecting anomalies before they trigger traditional threshold alerts.Visual Storytelling: Build executive and technical dashboards that correlate disparate data points (e.g., showing how a spike in 500-errors in Logs relates to a specific span in a Trace).Required Hands on Technical Skills1. Telemetry & Data SpecializationLogs: Proficiency in "Logging-in-Context." You must be able to link logs directly to trace IDs so developers can jump from a failing trace to the specific line of code in the logs.Metrics: Expertise in SignalFlow (Splunk’s background streaming analytics language). You should know how to calculate percentiles ($P95, P99$), rates of change, and historical averages.Traces: Deep understanding of Distributed Tracing. You must know how to instrument applications (Java, Python, Go) to capture spans and identify bottlenecks in microservices.2. Pattern Analysis & AggregationAnomaly Detection: Ability to configure Metric Finder and MDetector using standard deviations or "Mean Absolute Deviation" to find outliers.Data Scrubbing: Skills in using Splunk Ingest Actions or Edge Processors to filter, mask, or aggregate data at the edge to save on license costs and improve search speed.Pattern Discovery: Using Splunk’s machine learning commands (e.g., findkeywords, cluster) to group millions of log events into a few dozen "patterns" for faster root cause analysis.3. Hands on - Dashboards & VisualizationHigh-Cardinality Handling: Designing dashboards that don’t "break" when viewing thousands of containers.Contextual Drill-downs: Building "Glass Tables" (in ITSI) or Unified Dashboards that allow a user to click a metric and immediately see the associated logs.Frameworks: Familiarity with the Dashboard Studio and JSON-based dashboard definitions for version control (GitOps).🎓 Preferred Qualifications & CertificationsDevOps & IAC skillsSplunk Cloud Certified Metrics User: Focuses on the metrics and alerting side.Splunk Core Certified Power User: Essential for mastering complex SPL for log analysis.OpenTelemetry Expert: Knowledge of the OTel Collector configuration (receivers, processors, exporters) is currently the most "in-demand" skill for this role.Equal Opportunity Statement:Syncredible Solutions is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.