Senior Observability Operations Engineer

Job DescriptionSenior Observability Operations EngineerWe are seeking a highly skilled Senior Observability Operations Engineer to manage and enhance our enterprise observability platform. The ideal candidate will have deep expertise in Dynatrace, Splunk, OpenSearch/Elasticsearch, Kubernetes, Linux, and cloud-native observability solutions. Experience leveraging AI/ML and Generative AI to improve observability, automate operations, and accelerate incident resolution is highly desirable.The role is responsible for ensuring high availability, scalability, operational excellence, and continuous improvement of enterprise monitoring and logging platforms supporting mission-critical applications.Key ResponsibilitiesAdminister and optimize enterprise observability platforms including Dynatrace, Splunk, and OpenSearch/Elasticsearch.Design, deploy, configure, and maintain monitoring, logging, tracing, and alerting solutions.Manage large-scale OpenSearch/Elasticsearch clusters, including indexing strategies, performance tuning, shard optimization, backups, and capacity planning.Configure Dynatrace OneAgent, ActiveGate, Synthetic Monitoring, Real User Monitoring (RUM), Digital Experience Monitoring (DEM), Davis AI, and Application Performance Monitoring (APM).Administer Splunk Enterprise, Universal Forwarders, Indexers, Search Heads, Cluster Manager, Deployment Server, and Splunk ITSI.Develop dashboards, alerts, reports, and executive operational metrics.Support Linux-based infrastructure and Kubernetes environments (Docker/OpenShift/Rancher preferred).Implement observability best practices using OpenTelemetry, distributed tracing, metrics, logs, and events.Perform root cause analysis for production incidents using observability platforms.Collaborate with Platform Engineering, SRE, DevOps, Infrastructure, and Application teams.Automate operational tasks using Python, Shell scripting, REST APIs, Terraform, or Ansible.Participate in incident, problem, change, and release management processes.Drive platform upgrades, patching, security compliance, and operational governance.Improve platform reliability through automation, self-healing, and AI-assisted operations.Required Technical SkillsObservability PlatformsDynatrace AdministrationSplunk Enterprise AdministrationOpenSearch AdministrationElasticsearch AdministrationGrafanaPrometheusKibanaJaegerOpenTelemetryKafka (preferred)InfrastructureLinux AdministrationKubernetesDockerOpenShift or RancherNetworking (TCP/IP, DNS, Load Balancers, Firewalls)System AdministrationCloud & DevOpsAWS, Azure, or GCPCI/CD pipelinesGitTerraformAnsibleREST APIsScriptingPythonBash/ShellPowerShell (preferred)AI & Automation Skills (Preferred)Experience using Generative AI (ChatGPT, GitHub Copilot, Amazon Q, Microsoft Copilot, or similar) to improve operational efficiency.Knowledge of AIOps platforms and AI-driven observability.Experience with Dynatrace Davis AI for anomaly detection and root cause analysis.Understanding of machine learning concepts for predictive monitoring and intelligent alertingExperience building AI-assisted operational runbooks and troubleshooting workflowsKnowledge of Retrieval-Augmented Generation (RAG), vector databases, embeddings, and AI powered knowledge search is a plusExperience integrating AI with observability platforms using APIsFamiliarity with LLM's, prompt engineering and AI assisted automationExperience using Python with AI frameworks (LangChain, LangGraph, Open AI APIs or simialr) is desirableExposure to AI driven incident summarization, log analysis and automated ticket enrichmentRequired QualificationsBachelor's degree in Computer Science, Information Technology, Engineering or equivalent experience6-10+ years of IT infrastructure or observability operations experience4+ years administering Dynatrace, Splunk, OpenSearch or ElasticSearchStrong Linux system admin experienceExperience supporting enterprise scale production environments Strong troubleshooting and analytical skillsExcellent communication and stakeholder management skillsPreferred CertificationsDynatrace Associate or Professional CertificationSplunk Enterprise Certified AdministratorElastic Certified EngineerKubernetes (CKA/CKAD)AWS/Azure/GCP CertificationITIL FoundationAI/ML or Gen AI certification