Sr. Software Engineer - Distributed Systems
Overview:
In this role you will design, build, and enhance the observability platform that powers Workday's AI-enabled services. You'll work with a cross-functional team to ensure fast, reliable, and trustworthy production systems at massive scale. You'll apply AI-driven observability techniques to reduce noise and speed incident response, shaping how observability evolves for next-gen AI products. This is a hands-on, impact-focused role at a company committed to growth and collaboration. Compensation / Benefitsflexible hybrid work scheduleup to 50% remotestock grantsannual bonus plan or commission potentialopportunity for role-specific bonusesinclusive and growth-oriented environment
Responsibilities:
- Design, build, and improve observability services (Monitoring, Logging, Alerting, Tracing)
- Own end-to-end distributed tracing, instrument services, and analyze cross-boundary context
- Develop data capture/collection services across Kubernetes, Docker, OpenStack, bare metal
- Create core modules for real-time and batch data processing
- Build metrics ingestion pipelines and dashboard lifecycle automation
- Instrument AI-powered workflows for performance, cost, and quality
- Explore AI-driven observability techniques (anomaly detection, intelligent alerting, automated root-cause analysis)
- Collaborate with product/application teams to define SLOs/SLIs and production-readiness criteria
- Develop and deploy infrastructure components in production and ensure data quality/availability
- Evaluate new open-source/cloud-native tools as needed
- Participate in on-call rotation for the observability platform
Key requirements:
- 8 years of software engineering/design experience 7 years coding in Python, Go, or Java
- Experience with LinuxBS in Computer Science or related field, or equivalent experience
- Design/maintain/optimize Time-Series DBs, document stores, and log management systems
- Solid understanding of distributed tracing concepts and tooling
- Public cloud experience (AWS/GCP) with native observability tooling
- Experience with Docker, Kubernetes, Ansible, Chef, Terraform
- Experience with service mesh, Prometheus, and cloud-native technologies
- Hands-on with LLM orchestration frameworks (LangChain, LlamaIndex, Semantic Kernel) and agentic AI patterns
- Ability to build dashboards, alerts, and instrumentation for early AI-system signals
- Excellent interpersonal, technical, and communication skills
- Ability to prioritize multiple tasks in a fast-paced environmentMS Degree a plus
- Excellent interpersonal skills
- Strong communication abilities
- Ability to prioritize multiple tasks in a fast-paced environment
PythonGoJava