{"schemaVersion":"jobsearcher.job.v1","id":"fab9adf6e1e07c8cd6f60bc4","url":"https://jobsearcher.com/jobs/fab9adf6e1e07c8cd6f60bc4","canonicalUrl":"https://jobsearcher.com/jobs/fab9adf6e1e07c8cd6f60bc4","title":"Principal Software Engineer, Distributed Systems","description":"BackgroundMake your application after reading the following skill and qualification requirements for this position.We are hiring aStaff / Principal Site Reliability Engineer focused on distributed systems and platform reliability . This is not a traditional DevOps role.You will own the architecture and engineering practices that ensure Sigma workflows execute reliably even when individual components fail.Processes crash. Networks become unavailable. APIs time out. Workers hang. Messages may be delivered more than once. Databases experience contention. Kubernetes pods restart.Your job is to design Sigma so those conditions are expected, detected, and automatically recovered from.\r\nThe fundamental principle is simple:An accepted Sigma job may fail, but it should never disappear.You will work closely with engineering leadership to evolve Sigma's execution architecture, improve production reliability, establish observability standards, and eliminate classes of distributed-system failures before they affect customers.What You'll OwnDesign and improve the systems responsible for executing long-running and scheduled infrastructure workflows.\r\nBuild patterns for:\r\nDurable workflow execution\r\nIdempotent operations\r\nAutomatic retries and exponential backoff\r\nWorker leases and heartbeats\r\nStuck-job detection and recovery\r\nWorkflow timeouts and cancellation\r\nFailure isolation\r\nDead-letter handling\r\nCheckpointing and workflow resumption\r\nConcurrency control\r\nDistributed locking and lease management\r\nReconciliation loops\r\nGraceful degradation during dependency failures\r\nEvaluate and implement workflow technologies such asTemporalor equivalent systems where appropriate.Messaging and Event Infrastructure\r\nOwn and improve Sigma's asynchronous execution architecture.\r\nWork extensively with technologies such as:\r\nKafka\r\nDistributed consumers\r\nConsumer groups\r\nMessage delivery semantics\r\nPartitioning\r\nConsumer lag\r\nBackpressure\r\nRetry strategies\r\nEvent ordering\r\nDuplicate message handling\r\nPoison-message isolation\r\nEnsure event-processing failures cannot silently cause workflows to stop progressing.Database Reliability\r\nHelp design application patterns that remain reliable under high concurrency.\r\nDeeply understand and troubleshoot:\r\nPostgreSQL transactions\r\nRow and advisory locking\r\nDeadlocks\r\nLock contention\r\nTransaction isolation\r\nConnection pooling\r\nLong-running transactions\r\nQuery performance\r\nDatabase failover\r\nSchema and migration safety\r\nPartner with application engineers to identify and eliminate patterns that create production contention or reliability risks.Kubernetes and Platform Reliability\r\nImprove the reliability of Sigma deployments running in containerized environments.\r\nResponsibilities include:\r\nKubernetes workload architecture\r\nPod lifecycle and failure recovery\r\nResource limits and capacity planning\r\nAutoscaling\r\nHealth checks\r\nGraceful shutdown\r\nRolling deployments\r\nHigh availability\r\nInfrastructure-as-code\r\nDisaster recovery\r\nBackup and restore validation\r\nDesign systems that tolerate node, pod, process, and dependency failures without losing customer work.Observability\r\nMake the state of Sigma understandable in real time.\r\nBuild and improve:\r\nMetrics\r\nDistributed tracing\r\nStructured logging\r\nDashboards\r\nAlerting\r\nWorkflow-level telemetry\r\nKafka consumer monitoring\r\nDatabase performance monitoring\r\nWorker health monitoring\r\nQueue depth and backlog monitoring\r\nTechnologies may include:\r\nOpenTelemetry\r\nGrafana\r\nPrometheus\r\nSentry\r\nCloudWatch\r\nEngineers should be able to answer questions such as:\r\nWhy is this workflow still running?\r\nWhich worker owns this task?\r\nWhen did it last make progress?\r\nWhich dependency is causing the delay?\r\nDid this operation execute once or multiple times?\r\nCan the workflow safely resume?\r\nAre scheduled jobs starting when expected?Reliability Engineering\r\nHelp establish engineering practices expected of mature enterprise platforms.\r\nThis includes:\r\nService-level objectives and indicators\r\nError budgets\r\nCapacity planning\r\nLoad testing\r\nFailure-mode testing\r\nChaos testing\r\nIncident response\r\nRoot-cause analysis\r\nProduction readiness reviews\r\nRunbooks\r\nAutomated recovery\r\nDisaster-recovery testing\r\nMove Sigma from detecting incidents to preventing and automatically recovering from them.What We're Looking For\r\nYou have significant experience operating production distributed systems where reliability matters. xsgimln\r\nStrong candidates will have deep experience with several of the following:\r\nKubernetes\r\nKafka or similar event-streaming systems\r\nPostgreSQL\r\nDistributed systems\r\nAsynchronous worker architectures\r\nWorkflow orchestration\r\nTemporal, Cadence, Step Functions, Conductor, or similar systems\r\nPython backend systems\r\nCloud infrastructure\r\nInfrastructure-as-code\r\nObservability platforms\r\nHigh-availability architectures\r\nYou understand concepts such as:\r\nAt-least-once delivery\r\nIdempotency\r\nLeases and fencing tokens\r\nDistributed locks\r\nLeader election\r\nRetry semantics\r\nBackpressure\r\nEventual consistency\r\nTransaction boundaries\r\nFailure domains\r\nReconciliation\r\nSplit-brain scenarios\r\nDistributed tracing\r\nMost importantly, you have personally diagnosed and improved production systems experiencing real distributed-system failures.What Success Looks Like\r\nWithin your first several months, you will help Sigma establish an architecture where:\r\nScheduled workflows reliably begin when expected\r\nAccepted work cannot silently disappear\r\nWorker crashes do not result in lost workflows\r\nInfrastructure failures automatically recover where possible\r\nLong-running workflows can safely resume\r\nDuplicate execution is prevented or safely handled\r\nStuck workflows are automatically detected\r\nDatabase and messaging contention is visible before becoming an outage\r\nEngineers can trace a workflow from request through every execution step\r\nCustomer environments can scale without requiring manual infrastructure babysitting","company":"Sigma Automate","rawCompany":"sigma automate","city":"United","state":"PA","isRemote":false,"isActive":false,"createdAt":"2026-10-12T01:12:00.385Z","occupations":[{"code":"15-1299.08","title":"Computer Systems Engineers/Architects","slug":"computer-systems-engineers-architects"},{"code":"15-1252.00","title":"Software Developers","slug":"software-developers"},{"code":"15-1211.00","title":"Computer Systems Analysts","slug":"computer-systems-analysts"}],"industries":[{"code":"541512","title":"Computer Systems Design Services","slug":"computer-systems-design-services"},{"code":"513210","title":"Software Publishers","slug":"software-publishers"},{"code":"541511","title":"Custom Computer Programming Services","slug":"custom-computer-programming-services"}],"jobPosting":{"@context":"https://schema.org","@type":"JobPosting","title":"Principal Software Engineer, Distributed Systems","description":"BackgroundMake your application after reading the following skill and qualification requirements for this position.We are hiring aStaff / Principal Site Reliability Engineer focused on distributed systems and platform reliability . This is not a traditional DevOps role.You will own the architecture and engineering practices that ensure Sigma workflows execute reliably even when individual components fail.Processes crash. Networks become unavailable. APIs time out. Workers hang. Messages may be delivered more than once. Databases experience contention. Kubernetes pods restart.Your job is to design Sigma so those conditions are expected, detected, and automatically recovered from.\r\nThe fundamental principle is simple:An accepted Sigma job may fail, but it should never disappear.You will work closely with engineering leadership to evolve Sigma's execution architecture, improve production reliability, establish observability standards, and eliminate classes of distributed-system failures before they affect customers.What You'll OwnDesign and improve the systems responsible for executing long-running and scheduled infrastructure workflows.\r\nBuild patterns for:\r\nDurable workflow execution\r\nIdempotent operations\r\nAutomatic retries and exponential backoff\r\nWorker leases and heartbeats\r\nStuck-job detection and recovery\r\nWorkflow timeouts and cancellation\r\nFailure isolation\r\nDead-letter handling\r\nCheckpointing and workflow resumption\r\nConcurrency control\r\nDistributed locking and lease management\r\nReconciliation loops\r\nGraceful degradation during dependency failures\r\nEvaluate and implement workflow technologies such asTemporalor equivalent systems where appropriate.Messaging and Event Infrastructure\r\nOwn and improve Sigma's asynchronous execution architecture.\r\nWork extensively with technologies such as:\r\nKafka\r\nDistributed consumers\r\nConsumer groups\r\nMessage delivery semantics\r\nPartitioning\r\nConsumer lag\r\nBackpressure\r\nRetry strategies\r\nEvent ordering\r\nDuplicate message handling\r\nPoison-message isolation\r\nEnsure event-processing failures cannot silently cause workflows to stop progressing.Database Reliability\r\nHelp design application patterns that remain reliable under high concurrency.\r\nDeeply understand and troubleshoot:\r\nPostgreSQL transactions\r\nRow and advisory locking\r\nDeadlocks\r\nLock contention\r\nTransaction isolation\r\nConnection pooling\r\nLong-running transactions\r\nQuery performance\r\nDatabase failover\r\nSchema and migration safety\r\nPartner with application engineers to identify and eliminate patterns that create production contention or reliability risks.Kubernetes and Platform Reliability\r\nImprove the reliability of Sigma deployments running in containerized environments.\r\nResponsibilities include:\r\nKubernetes workload architecture\r\nPod lifecycle and failure recovery\r\nResource limits and capacity planning\r\nAutoscaling\r\nHealth checks\r\nGraceful shutdown\r\nRolling deployments\r\nHigh availability\r\nInfrastructure-as-code\r\nDisaster recovery\r\nBackup and restore validation\r\nDesign systems that tolerate node, pod, process, and dependency failures without losing customer work.Observability\r\nMake the state of Sigma understandable in real time.\r\nBuild and improve:\r\nMetrics\r\nDistributed tracing\r\nStructured logging\r\nDashboards\r\nAlerting\r\nWorkflow-level telemetry\r\nKafka consumer monitoring\r\nDatabase performance monitoring\r\nWorker health monitoring\r\nQueue depth and backlog monitoring\r\nTechnologies may include:\r\nOpenTelemetry\r\nGrafana\r\nPrometheus\r\nSentry\r\nCloudWatch\r\nEngineers should be able to answer questions such as:\r\nWhy is this workflow still running?\r\nWhich worker owns this task?\r\nWhen did it last make progress?\r\nWhich dependency is causing the delay?\r\nDid this operation execute once or multiple times?\r\nCan the workflow safely resume?\r\nAre scheduled jobs starting when expected?Reliability Engineering\r\nHelp establish engineering practices expected of mature enterprise platforms.\r\nThis includes:\r\nService-level objectives and indicators\r\nError budgets\r\nCapacity planning\r\nLoad testing\r\nFailure-mode testing\r\nChaos testing\r\nIncident response\r\nRoot-cause analysis\r\nProduction readiness reviews\r\nRunbooks\r\nAutomated recovery\r\nDisaster-recovery testing\r\nMove Sigma from detecting incidents to preventing and automatically recovering from them.What We're Looking For\r\nYou have significant experience operating production distributed systems where reliability matters. xsgimln\r\nStrong candidates will have deep experience with several of the following:\r\nKubernetes\r\nKafka or similar event-streaming systems\r\nPostgreSQL\r\nDistributed systems\r\nAsynchronous worker architectures\r\nWorkflow orchestration\r\nTemporal, Cadence, Step Functions, Conductor, or similar systems\r\nPython backend systems\r\nCloud infrastructure\r\nInfrastructure-as-code\r\nObservability platforms\r\nHigh-availability architectures\r\nYou understand concepts such as:\r\nAt-least-once delivery\r\nIdempotency\r\nLeases and fencing tokens\r\nDistributed locks\r\nLeader election\r\nRetry semantics\r\nBackpressure\r\nEventual consistency\r\nTransaction boundaries\r\nFailure domains\r\nReconciliation\r\nSplit-brain scenarios\r\nDistributed tracing\r\nMost importantly, you have personally diagnosed and improved production systems experiencing real distributed-system failures.What Success Looks Like\r\nWithin your first several months, you will help Sigma establish an architecture where:\r\nScheduled workflows reliably begin when expected\r\nAccepted work cannot silently disappear\r\nWorker crashes do not result in lost workflows\r\nInfrastructure failures automatically recover where possible\r\nLong-running workflows can safely resume\r\nDuplicate execution is prevented or safely handled\r\nStuck workflows are automatically detected\r\nDatabase and messaging contention is visible before becoming an outage\r\nEngineers can trace a workflow from request through every execution step\r\nCustomer environments can scale without requiring manual infrastructure babysitting","datePosted":"2026-10-12T01:12:00.385Z","dateModified":"2026-10-12T01:12:00.385Z","hiringOrganization":{"@type":"Organization","name":"Sigma Automate","sameAs":"https://jobsearcher.com"},"jobLocation":{"@type":"Place","address":{"@type":"PostalAddress","addressLocality":"United","addressRegion":"PA","addressCountry":"US"}},"identifier":{"@type":"PropertyValue","name":"JobSearcher","value":"fab9adf6e1e07c8cd6f60bc4"},"url":"https://jobsearcher.com/jobs/fab9adf6e1e07c8cd6f60bc4"}}