Principal Software Engineer, Distributed Systems

Background
We are hiring a Staff / Principal Site Reliability Engineer focused on distributed systems and platform reliability. This is not a traditional DevOps role.
You will own the architecture and engineering practices that ensure Sigma workflows execute reliably even when individual components fail.
Processes crash. Networks become unavailable. APIs time out. Workers hang. Messages may be delivered more than once. Databases experience contention. Kubernetes pods restart.
Your job is to design Sigma so those conditions are expected, detected, and automatically recovered from.
The fundamental principle is simple:
An accepted Sigma job may fail, but it should never disappear.
You will work closely with engineering leadership to evolve Sigma's execution architecture, improve production reliability, establish observability standards, and eliminate classes of distributed-system failures before they affect customers.
What You'll Own
Design and improve the systems responsible for executing long-running and scheduled infrastructure workflows.
Build patterns for:
Durable workflow execution
Idempotent operations
Automatic retries and exponential backoff
Worker leases and heartbeats
Stuck-job detection and recovery
Workflow timeouts and cancellation
Failure isolation
Dead-letter handling
Checkpointing and workflow resumption
Concurrency control
Distributed locking and lease management
Reconciliation loops
Graceful degradation during dependency failures
Evaluate and implement workflow technologies such as Temporal or equivalent systems where appropriate.
Messaging and Event Infrastructure
Own and improve Sigma's asynchronous execution architecture.
Work extensively with technologies such as:
Kafka
Distributed consumers
Consumer groups
Message delivery semantics
Partitioning
Consumer lag
Backpressure
Retry strategies
Event ordering
Duplicate message handling
Poison-message isolation
Ensure event-processing failures cannot silently cause workflows to stop progressing.
Database Reliability
Help design application patterns that remain reliable under high concurrency.
Deeply understand and troubleshoot:
PostgreSQL transactions
Row and advisory locking
Deadlocks
Lock contention
Transaction isolation
Connection pooling
Long-running transactions
Query performance
Database failover
Schema and migration safety
Partner with application engineers to identify and eliminate patterns that create production contention or reliability risks.
Kubernetes and Platform Reliability
Improve the reliability of Sigma deployments running in containerized environments.
Responsibilities include:
Kubernetes workload architecture
Pod lifecycle and failure recovery
Resource limits and capacity planning
Autoscaling
Health checks
Graceful shutdown
Rolling deployments
High availability
Infrastructure-as-code
Disaster recovery
Backup and restore validation
Design systems that tolerate node, pod, process, and dependency failures without losing customer work.
Observability
Make the state of Sigma understandable in real time.
Build and improve:
Metrics
Distributed tracing
Structured logging
Dashboards
Alerting
Workflow-level telemetry
Kafka consumer monitoring
Database performance monitoring
Worker health monitoring
Queue depth and backlog monitoring
Technologies may include:
OpenTelemetry
Grafana
Prometheus
Sentry
CloudWatch
Engineers should be able to answer questions such as:
Why is this workflow still running?
Which worker owns this task?
When did it last make progress?
Which dependency is causing the delay?
Did this operation execute once or multiple times?
Can the workflow safely resume?
Are scheduled jobs starting when expected?
Reliability Engineering
Help establish engineering practices expected of mature enterprise platforms.
This includes:
Service-level objectives and indicators
Error budgets
Capacity planning
Load testing
Failure-mode testing
Chaos testing
Incident response
Root-cause analysis
Production readiness reviews
Runbooks
Automated recovery
Disaster-recovery testing
Move Sigma from detecting incidents to preventing and automatically recovering from them.
What We're Looking For
You have significant experience operating production distributed systems where reliability matters.
Strong candidates will have deep experience with several of the following:
Kubernetes
Kafka or similar event-streaming systems
PostgreSQL
Distributed systems
Asynchronous worker architectures
Workflow orchestration
Temporal, Cadence, Step Functions, Conductor, or similar systems
Python backend systems
Cloud infrastructure
Infrastructure-as-code
Observability platforms
High-availability architectures
You understand concepts such as:
At-least-once delivery
Idempotency
Leases and fencing tokens
Distributed locks
Leader election
Retry semantics
Backpressure
Eventual consistency
Transaction boundaries
Failure domains
Reconciliation
Split-brain scenarios
Distributed tracing
Most importantly, you have personally diagnosed and improved production systems experiencing real distributed-system failures.
What Success Looks Like
Within your first several months, you will help Sigma establish an architecture where:
Scheduled workflows reliably begin when expected
Accepted work cannot silently disappear
Worker crashes do not result in lost workflows
Infrastructure failures automatically recover where possible
Long-running workflows can safely resume
Duplicate execution is prevented or safely handled
Stuck workflows are automatically detected
Database and messaging contention is visible before becoming an outage
Engineers can trace a workflow from request through every execution step
Customer environments can scale without requiring manual infrastructure babysitting

Work arrangement
Yes