Principal Engineer, Observability

Firmus Technologies
Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific.
Founded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability.
At Firmus, we are unique in our approach. We design, build, and operate a new class of digital infrastructure – the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. For our customers, this approach allows us to make every watt count and deliver low-cost AI tokens globally.
Firmus AI Cloud
Our large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers.
It empowers developers, enterprises, educational institutions, and government users to train and deploy AI models with unmatched efficiency and cost savings. With an ever-growing suite of services and applications, we are committed to delivering a cloud experience that is market-leading, proprietary, and built to scale.
ROLE SUMMARY
Firmus Technologies is seeking a Principal Engineer, Observability to own how Firmus operations teams and customers see the AI factory as one system, from site power and cooling to the GPU and the workload running on it. You will own the telemetry standard every team emits to, the platform that collects, stores, and queries it, and the applications that operations teams and customers use. Every team should be able to build its own dashboards and alerts without waiting on yours. Every new site adds millions of samples per second, and the platform must keep up as Firmus grows to hundreds of thousands of GPUs in gigawatt-scale AI factories, without losing critical data or slowing queries.
KEY RESPONSIBILITIES
Architecture and decisions
Own the architecture of the observability platform across every data centre and cloud region, from collection and transport to storage, query, alerting, and the applications on top. Set a target architecture two to three years ahead and plan the steps from today's system to it.
Design the platform to grow tenfold without a redesign, through sharding, retention tiers, cardinality budgets, and capacity plans. Prove the headroom with load tests before each new site goes live.
Design around the same repeatable unit as the estate. Each site keeps observing itself when its link to the rest of Firmus is down, one global view spans every site, and new sites come on through the site deployment kit with no local fork.
Own the te