The Agent Doing Our SRE's Root-Cause Work

An Alert Isn't an Answer

A monitoring webhook firing tells you something broke. It doesn't tell you what, why, or what to do about it.

That gap used to be a person's job. Something goes wrong on Railway — the platform JoyStream runs its own services on — and someone has to go pull the logs, pull the service metrics, cross-reference them against whatever the alert actually said, and piece together a diagnosis before anyone can act. It's the kind of work that's necessary and repetitive in exactly the proportions that make it a good candidate for an agent, as long as the agent can be trusted to do it right. So Vivek built one: a webhook comes in from the monitoring service, the agent decides which parts of Railway's systems it needs to query to get the full picture, gathers that context, and synthesizes a root cause and a recommendation — posted straight to Slack, with the right people and teams tagged.

The alert used to be where the human's job started. Now it's where the agent's job starts, and the human's job starts after the diagnosis is already sitting in Slack.

Parallel Where It Can Be, Chained Where It Has To Be

Not every alert needs every system checked.

The agent's first move is deciding which of Railway's systems are actually relevant to this specific failure, and querying them in parallel where it can — the dashed lines in Vivek's diagram aren't decoration, they're the agent choosing a conditional subset of lookups rather than running everything every time. Those parallel results all feed forward into a single diagnostics step, and that step can't fire until every lookup it depends on has actually finished. The handoffs between steps aren't vague either: a Slack thread ID gets passed from one step straight into a later one, so the final recommendation lands as a reply in the right thread instead of starting a new conversation nobody notices. Each step's input is a fixed, known shape — not a loose blob of context the next step has to guess at.

Parallel execution makes it fast. Fixed, chained inputs make it correct. Neither one works without the other.

Why It's Frozen, Not Freeform

Vivek's own reasoning for building it this way is blunt: "we can't risk this kind of diagnostic agent to go" off script.

That line is the actual thesis of the whole thing. A run of this agent isn't a script executing in some ad hoc order — it's a versioned epic made up of individual stories, visibly labeled v002 on screen, frozen the way a software release is frozen. Freezing it doesn't make it rigid in the way that sounds like a limitation. It makes every run inspectable after the fact: drill into any single step — step 12, say, "compose alert diagnostics" — and see exactly what went in and what came out, the same way you'd read a commit. Share a specific run with a teammate, and they're looking at precisely the context that matters, not a vague secondhand description of what happened.

An agent you can't inspect is a black box you're hoping behaves. An agent you can version is one you can actually trust to run again.

Forty-Five Minutes, JoyStream on JoyStream

Building this took about 45 minutes.

Most of that time wasn't spent on the interesting part — the webhook had to be coaxed into a shape the agent could actually work with, which is the kind of plumbing every integration eventually makes you do regardless of how smart the agent on the other end is. Once that was settled, the rest came together fast enough that Vivek had it running in simulation inside the hour. What's left afterward is JoyStream running JoyStream on itself: an agent doing the root-cause-analysis-and-recommendation work that used to sit entirely on one SRE's plate, built by the same person who'd otherwise be the one getting paged.

The alert still shows up at 3 a.m. It's just not a person doing the diagnosing anymore.