Multi-agent incident triage on LangGraph. Specialists investigate an alert, a proposer picks one action from a fixed allow-list, and nothing runs until a human approves, edits or rejects it.
An agent that can restart services needs a stop that no prompt can talk its way past. Here the allow-list lives in one module and every layer checks it again, and the pause for a human is a checkpoint on disk that survives the process being killed. The same investigation was rebuilt in CrewAI to compare the two frameworks from experience.
It is a portfolio project: four synthetic incidents, a deterministic fake model, and a dry-run adapter that only records what it would do. It has never been used on a real incident.
Plays the six stages below in order, about 55 seconds. Scroll or press Esc to stop.
Real output from the CLI on the bundled fixtures, captured for this page with the fake model. Some investigation lines are cut where marked. Proposal ids are random on every run. Illustrative input The thread ids, the reason and the approver name were typed for the capture.
A supervisor node calls the model with a route tool and sends one specialist at a time: a log investigator, a metrics investigator or a runbook agent. Each specialist is its own small LangGraph subgraph, with private messages and at most four tool rounds.
route call uses one too, so a model that never answers properly still stops.escalated and goes straight to the report. It never reaches the approval gate, so nothing is executed.LangGraph's own recursion limit is no guard here. Its docs say 1000; the installed source says 10007. The repo sets it to twice the step limit plus 10, just above its own check.
The same alert with 6 steps, then with 2
triage run on the high-latency fixture, with --max-steps 6 and --max-steps 2.The tools read fixture files that stand in for log and metrics backends. The LangGraph graph and the CrewAI port wrap the same functions, so both see the same evidence.
The fixed amount stops a tiny baseline from turning noise into an anomaly. Here the error rate peaked at 3.2 times its baseline and still didn't count, because it only rose by 0.44 points.
checkout-api, the hours before the alert
search_logs and detect_metric_anomalies, run on the high-latency fixture with the windows the investigators used.The runbook agent searches the markdown runbooks with TF-IDF, counting headings twice. It passes the alerting service, so only that service's runbook and the general policy are searched.
Every runbook section ends with a suggested action, and each incident's matching section suggests the expected answer. The repo says so, and its eval has a variant that hides those lines. Lexical search is the weak spot it admits: without the service filter, the right section for the ambiguous incident only narrowly wins.
The ambiguous incident: bursts of 5xx errors on orders-api
Queryorders-api 5xx error rate above 1% orders-api crossed a 1% 5xx rate several times in the last 30 minutes; each burst lasts about two minutes.
search_runbooks on the bundled runbooks, with the exact query the runbook agent sent. Top three results in each mode.The proposer returns a diagnosis and one action. Code then checks it against actions.py, whose first line reads “This module, not any prompt, decides what may run.”
page_human, and what the model asked for is recorded so the human sees the attempt.What the model asked for, and what the human is shown
restart_servicescale_uprollback_deploypage_humantests/scripted.py, run through the real graph on the high-latency and intermittent-5xx fixtures. The notes are the program's own text.The approval node calls interrupt() with the proposal and the allow-list. The run is saved to SQLite and the command exits. The decision can come hours later, from another shell, with triage resume.
The node accepts only a well-formed decision that names this proposal. Anything else ends the run as refused, with nothing executed. A rejected or refused run can't be resumed; a person starts a new one, which is why the CLI checks an edit before it sends it.
One paused proposal, six possible decisions
ALERT-1042, checkout-apiProposal 8bedf438cdebscale_up on checkout-api
dry_run_ledger.jsonlClearedEverything above reads fixture files and changes nothing. Below this line, an approved decision reaches the executor, and the executor only records what it would do.
The executor doesn't trust the route that led to it. authorize() re-checks the saved decision from scratch, and only then calls the dry-run adapter.
Nothing is restarted, scaled or rolled back: every method returns “would …” text. The gate also trusts the checkpoint file. Anyone who can write to it could forge a decision, because decisions aren't signed; the README lists this as a limitation.
After the gate, two ways it can go
triage run … --waitmemory-leak alert, proposal 9e269d82445f, then waits on stdinkill -9, exit status -9, no ledger filetriage resume crash --approvea new process on the same file12 calls, 8,688 in, 721 out: no step re-ranEach state is handed straight to executor(), with no gate in front of it. Illustrative input These are the forged states tests/test_approval.py uses.
tests/test_crash_resume.py: kill -9 while triage run --wait waits for a decision, then triage resume in a new process. Like the test, both ran as python -m triage_graph.cli with a temporary --state-dir, left out here. Model-call and token counts were read from the checkpoint before and after. The forged-state errors are the program's exceptions, verbatim.Every run ends with a markdown report built from the saved state: the outcome, the model, the steps, the human's decision, what the executor recorded, and every tool call with what it returned. This is the report from the Approve tab above. The log and runbook evidence is cut here; the full file has it.
The investigation and the proposal were rebuilt as a CrewAI crew, with the same fixtures, tools, prompts and allow-list check. The rows below quote the repo's written comparison. Both versions were only ever run on fake models, so this says nothing about how either behaves with a real one.
escalated outcome”“max_iter ends with a forced final answer and no signal; took a tool hook plus a ConditionalTask”interrupt() inside any node; resume with any JSON value”“Crews: a blocking input() review loop. Flows: @human_feedback with a pending state; resume with a string”“For this workflow, LangGraph. Its main requirements are a hard approval gate, a step limit with a clear outcome, and a pause that survives a crash, and in LangGraph those are ordinary code at points I choose.”
“CrewAI was quicker for describing agents and their tools, and delegation needed no routing code.”
Read the comparisonThe port proposed the same action as the LangGraph version on all four alerts, with 12 model calls each:
The repo's tests pin the restore problem: after a crash, the custom LLM came back as an OpenAI client and the conditional task as a plain task. They passed.
Running the port also wrote a third file the docs didn't list: a version cache, written after CrewAI asked pypi.org for its latest release. The port now turns that check off, as it does CrewAI's telemetry, and a test makes sure it stays off.
Every result in the repo comes from the fake model. No real-model run has been made, and the README says so in bold. These numbers were re-run for this page.
Four alerts, six runs each, in two variants: with the runbooks' suggested actions, and with them hidden from every agent.
That is by construction: the fake model's rules were written against these fixtures, and the eval report opens by saying so. Its tables matched the committed report exactly.
Estimated cost per run of the eval, against a $2 cap.
The line is $0.25, the most a run can cost for one run of each of the 8 cells to fit under $2. The planner fits 6, 2 and 1 runs per cell, and refuses the last. Without a key it prints the plan and stops. A callback refuses any call whose worst case could cross the cap.
Two places where LangGraph's docs and its installed source disagree, both re-checked in langgraph 1.2.12:
The default recursion limit is documented as 1000 and set to 10007 (_internal/_config.py:32).
Command(resume=None) crashes with UnboundLocalError, reported upstream as langgraph issue #7034. The runner refuses None before LangGraph sees it, and the thread stays paused.
The skipped test is the real-model eval. It spends money, so it runs only when both an API key and TRIAGE_EVAL_REAL=1 are set. The other 134 run on the fake model with no key.
CI runs ruff, the tests and the README quickstart on Ubuntu and macOS with Python 3.11, 3.12 and 3.13, with no secrets.
git clone https://github.com/ahmed-hashim-pro/triage-graph.git cd triage-graph uv sync # Investigates, then stops at the gate. No key needed. uv run triage run --alert fixtures/alerts/high-latency.json --thread-id demo # A separate process decides. uv run triage resume demo --approve uv run triage show demo
Try --reject --reason "..." or --edit restart_service instead, or --wait to decide in the same terminal. The CrewAI port needs uv sync --extra crewai, then uv run python -m crewai_port --alert fixtures/alerts/error-spike-after-deploy.json.