Multi-agent systems fail quietly. A planner agent starts producing slightly vaguer task decompositions after a prompt tweak, a retrieval agent’s recall drops because a knowledge base grew past its chunking assumptions, a tool-calling agent begins hallucinating parameters for an API that changed its schema. None of these failures throws an exception. Each one degrades output quality in ways that only become visible when a user complains — or worse, when they don’t complain and simply leave.
This is why evaluation cannot be a one-time exercise you run before launch. It has to be a pipeline: automated, versioned, and wired into every deployment. Replit turns out to be a surprisingly good home for this kind of pipeline, because the platform collapses the distance between your agent code, your evaluation harness, your database, and your scheduled jobs into a single workspace. This article walks through how to build that pipeline end to end.
Why multi-agent systems need a different evaluation approach
A single-agent chatbot can be evaluated on one axis: given an input, was the output good? Multi-agent systems break this assumption in three ways.
First, quality is trajectory-dependent. The final answer might be correct even though the orchestrator routed the query to the wrong specialist and the system recovered by luck. Evaluating only final outputs hides brittleness that will surface under load. You need to evaluate the path, not just the destination — was the routing decision correct, did each agent receive the context it needed, were tool calls well-formed, and did handoffs preserve state?
Second, failures compound. If your planner is 95% accurate and each of three downstream agents is 95% accurate, your end-to-end reliability is closer to 81% than 95%. This means per-agent evaluation and system-level evaluation are both necessary, and you need to know which layer regressed when the composite score drops.
Third, the system changes continuously even when your code doesn’t. Model providers update models, knowledge bases grow, user query distributions drift. A pipeline that ran green in March can be quietly failing in July with zero code changes. Continuous evaluation is the only defense.
The architecture at a glance
The pipeline has five stages, all of which can live inside a single Replit workspace or a small set of connected Repls:
Instrumentation captures every agent interaction as structured traces. Dataset management maintains golden datasets and continuously harvests new test cases from production. Evaluation execution runs scorers — deterministic checks, statistical metrics, and LLM-as-judge — against traces and datasets. Gating blocks deployments when scores regress past thresholds. Monitoring runs the same evaluators on sampled production traffic on a schedule, so drift is caught between releases.
On Replit specifically, the natural mapping is: your agents run as a Reserved VM or Autoscale Deployment, traces land in Replit’s built-in PostgreSQL database, evaluation runs execute as Scheduled Deployments (cron jobs), and regression gates run in your deployment workflow before promotion. Secrets for judge-model API keys live in Replit Secrets, and a small dashboard Repl can visualize score trends.
Stage 1: Instrument every agent hop
You cannot evaluate what you cannot see. Every message between agents, every tool call, every retrieval, and every model completion needs to be logged as a span within a trace, with a shared trace ID tying the whole task together.
A minimal tracing layer in Python looks like this:
import time, uuid, json
import psycopg2
from contextlib import contextmanager
class Tracer:
def __init__(self, db_url):
self.conn = psycopg2.connect(db_url) # Replit PostgreSQL URL from env
@contextmanager
def span(self, trace_id, agent, step_type, inputs):
span_id = str(uuid.uuid4())
start = time.time()
record = {"trace_id": trace_id, "span_id": span_id,
"agent": agent, "step_type": step_type,
"inputs": inputs, "outputs": None, "error": None}
try:
yield record
except Exception as e:
record["error"] = str(e)
raise
finally:
record["latency_ms"] = int((time.time() - start) * 1000)
self._write(record)
def _write(self, r):
with self.conn.cursor() as cur:
cur.execute(
"""INSERT INTO agent_spans
(trace_id, span_id, agent, step_type, inputs, outputs, error, latency_ms)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s)""",
(r["trace_id"], r["span_id"], r["agent"], r["step_type"],
json.dumps(r["inputs"]), json.dumps(r["outputs"]),
r["error"], r["latency_ms"]))
self.conn.commit()
Wrap every agent action in a span. For a RAG agent, log the query, the retrieved chunk IDs and scores, and the final generation separately — you will need that granularity later to distinguish retrieval failures from generation failures. For tool-calling agents, log the raw tool call JSON before execution so schema violations are visible even when the tool itself swallows the error.
If you prefer standards over homegrown code, OpenTelemetry with GenAI semantic conventions, or a framework like Langfuse or Arize Phoenix (both self-hostable, and Phoenix runs comfortably in a Replit workspace), gives you the same capability with better tooling. The principle is what matters: one trace per user task, one span per agent action, structured and queryable.
Stage 2: Build and grow your golden dataset
Your evaluation dataset is the most valuable asset in this pipeline, and it should live under version control right next to your code — a datasets/ directory of JSONL files works fine, with each case carrying an ID, the input, the expected behavior, and metadata tagging which agents it exercises.
Seed it three ways. Start with hand-written cases covering your core flows, including deliberately adversarial ones: ambiguous requests that test routing, inputs that should trigger clarification rather than action, and requests one agent should refuse and escalate. Add failure-derived cases: every production bug becomes a permanent regression test, the same discipline you’d apply to unit tests. Finally, harvest from production continuously — a scheduled job samples recent traces, an LLM clusters and deduplicates them against existing cases, and novel patterns get queued for human review before joining the golden set.
For multi-agent systems, define expectations at two levels per case. The system-level expectation describes the final outcome (“the response cites at least one retrieved document and answers the pricing question”). The trajectory-level expectation describes the path (“the orchestrator routes to the billing agent, which calls get_invoice exactly once with a valid customer ID”). Cases that only check final outputs will pass while your routing logic rots.
Stage 3: Layer your evaluators
No single scoring method is sufficient. A robust pipeline layers three kinds of evaluators, cheapest first.
Deterministic checks run on every trace and cost nothing: did the tool call validate against its JSON schema, did the response contain required citations, did the trajectory stay under the step budget, did latency stay within SLO, was the routing decision the expected one? In practice these catch the majority of regressions, and they are the only evaluators fast enough to run inline on all production traffic.
Statistical metrics handle the retrieval and generation quality layer. If your agents include RAG components, RAGAS-style metrics — faithfulness, answer relevancy, context precision and recall — are the standard vocabulary, and they run fine as a library inside your evaluation Repl. Track them per retrieval agent, not just system-wide, so you can tell which knowledge base or which retriever configuration regressed.
LLM-as-judge covers everything rubric-shaped: response quality, tone, adherence to task decomposition, whether an agent handoff preserved the user’s intent. Two disciplines make judges trustworthy rather than noisy. Use a rubric with explicit criteria and few-shot anchor examples, not a bare “rate this 1–10.” And calibrate the judge itself: maintain a small set of human-labeled traces, and periodically measure judge–human agreement. A judge that agrees with your human labels 90% of the time is a usable signal; one you’ve never calibrated is a random number generator with confidence.
A dedicated multi-agent evaluator worth adding is the trajectory judge — an LLM that reads the full span sequence and answers structured questions: was each routing decision justified, did any agent act on stale or missing context, were there redundant loops, did the system recover from tool errors gracefully? This is where multi-agent-specific failure modes (ping-ponging between agents, context loss at handoffs, one agent silently overriding another’s output) actually get caught.
TRAJECTORY_RUBRIC = """You are auditing a multi-agent execution trace.
For each dimension, answer PASS or FAIL with one sentence of justification.
1. ROUTING: Was every delegation to a sub-agent appropriate for the task?
2. CONTEXT: Did each agent receive the information it needed from prior steps?
3. EFFICIENCY: Were there redundant tool calls or agent loops?
4. RECOVERY: If any step errored, did the system handle it without corrupting state?
Respond only in JSON: {"routing": {...}, "context": {...}, "efficiency": {...}, "recovery": {...}}"""
Stage 4: Wire evaluation into deployment as a gate
Continuous evaluation earns its name when it blocks bad releases automatically. The flow on Replit: your evaluation harness is a script (run_evals.py) that loads the golden dataset, executes the full multi-agent system against each case in a staging configuration, applies all evaluator layers, and writes a scored run to the eval_runs table with the current git commit hash.
The gate is a comparison against the last accepted baseline. Define per-metric thresholds with different strictness: deterministic checks should be zero-tolerance (any schema-invalid tool call fails the gate), statistical metrics get a regression budget (faithfulness may not drop more than two points), and judge scores get both a floor and a regression budget. Fail the gate if any threshold breaks, and print exactly which cases regressed — a gate that says “score dropped” without naming the failing cases will get bypassed within a month because nobody can act on it.
Run this before promoting a deployment. A Replit workflow that chains run_evals.py && deploy gives you the gate; for teams, running the eval script in CI against a preview deployment before merging to main is the more disciplined version. Either way, the invariant is the same: no prompt change, model swap, or orchestration edit reaches production without a scored run attached to it. Prompt changes deserve the same rigor as code changes, because in an agent system, prompts are code.
Stage 5: Monitor production between releases
The final layer catches the failures that happen with no deployment at all. A Scheduled Deployment on Replit — effectively a managed cron job — runs every hour or every night, samples recent production traces from the spans table, applies the cheap deterministic checks to all of them and the LLM-judge evaluators to a random subset (judging everything gets expensive fast; 5–10% sampling with stratification by agent and task type is usually enough), and writes scores to a time series.
On top of that time series, alert on three things. Absolute threshold breaches: faithfulness below your floor, tool-call validity below 99%. Trend drift: a metric declining across several consecutive windows even while above the floor — this is your early warning for knowledge-base staleness and model-provider changes. And distribution shift in inputs: if the embedding distribution of incoming queries moves away from your golden dataset’s distribution, your evaluations are losing relevance and the dataset-harvesting loop from Stage 2 needs to catch up.
Close the loop by feeding flagged production traces back into the review queue. A trace that fails in production today should be a golden dataset case next week and a gate that prevents its recurrence forever after.
A note on cost and pragmatism
A common objection is that running LLM judges continuously is expensive. In practice the pipeline is cheap if you respect the layering: deterministic checks are free and catch most regressions; statistical metrics are compute-light; judges run on samples and on the golden set at deploy time, not on every production request. Use a smaller, faster model as the judge for routine scoring and reserve your strongest model for calibration runs and trajectory audits. On a system doing tens of thousands of tasks a day, a well-layered pipeline typically costs low single-digit percent of total inference spend — and it is the difference between knowing your system works and hoping it does.
Closing
The pattern generalizes beyond Replit, but the platform’s combination of always-on deployments, built-in PostgreSQL, scheduled jobs, and secrets management means the entire pipeline — agents, traces, datasets, evaluators, gates, and monitors — lives in one place that any team member can open and inspect. That legibility matters. Evaluation pipelines die when they live in a separate system nobody looks at. Put the scores next to the code, make the gate name the failing cases, and treat every production failure as a future test, and your multi-agent system stops degrading in silence.
