When Your Conversion Program Stalls: How to Run an Agency Replacement Review Without Guessing

Signals to watch

flat conversion curve after two quarters
repeated development delays on experiment builds
traffic allocation disputes between agency and internal team

Composite industry case. This page describes a reusable operating problem and decision method. It does not represent a named customer, real conversation, contract, revenue result or testimonial.

The Program That Stopped Producing Answers

You are six months into a conversion optimization program with an external agency. The first batch of experiments moved the needle. Then the curve flattened. Now the pipeline is full of tests that either run for weeks without reaching significance or get cancelled mid-flight because development took too long and the traffic window closed.

Your internal stakeholders are asking whether the agency should be replaced. But you cannot give a clean answer because four different explanations are tangled together:

Experiment quality: Are these bad hypotheses or bad execution?
Development dependency: Does the agency wait too long for engineering, or does engineering block them?
Traffic conditions: Is the site getting enough visitors to reach significance in a reasonable window?
Business outcome: Even if an experiment wins, does the lift translate to revenue or just vanity metrics?

Each thread alone is actionable. Together they paralyze decision-making. The common instinct is to escalate to a bake-off, an RFP process, or a leadership pitch to swap vendors. But all of those actions skip the diagnostic step that would tell you whether the agency is the problem at all.

Why Teams Misread a Stalled Program

The trap is attribution by chronology. When a program shows early wins then stalls, the natural narrative is that the agency coasted after the first deliverables. But that story ignores three structural confounds that appear in nearly every multi-quarter conversion program.

First, experiment inventory shifts over time. Early tests target low-hanging fruit — button colors, headline variants, layout changes — where effect sizes are large and sample sizes small. Later tests target deeper funnel mechanics or checkout flows, where effect sizes shrink and required sample sizes grow. The same agency running the same methodology will appear to slow down because the tests themselves are harder.

Second, traffic is never stationary. Seasonal dips, paid media budget changes, organic algorithm shifts, and site-wide performance updates all change the volume and composition of visitors. An agency that delivered a winning test in Q4 with high traffic and strong purchase intent may deliver an inconclusive test in Q1 with low traffic and browsing traffic. The agency did not change. The environment did.

Third, development dependency decouples agency output from agency quality. When an agency submits a spec and waits three weeks for engineering to build it, the elapsed calendar time is not a reflection of agency effort. But on a dashboard, both look like “slow progress.”

An Evidence Review Framework for Growth Operations Leads

When the four threads cannot be evaluated separately, the fix is not to try harder — it is to impose a structured review protocol that surfaces the dominant bottleneck. Use the following three-document review before any decision to replace or retain an agency.

Document 1: Experiment Quality Audit

Pull the last six experiments that reached a conclusion. For each one, answer three questions:

Was there a written hypothesis before the test launched? (Not a description — a directional prediction with a rationale.)
Was the minimum detectable effect specified before the test started?
Did the experiment report include the starting sample size, the actual sample collected, and the statistical power at close?

If two or more experiments fail the first question, the program has a hypothesis-quality problem. That problem may sit with the agency (if they design the experiments) or with your team (if you write the briefs). Assign ownership accordingly.

Document 2: Timeline Decomposition

Map every experiment from brief to decision on a simple calendar. For each one, note:

Date the brief was handed to the agency
Date the agency returned the spec
Date development started
Date development completed
Date the experiment launched
Date it reached a decision

Look for the widest gaps. If the gap consistently sits between spec-ready and dev-start, the bottleneck is engineering capacity, not agency work. If the gap sits between brief and spec, the bottleneck is the agency’s design velocity.

Document 3: Traffic Context Log

For each experiment in the same batch, record:

The traffic volume during the run period (sessions or unique visitors per day)
Whether the period coincided with a known traffic event (holiday, campaign, algorithm update)
Whether the experiment was originally designed for a different traffic tier

If experiments designed for high-traffic periods were run during low-traffic periods due to development delays, the traffic and dependency problems are linked — and neither is the agency’s fault.

Your Next Step: Produce a Human Review Action

After the three documents are assembled, write a single-page brief with exactly four sections:

Dominant bottleneck — one sentence identifying whether the primary constraint is hypothesis quality, engineering capacity, traffic conditions, or metric definition.
Owner of the bottleneck — one named person or team (agency lead, your engineering manager, your growth director, or the agency’s optimization lead).
Evidence summary — three data points from the documents above that support the diagnosis.
Decision window — a calendar date by which a fix must produce a measurable change, or a different action (retain with new scope, replace agency, pause program, restructure internally) is triggered.

This brief is not a dashboard. It is a human judgment document. It forces the team to pick one diagnosis and one owner, rather than continuing to monitor all four signals indefinitely.

What Automation Cannot Replace — and Where It Helps

No tool can decide whether to fire an agency. That decision requires reading the nuance in a hypothesis statement, understanding organizational context, and weighing the cost of switching against the cost of staying.

But automation changes what you need to read. Continuous signal discovery — surfacing experiments that ran without a hypothesis, tests that closed below statistical power, or specs that sat in queue for longer than a threshold — turns the evidence review from a manual archaeology project into a triage session. You still make the call. You just stop spending three days gathering the documents.

The same principle applies to evidence organization. When signals are structured by dimension (quality, dependency, traffic, outcome) and surfaced in a human-readable timeline, the bottleneck reveals itself without a statistical model. The framework works the same way whether you use a spreadsheet, a project management tool, or a dedicated orchestration layer. The method comes first. The tool shortens the cycle.

Frequently asked questions

How do I distinguish a bad agency from a bad brief?

Compare the quality of the experiment design brief the agency received against the experiments they delivered. If the briefs lack a hypothesis, sample size estimate, and success metric, the issue starts upstream.

What is the fastest signal that an agency is misreading traffic conditions?

If the agency runs a prioritized experiment during a known low-traffic period (holiday, off-season, algorithm update) and the report does not mention the traffic context, that single pattern strongly suggests the traffic dimension is being ignored.

Does replacing an agency automatically fix a stalled program?

No. If the problem is internal capacity, unclear ownership, or insufficient traffic for statistically valid tests, a new agency inherits the same constraints. The review must rule out these causes first.