Two programs can post the same win rate this year and end up in entirely different places next year. One gets faster and sharper with every quarter. The other runs the same tests, on the same kinds of pages, discovering the same things, indefinitely. The difference has nothing to do with test quality, traffic volume, or how good either team is at writing a hypothesis. It’s what happens to a result after the test ends — the difference between a program that treats testing as a series of one-off wins and one that runs a genuinely continuous CRO strategy. This article assumes you already know the six-stage loop this program runs on. If you need that definition first, Optimization Intelligence is the hub for it. What follows is the part that isn’t in the framework diagram: the specific, named ways a program’s loop breaks in practice, and what actually has to exist for it to close.
Key takeaways
- Win rate measures what happened in a test. Compounding measures what the organization kept from it. A program can be excellent at the first and still fail at the second.
- “Test and learn” is really a six-link chain: Evidence, a stated Decision Rule, a Repository, Distribution, the Next Decision it should change, and an Outcome that becomes new Evidence. Most programs run link one fine and never build the other five.
- Three failure modes account for most stalled loops, mapped to exactly where the chain snaps: evidence never reaches a repository, a repository entry never reaches distribution, or a distributed result never reaches the next planning decision.
- Plateau isn’t always evidence of a broken loop. A program too young to have banked enough results, or one that’s already captured its highest-leverage wins, will look flat for reasons no loop fix touches.
- The loop health check at the end is five questions a team can answer this week without buying any tooling.
Why do two programs with the same win rate compound differently?
Because a win rate describes the test, and compounding describes what happens to the result after the test is over. A program that wins a healthy share of its tests and writes every result into a place the whole team consults before planning gets measurably smarter each quarter: fewer duplicate tests, faster hypothesis generation, a shrinking list of “things we already know don’t work.” A program with that identical win rate that lets each result live only in the tester’s head, a Slack thread, or a slide nobody reopens starts every quarter close to where it started the last one.
This is the gap most programs don’t notice, because the visible metric (win rate, number of tests shipped, uplift reported) looks fine in both cases. The compounding failure is invisible on a dashboard. It shows up as a slower, subtler symptom: the team keeps discovering things it already knew, keeps re-litigating decisions someone already settled, and keeps building next quarter’s roadmap from scratch instead of from evidence sitting one search away.
What breaks the loop most often?
Three failure modes, and they tend to happen in this order as a program matures, not simultaneously. A young program usually fails at the first one. A program that survives a year or two graduates to failing at the second or third instead, which is a subtler and more expensive version of the same underlying problem.
Failure mode 1: the result never gets written down
This is the most common failure and the easiest to fix, which is exactly why it’s worth naming first. A test finishes, someone reports the headline number in a meeting or a Slack message, and that’s the only record of it. The evidence never gets converted into a stated decision rule, and neither survives past the meeting where it was mentioned. Six months later, someone proposes testing almost the identical hypothesis on a different page, because nobody remembers, or nobody can find, that the question was already answered.
The symptom is specific and checkable: ask your team to find last quarter’s third test result without asking a person who remembers it. If the honest answer involves scrolling a chat history or guessing which deck it’s in, this is the failure mode you have.
Failure mode 2: the result gets written down, but never leaves the testing team
This is the failure mode that shows up once a program fixes the first one and starts feeling mature. The results exist. They’re in a doc, a dashboard, a wiki page, somewhere real. But the only people who ever open that page are the people who ran the tests. Product decides a new feature’s default behavior without checking whether a test already touched that exact interaction. Marketing writes new landing-page copy that repeats a claim a test already showed underperforms. Nobody did anything wrong; the information simply never reached them.
The symptom: a decision gets made elsewhere in the business that directly contradicts a result your program already has, and the people who made that decision genuinely didn’t know the result existed. That’s not a communication failure in the abstract; it’s a specific, traceable instance of a result staying contained to the team that produced it.
Failure mode 3: the result exists and is known, but never gets revisited when planning the next quarter
This is the quietest and most expensive failure, because from the outside the program looks functional. Results are documented. People know they exist. But the planning process for the next quarter’s test roadmap starts from a blank page (new ideas, a fresh backlog, a prioritization session) without anyone opening the results log first. A log that exists but is never consulted before planning behaves, in practice, the same as a log that was never created.
The symptom: your team can produce a documented result the moment you ask for it, but the next quarter’s roadmap was built without opening that document. Ask where the last planning session started from. If the honest answer is “a brainstorm,” not “last quarter’s results plus what they ruled out,” this is the failure mode running underneath an otherwise well-documented program.
| Failure mode | What’s missing | Checkable symptom |
|---|---|---|
| Not written down | A durable, searchable record | Can’t find last quarter’s result without asking a person |
| Not shared beyond the testing team | A real distribution point | Another team contradicts a known result, unaware it exists |
| Not revisited at planning time | A standing consultation step | Next quarter’s roadmap starts from a blank page, not the log |
What’s the actual CRO learning loop, beyond “test and learn”?
“Test and learn” describes an intention. The mechanism underneath it is six specific links, and a program only compounds when all six are actually built. None require new software. They require someone to own each link as a real step in the process, not an aspiration.
- Evidence. The test concludes with a number: it won, lost, or landed flat, on what metric, with what confidence.
- Decision Rule. Evidence alone isn’t a rule. Someone converts “the shorter checkout form won” into a stated belief the program will act on: “below three fields, checkout completion improves on mobile; above three, it doesn’t move.” This link is the one most programs skip.
- Repository. The rule and its evidence get one searchable entry: what was tested, what happened, what rule it produced, what it implies for related pages.
- Distribution. The rule reaches product, marketing, and whoever else makes customer-facing decisions, on a predictable cadence, not only when someone happens to ask.
- Next Decision. The rule visibly changes what the program builds or tests next: a roadmap item exists because the rule ruled something in, or gets cut because the rule ruled it out.
- Outcome. The next decision, once shipped or tested, produces its own result, which becomes new Evidence, and the loop runs again.
A program that closes all six links gets faster and more specific every cycle, because each pass narrows the set of untested assumptions instead of restarting from zero.
Where the chain snaps
The three failure modes above aren’t a separate list. They’re the three places this chain most often breaks: between Evidence and Repository (mode one, nothing gets written down), between Repository and Distribution (mode two, the record exists but stays inside the testing team), and between Distribution and Next Decision (mode three, the record is known but never consulted at planning time). A program can have a flawless repository and still plateau if link five never fires, because the six links have to hold end to end, not just the ones a team happens to have built already. That’s also why fixing mode one first matters operationally, not just as a sequencing preference: links three through six have nothing to distribute, consult, or act on until link two actually produces a rule worth recording.
A worked example (hypothetical)
Picture a mid-sized B2B SaaS company’s optimization program, eighteen months in, win rate holding steady and flat anyway. A pricing-page test that quarter finds that a three-field signup form beats a six-field one by a solid margin on mobile. The team ships the winner, reports the lift in the quarterly review, and moves to the next test. Two quarters later, a different tester, working on a different page, proposes testing form length on the trial-signup flow, unaware the question already has an answer one page over. The test reruns, finds the same result, and gets reported as a new win. The program’s win-rate dashboard shows two successes. What it doesn’t show is that the second test produced zero new information: it re-spent a quarter’s traffic confirming a rule the program already had, because the rule was never written down as a rule (link 2), only reported as a one-time result.
What the fix looked like
The fix wasn’t a new tool. It was making link 2 mandatory: every test write-up now ends with one line stating the decision rule in a form usable elsewhere (“forms above three required fields lose completion on mobile across signup-style flows, not just pricing”), and that line goes into a repository entry tagged by flow type, not just by page. Six months after adding that one step, a different team proposing a checkout redesign found the pricing-page rule in under a minute, applied it directly, and spent that quarter’s traffic on a question that didn’t already have an answer. The win rate didn’t change. What changed was how much of each quarter’s testing budget went toward genuinely new evidence instead of re-confirming an old rule that was never given a durable, reusable form. (HYPOTHETICAL.)
Loop health check
Answer these honestly, without checking a dashboard first. The honest gut answer is the diagnostic:
- Can someone who wasn’t on the testing team find last quarter’s three biggest test results in under two minutes, unassisted?
- Has a decision been made elsewhere in the business in the last quarter that a known test result would have changed, had the decision-maker seen it?
- Did the most recent planning session for new tests start by reviewing prior results, or by generating new ideas?
- If you removed the person who ran your most important test last quarter, would the result survive in a form anyone else could act on?
- Can you name one test from a year ago whose result changed a decision made this quarter? If not, the loop isn’t closing yet, whatever the win rate says.
A program failing question 1 has failure mode one. A program passing 1 but failing 2 has failure mode two. A program passing both but failing 3 or 5 has failure mode three, the quietest one, and the one most worth checking for in a program that otherwise looks mature.
When plateau isn’t a broken loop
Not every flat quarter is a loop problem, and treating one as if it were wastes effort chasing process fixes that won’t move the number. Three situations look identical to a broken loop from the outside and aren’t:
- The program is too young to have plateaued yet. Compounding needs enough closed cycles to compare against: a handful of tests in the first two or three months isn’t a plateau, it’s a program that hasn’t run long enough to have a trend. If you can’t yet point to a full year of results, the honest read on a slow quarter is “early,” not “broken.”
- The program already captured its real high-leverage wins. A page with three obviously broken friction points gets three big wins in its first year. Once those are fixed, the remaining tests are legitimately smaller (different button copy, minor layout shifts), and a lower average lift per test is the correct, healthy outcome of having already fixed the expensive problems, not a sign the loop stopped working.
- The constraint is upstream of testing entirely. Traffic too low to reach significance in a reasonable window, a product or offer with a real market-fit problem, or a page nobody actually visits: none of these are fixed by a better decision-rule repository. Run the loop health check first; if it passes clean and the plateau persists, the problem is almost certainly outside what a learning loop can touch.
If any of these describe the situation, the fix isn’t a repository, a distribution channel, or a planning-agenda change. It’s recognizing the program is doing exactly what it should be doing given real constraints, and redirecting effort toward the constraint itself (more traffic, a different page, patience) instead of the loop.
Where to start this week
- Run the five-question loop health check above with your actual team, out loud, not as a self-assessment doc filled in alone. The answers are usually more honest said aloud.
- Pick your single most recent test result and trace where it currently lives. If that takes more than two minutes or requires asking a specific person, that’s your program’s actual failure mode, not the one you’d have guessed.
- Before the next planning session, add one fixed agenda item: “what do we already know from the results log.” Do it once, deliberately, before deciding whether it becomes permanent.
- If the loop is failing at mode one (nothing written down), fix that before anything else. Modes two and three are unreachable if there’s no record to share or revisit yet.
- If none of the failure modes fit, check the boundary section above before assuming the process is broken.
Alex’s Perspective
Across the 100+ optimization and experience programs I’ve been part of, the ones that plateaued almost never plateaued from a bad idea or a losing streak. They plateaued because a result that took real effort to produce (traffic, time, a contained risk) got treated as disposable the moment the test ended. Across 7,000+ experiments and personalization experiences, the single clearest predictor of whether a program was still compounding two years in wasn’t win rate. It was whether anyone could find last year’s results without asking me.
In the framework
This sits primarily at Learn, with a direct hit on Prioritize: a program that isn’t closing its loop at Learn is, by definition, prioritizing its next quarter blind to what the prior one already proved. It assumes you already have the six-stage framework loop from Optimization Intelligence and the decision-system foundation; this piece is the operating detail underneath that loop’s “Learn” and “Scale feeds Discover” links.
Ready to see where your own loop is actually breaking? Start with a conversion audit and we’ll trace it together.
Alex Harris leads AlexDesigns’ conversion optimization practice, where closing this exact loop, not just running more tests, is the difference between a retainer that compounds and one that plateaus at month six.
