Prove what worked before you scale it.
I don’t call a test a win until it survives a cross-check. A result earns trust in steps, not in one dashboard screen, and that same discipline is what tells me when the audience is wrong, or when a rule from one program belongs in another.
- 1A result that held upSeven experiments, one cleared the confidence bar. The rest weren’t claimed.
- 2An audience that turned out wrongA growth list, checked against membership data before anyone personalized anything.
- 3A rule that outlived the testOne program’s decision rule became the standard a university’s enrollment leadership still uses.
What value means
A renewal doesn’t happen because of one percentage.
The fabrics result below is a real lift. The other two stories on this page aren’t percentages at all, and they mattered just as much to the people running the program.
Measurement
We can trust the result, because it survived a cross-check before anyone acted on it.
Operating
The team runs the process on its own, the same way each time.
Customer data
We didn’t personalize against the wrong audience, because we checked first.
Governance
Two different teams read a result the same way, because the decision rule is written down.
Business
A behavior or conversion changed, and the result cleared the confidence bar.
Program
What one team learned became reusable by a team that never saw the original test.
The result
Partner clicks moved from 2.8% to 8.4%.
A B2B brand learning to route people between discovery and buying.
- The context
- A B2B manufacturer was building direct-to-consumer experiences and needed to send visitors between discovery and buying through its partners.
- The result
- In the program’s first quarter the team ran seven experiments. One moved partner clicks from 2.8% to 8.4%, a 204.5% lift at better than 99% confidence.
- My role
- I led the new Optimizely program and presented the quarterly review. The result belongs to the team.
- How to read it
- Only the partner-click result cleared the confidence bar. Other lifts from the same quarter came from small samples, so they aren’t claimed here.
The audience
We stopped the personalization before it reached the wrong audience.
The growth list was the wrong audience.
- Original assumptionAcquisition: grow membership from this list
- Customer dataCheck the list against membership records first
- The assumption failsNone of the dated profiles were current members
- New decisionWin-back: different message, different success measure
- The problem
- A membership-growth campaign was planned around an audience list, with personalization ready to go.
- What changed
- Before we personalized anything, I checked the growth list against membership data. None of the dated profiles were current members. The campaign became a win-back campaign.
- What it prevented
- A personalized growth campaign aimed at people who weren’t members.
What I did
- Checked the list against membership and customer-data records
- Flagged bounced profiles the customer data platform still counted as reachable
- Flagged consent that was being treated as implied
- Reframed the campaign, its message and its success measure before launch
Trust
I don’t trust an experiment because the dashboard says “winner.”
A result earns trust in steps: an A/A check when the program is new, a cross-check against an independent analytics tool, and guardrail metrics so a win on clicks can’t hide a loss on applications or sales. That’s what lets experimentation, customer data and measurement work as one system instead of three separate opinions.
- ExposureWho actually saw the change
- Optimizely resultThe testing tool’s answer
- Analytics cross-checkDoes an independent system agree?
- Business outcomeDownstream: applications, accounts, sometimes offline sales
- DecisionShip, iterate or stop, written down
Reuse
Fix the winner rule before running the next test.
The fabrics result above and the association call above both fed the same kind of thing: a written rule for how the next decision gets made. A large private university took that habit and pointed it at its entire backlog, not one test at a time.
I reviewed roughly a hundred concluded experiments, some from an older tool and some on Optimizely, and rewrote the decision rules. Enrollment leadership adopted them. A rule written down for one decision outlives the person who wrote it and travels to teams who never saw the original test. The full governance story is told on the Onboarding & Enablement page.
The next step isn’t adding AI to the platform. It’s deciding which work should become agentic.
The next form of scale isn’t more manual work.
Optimizely and related product names are trademarks of their owner. This section reflects my own professional experience with the platform. It is not affiliated with or endorsed by Optimizely, and client examples are described by industry, not by name.