Prioritize A/B tests by expected value, instead who shouts loudest
Most teams think they need more test ideas. They don't. With limited traffic and a win rate that means most tests lose, the highest-leverage decision you make is which few tests to run. This guide gives you the four established frameworks, the five inputs that actually matter, and a scoring model you can run on your own backlog today.
Last reviewed: June 23, 2026
Backlog size is not your bottleneck
Most teams believe their problem is that they don't have enough test ideas. They are wrong. The bottleneck is almost never backlog size. It is evidence quality and expected value per unit of effort. Two facts make this concrete.
Win rates vary by definition and confidence threshold, but the shape is sobering. One e-commerce database reports roughly 36% of tests producing a winner, 22% producing a statistically significant loss, and 42% inconclusive. Other practitioners cite ranges as low as 12 to 15% significant winners and treat anything below a 20% win rate as a sign of weak hypothesis generation. The majority of what you ship will not move the metric.
To detect a real effect you need statistical power. The field treats about 80% power and 95% confidence as the floor, which sets a minimum sample per variation. A common rule of thumb is on the order of 5,000 visitors and 100 conversions per variation before you can read a modest effect. If you can only power a handful of tests per quarter, choosing which ones is the single highest-leverage decision in the program.
You will run fewer tests than you want, most will not win, and each one costs scarce traffic and dev time. Prioritization is not backlog hygiene. It is the act of spending a fixed, scarce budget on the bets most likely to pay back.
This reframes what a prioritization framework is for. It is not a ritual to rank 200 ideas. It is a forcing function to (1) surface the evidence behind each idea and (2) make expected value comparable across ideas, so the scarce slots go to the best bets.
Four frameworks worth knowing
They sit on a spectrum from fast and subjective to slow and objective. None is universally best. The right one depends on your team's maturity and how much you trust your own scoring.
Created by Sean Ellis (who coined growth hacking; author of Hacking Growth) for rapid experiment ranking on growth teams at Dropbox and LogMeIn.
How it works
Score each idea 1 to 10 on Impact (potential value), Confidence (certainty it will work), and Ease (inverse of effort), then combine into one ICE score.
Fast. Three inputs means you can score a whole backlog in an afternoon. It stays the most widely adopted growth-team framework precisely because it is cheap.
Subjectivity, and it is the big one. With three broad dials, two people score the same idea very differently. Impact is meaningless without naming the reference metric, and Confidence quietly becomes a gut-feel score. Ward van Gasteren's fix is telling: rename Confidence to Evidence and score independently so nobody anchors on the HiPPO. ICE is also easy to game.
Popularized by Chris Goward / WiderFunnel in You Should Test That! (2012); built specifically for CRO.
How it works
Score 1 to 10 on Potential (how much the page could improve), Importance (traffic and revenue value of the page), and Ease (resources to ship). Average the three.
Purpose-built for conversion work. Importance is a genuine improvement over ICE because it forces you to weight by page value and traffic rather than treating all surfaces as equal.
Still fundamentally subjective. Potential is vague and invites the same gut-feel scoring as ICE's Impact, just with a more analytical label. It also does not separate evidence from opinion, so a well-loved idea with no data can score a high Potential.
Built by Sean McBride on Intercom's growth team to compare ideas against a single conversion goal. Not CRO-specific, but its structure is the most useful to borrow.
How it works
Score = (Reach x Impact x Confidence) / Effort. Reach = people affected in a time window; Impact = a fixed multiplier; Confidence = 100/80/50%; Effort = person-months.
Two things ICE and PIE miss. It makes Reach explicit (so a tweak on a high-traffic page outranks the same tweak on a dead page), and it divides by Effort so a cheap, broad win beats an expensive, narrow one. This is the shape worth copying.
Still leans on subjective Impact and Confidence multipliers, and it shines only when ideas share one common goal metric.
Designed by CXL (now Speero) explicitly to fix the subjectivity of ICE and PIE. It replaces vague 1 to 10 dials with binary and bracketed questions anchored to evidence and observable facts.
How it works
Score about 10 factors in three groups: potential-to-be-noticed (binary), evidence (weighted, with user testing counting double), and ease (bracketed by build time). Sum the factors; higher means test sooner.
Objectivity. Binary questions cut the is-it-a-6-or-an-8 argument in half and make scores reproducible across people. Crucially, it bakes evidence directly into the score, so an idea with zero supporting data simply cannot rack up points.
Heavier and less intuitive. Overkill for teams new to prioritization, and the binary factors can feel reductive for nuanced ideas. You also have to actually have the research (user tests, heatmaps, analytics) for the evidence factors to mean anything.
The takeaway across all four: as you move ICE to PIE to RICE to PXL, you trade speed for objectivity, and each step pushes evidence and reach over effort further into the math. The model below steals the best parts: RICE's value-over-effort shape, PIE's page-importance idea, and PXL's evidence weighting.
Strip every framework down to five inputs
These five capture who sees the change, how painful the current experience is, how close it sits to revenue, how much evidence backs it, and what it costs to ship. Score each on a 1 to 5 scale, anchored to a definition rather than to gut feel.
| Input | What it measures | 1 (low) | 5 (high) | Where the data comes from |
|---|---|---|---|---|
| Reach | How many sessions actually hit the element or page | Niche page or rarely-seen element | Site-wide / every PDP, high-traffic | Analytics (GA4), pageviews, device split |
| Friction severity | How broken or painful the current experience is | Cosmetic / nothing is wrong | Users visibly struggling or abandoning | Session replay (Clarity), rage/dead clicks, heuristic audit |
| Revenue exposure | How close the change sits to the money | Far from purchase (e.g. footer) | On the buy action / checkout path | Shopify revenue by page, funnel value |
| Evidence / confidence | How much real signal backs the hypothesis | Pure opinion / HiPPO request | Multiple independent sources agree | User testing, GA4, Clarity, reviews, prior test results |
| Effort | Cost to build, QA, and ship the variant | Heavy build / cross-team | Trivial theme or app tweak | Dev estimate (person-days) |
Reach and Effort are unforgiving
Traffic caps which tests can even reach significance; effort caps how many you can run at all. A great idea on a low-traffic page that takes three weeks is usually a worse bet than a good idea that is site-wide and ships in a day.
Evidence is the input people fake
It is the one most easily inflated to justify a pet idea. Require a named source for any Evidence score above 2 (Clarity replay shows X, review mining surfaced Y), or it does not count.
A great model but shitty acronym
Borrow RICE's shape, sum the value and divide by effort, but feed it the five CRO inputs. This keeps a high-effort idea from winning just because its value looks big. (REEFR? FREER?)
Formula
Effort divisor
Trivial
≤1 day
Small
2-3 days
Medium
4-5 days
Large
1-2 weeks
Very large
>2 weeks
Higher score means test sooner. Sort the backlog descending; the top N that fit your traffic budget for the quarter are your roadmap.
Blank template
| Test idea | Reach (1-5) | Friction (1-5) | Revenue (1-5) | Evidence (1-5) | Value (Σ) | Effort (1-5) | Score (Value/Effort) | Rank |
|---|---|---|---|---|---|---|---|---|
| … | ||||||||
| … | ||||||||
| … |
Rules for using it well
- 1Score Evidence from named sources only. No source means a max of 2.
- 2Score independently, then reconcile. Don't let one voice set the table (that defeats the HiPPO).
- 3If two ideas tie, prefer the one with stronger Evidence. It is the better risk-adjusted bet.
- 4Re-score after every test. A winner raises the Confidence of related ideas; a loser lowers it. The backlog is a living thing.
Five Shopify PDP tests, prioritized
Invented to demonstrate the method on a hypothetical apparel Shopify store. Do not treat them as benchmarks. Five candidate PDP experiments sitting in the backlog:
Sticky add-to-cart bar on mobile
Keep the ATC button pinned as users scroll the PDP.
Move review stars next to the product title
Reuse the already-installed reviews app; surface the rating above the fold.
Reorder PDP: pull shipping & returns info up
Move shipping and returns info up near the price.
Replace the size dropdown with a size-guide + fit-finder modal
Swap the plain dropdown for a guided fit-finder.
Add a low-stock urgency badge
Requested by the founder.
Scored and ranked (illustrative)
| Test | Reach | Friction | Revenue | Evidence | Value | Effort | Score | Rank |
|---|---|---|---|---|---|---|---|---|
| B: review stars by title | 5 | 2 | 4 | 5 | 16 | 1 (trivial, move existing widget) | 16.0 | 1 |
| A: sticky mobile ATC | 5 | 3 | 5 | 4 | 17 | 2 (small, mobile QA) | 8.5 | 2 |
| C: reorder shipping/returns | 5 | 3 | 3 | 3 | 14 | 2 (small) | 7.0 | 3 |
| E: low-stock urgency badge | 5 | 1 | 3 | 2 | 11 | 2 (small) | 5.5 | 4 |
| D: size-guide + fit-finder | 3 | 4 | 4 | 3 | 14 | 4 (large build) | 3.5 | 5 |
Reading the ranking: this is the whole lesson
B wins, and it is the boring one
Repositioning an existing review widget is trivial effort, hits every PDP (reach 5), and is backed by strong evidence (review data, replays of users hunting for ratings, a deep external base of social-proof research, so evidence 5). Highest value per effort. The unglamorous reuse beats everything.
A is a strong second
It sits directly on the money action (revenue 5) with solid replay evidence, and stays cheap.
D ranks last despite being the team's favorite
Wrong-size returns are a clear friction point and revenue exposure is decent, but reach is narrow (only undecided apparel buyers) and the build is heavy. High effort plus limited reach sinks an otherwise appealing idea. This is exactly the trap effort-blind scoring falls into.
E (the founder's urgency badge) lands 4th and gets flagged
Reach is high, but Friction is about 1 (nothing is broken) and Evidence is thin (2, it is a HiPPO request rather than a data finding), plus forced scarcity carries deception and legal risk. The model correctly refuses to let seniority float it to the top.
Same five ideas, ranked by expected value per unit of effort instead of enthusiasm, and the order is nearly the inverse of what a room full of opinions would have picked.
Six ways prioritization breaks
HiPPO override
The Highest-Paid-Person's-Opinion jumps the queue regardless of score (the term is widely attributed to analytics author Avinash Kaushik).
Score independently before discussing, make every score show its evidence source, and let the founder's idea compete on the same sheet as everyone else's.
Scoring without evidence
Treating Confidence or Impact as a gut feeling is how opinion-driven testing sneaks back in. Ideas derived from opinions have less chance of success.
Rename Confidence to Evidence and require a named source. PXL enforces this structurally.
Ignoring effort and traffic
A high-value idea that takes three weeks, or that runs on a page without enough traffic to ever reach 80% power, is a bad bet no matter how exciting.
Always divide by effort, and pre-check each candidate against your sample-size math before it makes the roadmap.
Gaming subjective scores
Wide 1 to 10 dials let advocates nudge numbers until their idea wins.
Move toward binary or bracketed questions (PXL), anchor every scale point to an observable definition, and score blind.
Set-and-forget backlogs
Prioritizing once and never re-scoring wastes your best data source: your own results.
After each test, feed the outcome back into the Evidence scores of related ideas.
Confusing a big list with a good pipeline
A 300-idea backlog is not progress; it is noise. The constraint is traffic and evidence, not idea count.
Prune ruthlessly to the bets you can actually power and stand behind.
Replace the 1-to-5 guess with a revenue projection
The model above gets you a defensible ranking from a spreadsheet, and for most teams that is already a big step up. Spar pushes the same value-per-effort idea one level further: it replaces the Value estimate with a projected dollar figure, computed from the segment the change touches, the conversion delta we expect it to move, and the value of each conversion, all bounded by whether the segment has enough traffic to reach significance. Most teams score these inputs from memory, which is how subjectivity and HiPPO bias creep back in. Spar is an AI growth platform for Shopify brands and agencies that gathers the inputs automatically, then ranks the backlog by projected revenue.
Revenue projection
The sample-size check is what separates this from a wishlist of big-sounding numbers. An idea that would need three months of traffic to settle is flagged, so a modest lift on a high-traffic segment can correctly outrank a dramatic lift on a segment too thin to ever measure.
A second column: overall KPI impact
Projected revenue is one lens, but two ideas with similar dollar projections can move very different metrics, and a healthy program does not optimize a single number into the ground. So Spar scores each opportunity a second way: an overall KPI impact, rolled up from 6 to 8 underlying KPIs depending on the site's business model. A change that nudges conversion, add-to-cart, and average order value at once scores higher than one that buys the same revenue by lifting a single metric, which keeps the roadmap balanced rather than narrowly tuned.
Ecommerce store (8 KPIs)
Lead-generation site (6 KPIs)
A subscription brand weights lifetime value and return rate heavily; a one-time-purchase store leans on conversion and average order value; a lead-gen site swaps the commerce metrics out for form submissions and demo bookings. The KPI set follows the business model, so the overall score reflects what actually matters to that site.
The same backlog, scored Spar's way
Illustrative numbers| Test | Affected segment | Modeled delta | Sample size | Projected revenue | KPI impact | Rank |
|---|---|---|---|---|---|---|
| A: sticky mobile ATC | Mobile PDP sessions (large) | +0.6pp add-to-cart | ~2 weeks to power | ~$48k / yr | 86 | 1 |
| B: review stars by title | All PDP sessions | +0.4pp add-to-cart | ~1 week to power | ~$31k / yr | 78 | 2 |
| C: reorder shipping & returns | All PDP sessions | +0.3pp conversion | ~3 weeks to power | ~$22k / yr | 64 | 3 |
| D: size-guide + fit-finder | Undecided apparel buyers (narrow) | +1.1pp conversion | ~9 weeks (underpowered) | ~$14k / yr | 71 | 4 |
| E: low-stock urgency badge | All PDP sessions | +0.2pp add-to-cart (thin) | ~2 weeks to power | ~$9k / yr | 41 | 5 |
D is the same trap as before: the per-buyer delta is the largest in the set, but its segment is narrow enough that the test needs roughly nine weeks to reach power, so the revenue projection and the sample-size check together push it down. A's modest delta wins because it lands on a large mobile segment that settles fast.
Where every number comes from
Pulled from your analytics, Spar sizes the exact slice of traffic the change touches (mobile PDP sessions, paid-search first-time visitors), so reach is a real session count rather than a 1-to-5 guess.
Modeled from a storefront UX audit plus Microsoft Clarity behavioral signals (rage clicks, dead clicks, drop-off) against benchmark conversion, so the expected lift comes from observed behavior and not from optimism.
Read from Shopify (average order value, units per transaction, and where relevant repeat-purchase value), so a change near the money is weighted by the dollars actually moving through that step.
Computed from the segment's traffic and baseline rate, so an idea that cannot reach significance in a reasonable window is flagged before it ever reaches the roadmap.
Assembled from multiple independent sources (GA4, Clarity, product reviews, and prior test results), the multi-source bar this guide argues for, so a projection is never resting on a single anecdote.
You end up with a backlog ordered by projected revenue and overall KPI impact, fed by live data instead of a spreadsheet full of guesses, which is the entire point of this guide.
The limits, stated plainly
Projections are models, not promises: they narrow where to spend scarce test slots, they do not guarantee the lift will land. Spar is focused on the Shopify storefront, is not SOC 2 or ISO certified, and publishes no public uplift statistics. It is a prioritization and evidence engine rather than a guarantee of wins. No tool changes the fact that most tests don't win; it just helps you spend your scarce slots on the better bets.
Get the inputs from your store data
Pick your next five test ideas and run them through the scoring model above. If the gut-feel order changes, the model is already earning its keep. Want Reach, Friction, Revenue, and Evidence filled in from your store instead of estimates? See what Spar would rank first.