Skip to content
Engineering

What it actually takes to automate CRO

You could spin up a working prototype. Keeping one correct on a live storefront is the cost almost nobody prices.

Eric Kim, co-founder
August 5, 2026 · 7 min read

Shopify sets a cookie called _shopify_y. It arrives in the Set-Cookie header on the very first HTML response, before the visitor has clicked on anything or seen a single product.

We were reading it as evidence that someone had been to the store before.

You can see where this goes. Every brand-new visitor came back marked as returning. Any test we'd pointed specifically at new visitors was running against an audience of almost nobody. Nothing crashed. No alert fired. The reports arrived on schedule with numbers in them, and the numbers looked reasonable.

I fixed that. _ga turned out to carry the same risk, so that went too, and detection now runs off our own identifier plus the Liquid customer account.

Now 2 years into the build, Spar audits any website, ranks what's costing you conversions, writes the variant code, ships it to your testing platform and calls the winner. Of course, even this is an oversimplification. Every few weeks somebody tells me they could build that internally, and for the first version they're right. Wire a model to GA4, hand it a few screenshots of a product page, and it'll tell you the add-to-cart button is below the fold on mobile. That's a weekend. It's also a genuinely useful thing to know.

What I want to describe is the part after the weekend, because I've been living in it.

Over the last two months our repo took over 1000 commits. Over 200 of them have "fix" in the subject line. I'm not reporting that as a confession. It's roughly the running cost of keeping something like this honest while Shopify, themes, analytics providers and the models themselves all move underneath you.

Before you start

The estimate you write at the beginning covers what's visible from outside. Connect the analytics. Prompt the model. Put the findings somewhere a person can read them.

The months go into the parts you can't see yet. A crawler that can photograph a live storefront without breaking it. Some way to get a month of store data small enough for a model to actually read, which is a much tighter constraint than it sounds. Generated code that's safe to run in front of buyers. A path into whatever testing platform the store already uses. And a statistics engine at the end producing a number you'd be willing to act on.

None of that is exotic. All of it is a project, and every one of them drags a maintenance tail behind it.

If I could tell my earlier self one thing, it's that finding the problem is the cheap step. A decent model does it in seconds. Everything downstream of the finding is where the build actually lives.

While you're building

A sample from the last few months.

Storefronts don't want to be photographed. We sweep popups and overlays off a page before capturing it, because otherwise every screenshot we take is a newsletter modal. That sweep was also closing the cart drawer, which meant the single part of the flow we most needed to see was reliably absent from the capture. It took a while to notice, because a capture with no cart in it doesn't look broken. It looks like a page. In the same stretch, stores with multi-segment locale redirects were failing the crawler's preflight, and a store with no best-selling product was throwing during onboarding rather than carrying on without that highlight.

Store data doesn't fit in a model. We were handing page captures to the model as very tall images. It took them, downscaled them to fit its own limits, and the text in the screenshots came out unreadable. Nothing refused. It answered questions about a page it couldn't actually read, fluently, in complete sentences, which is a worse outcome than an error and took far longer to suspect than it should have. Clarity data and PostHog events turned out to have the same shape of problem.

The statistics have to survive their own math. Our win probability comes out of a Monte Carlo simulation and the simulation is unseeded, so two identical inputs can land on 0.49 and 0.51. A test asserting they'd be equal failed, correctly, and it was a useful thing to be reminded of. The number you put in front of a merchant has variance baked into it. You need to decide how much of that you'll tolerate before you tell somebody to ship a variant on the strength of it.

Each of these cost somebody a day or more to track down. None of them are the interesting part of the product, and all of them had to be right before the interesting part meant anything.

After it ships

Here's the one that changes the arithmetic, and the one estimates almost never price.

We show merchants a live preview of their own storefront with our suggested change overlaid on it, which means running their page inside a proxy. Over two days at the end of July I shipped four separate fixes to it. Each one closed a different way a page could slip out from under the proxy, and none of the four was the same bug as the others. None of them were on anyone's list the week before.

Four fixes. Two days. One feature, and four different routes out of it that nobody had thought of.

That isn't a story about a badly built proxy. It's what owning a surface like this looks like in an ordinary week, and the work doesn't taper off after launch. Tokens expire and somebody re-authorizes them. A provider changes the shape of a field and somebody rewrites the query layer. A theme update moves the selectors your crawler leans on, and the crawl comes back clean, full of pages it read successfully and understood incorrectly. Your scoring library gets tuned by one person on a spare afternoon with no way to check the new version against the old. Every page type past product pages is fresh generation logic and a fresh way for the output to misbehave in front of customers. Each testing platform's API moves on a schedule you don't control.

There's a staffing question underneath all of this that's easy to defer. Today the person who wrote it can fix anything on it in an afternoon. That stops being true the moment they change teams, and nothing about the system announces the change. It carries on producing output with exactly the confidence it had the week before.

That's the part I'd want weighed honestly, because a CRO pipeline breaking doesn't behave like a service going down. There's no alert and nothing to wake anyone up. The reports keep arriving, the winners keep getting declared. What tells you something is wrong is a quarter of testing that didn't earn anything, which is a slow and expensive way to find out.

When you should build it anyway

I'd be lying if I said nobody should, and we wrote up the honest version of that on the comparison page. My own short version is narrower.

The question isn't whether your team is capable. It's whether this lands as work somebody is actually resourced to do, or as a project that gets built once with real care and then slowly stops being anyone's priority while everyone keeps reading its output as though nothing has changed.

If you can name the team, and they have the hours and expertise, build it. You'll end up with something fitted to your store, which is worth something on its own.

What I'd push back on is the version where this gets scoped as a quarter of one engineer's time alongside their existing roadmap, on the strength of a prototype that worked. That's the estimate that goes wrong, and it goes wrong in month five rather than week two, which is the expensive way to be wrong about a number.

Why I'm writing this down

We keep fixing these because it's the product. All those optimizations and fixes affected every store on Spar without anyone filing a ticket, and the same will be true of whatever we turn up next month. On a build of your own, every one of them is a ticket somebody picks up, and peak season won't pause while they do.

If you're sizing this up internally I'd rather you go in with an accurate picture than a flattering one. The weekend version is a nice prototype. It's month six that tells you whether you wanted to own it.

See what Spar finds on your storefront

A free audit ranks the conversion gaps on your store by revenue impact, with the variation code written for the ones worth testing.