How to A/B Test a Shopify Product Page Properly

Most Shopify brands test a product page like they test an ad. They swap a headline, glance at the dashboard for a week, and decide by feel. That is not a test, that is a guess with a screenshot attached. If you actually want to know whether a new page makes more money, you have to test it properly: control against new page, split on the brand’s own live traffic, read from the testing platform, not the ad account.

How to A/B test a Shopify product page

The setup is simple even though most brands skip it. You take the page that is currently live, the control, and you build a real alternative, not a tweak. Then you split your actual store traffic between the two, fifty fifty, using a testing tool built for this. We have run these on VWO, ABConvert, and Eraya depending on the brand. Every visitor who lands on the product gets randomly assigned to one version or the other and stays there. No opinions, no gut calls, just the number the traffic gives you.

The part brands get wrong is the comparison itself. A new page tested against nothing, or judged against last month’s numbers, tells you nothing, because traffic quality, season, and ad spend all move at once. The only honest test is the same traffic, at the same moment, split between two pages.

Split on your own live traffic, not an ad platform

Your ad account will tell you a story you want to hear. It reports purchases and revenue based on its own attribution window, which can double count, miss returning visitors, or credit the wrong campaign entirely. None of that touches your actual store performance.

A real A/B test runs the split on your site, not inside Meta or Google. Half your visitors, whoever they are and wherever they came from, see the control. Half see the new page. The testing platform tracks what each group actually spent on your store, not what an ad platform thinks it drove. When we ran Sonic Brush’s new page against their original PDP on Google Ads traffic through ABConvert, the plus 24% number came from ABConvert’s own reporting, not from Google’s dashboard. That is the only number worth trusting.

Read revenue per visitor, not conversion rate alone

Conversion rate on its own is easy to game. Add a discount, watch the rate climb, and average order value quietly drops enough to erase the gain. Revenue per visitor closes that loophole because it is conversion rate and average order value combined into one number. A page can only win on revenue per visitor by actually making the store more money per person who lands on it.

This is the metric behind every result we have measured. Particle for Men’s offer page beat their own PDP by 21.4% in revenue per visitor, at 99.5% confidence across 12,893 visitors on VWO, and it is still the live page two years later. Bareline’s new PDP template beat their old template by 13.5%, at 99% probability to win across 92,484 sessions on Eraya. Both are large samples, both read straight from the testing tool, and neither could have been faked with a discount code.

Run it long enough to trust the number

A test that ends after three days on a low traffic store is not a result, it is noise. The tools we use report a confidence level and a sample size alongside the lift for exactly this reason. DuraDry’s test against their own PDP ran two weeks on Google Ads traffic before we called it. Some tests on smaller stores, like Prime Natural and Snuff Cup, land big numbers, over 300% each, but on small samples, and we say so rather than pretending a small sample carries the same weight as a 92,000 session test.

This is also why our own guarantee is tied to this exact metric: a 10% lift in revenue per visitor in a controlled test, or your money back, across up to 5 test iterations. The full terms are on the guarantee page.

What this looks like across real brands

We have run this process across dozens of stores, and the honest picture is a range, not a single number. Free the Roots beat a collection page by 33% on Meta traffic through ABConvert. Gum of Gods beat their existing offer page by 97%. Habit Nest and Pure Path Shop each landed a steadier 10%. Every one of these is the new page against the old one, on that brand’s own traffic, read from the testing platform. You can see the full set of measured results and how each one was tested.