Optimizing for Conversion Rate Is a Mistake.

If you’re in eCommerce and anything like me you’re probably spending a great amount of time trying to optimize your site through A/B tests. And when doing so, you’re most likely comparing the conversion rates of your experiments to select a winner.

This makes sense. If variant B results in more sales, variant B sounds like the better option. But what if I told you it’s actually not as great a metric as you might think?

The problem

The problem with using conversion rate as your guiding stick is that it’s one-dimensional. It doesn’t consider how much you're getting out of a conversion, only how often you’re getting a conversion. In other words, it doesn’t consider the order value, only how often you’re getting an order.

But why is that a problem? To illustrate this, consider an extreme experiment in which you reduce your prices by 50%. It doesn’t take much reasoning to conclude that this will make your conversion rate skyrocket. Percentage-wise, more people will buy from you with reduced prices. Does that mean that you should just reduce prices? No.

To offset a 50% reduced price and make the same amount of revenue for any given conversion rate, you’ll need to double your conversion rate. And assuming a 30% COGS and no tax, you need to 3.5x your conversion rate to offset the reduced price if you want to make the same amount of profits.

In other words, only considering the conversion rate isn’t enough when what you’re testing can impact how much a conversion is worth. Which pretty much all A/B tests can.

Before heading to the solution, remember to sign up to be notified of more articles like this if you aren’t already:

Subscribe now

The solution

So if optimizing for conversion rate is suboptimal, what do you optimize for instead? In my opinion, revenue per session (RPS) is a better metric. If you do the algebra, you can actually write Revenue per Session = Average Order Value * Conversion Rate. In non-math, RPS considers both the how much and the how often.

An Example

I recently did an experiment where I tried removing the dynamic checkout button from our product pages. I was afraid that it did more damage than good for our target audience (+65-year-old women).

Here’s the result comparing conversion rates:

In terms of conversion rate, removing the dynamic checkout button was performing better in-sample. While surprising, the difference was not enough to be of statistical significance. If I just looked at these results, I would either not remove the dynamic checkout - for most audiences, it doesn’t make sense that a faster checkout reduces conversion - or let the experiment run longer to get more data.

Luckily, I had a hypothesis. I expected a decrease in conversions by removing the dynamic checkout button. But I also expected an increase in the average order value. Why? Because I don’t think my audience understands the difference between the “Add to cart” and “Buy Now” buttons. I had a hunch that those randomly clicking the Buy Now instead of the Add to Cart button would be confused as to why they couldn’t buy multiple items and just check out the one item.

Therefore, I made sure to also measure the revenue per session:

The results? A 17.5% in-sample increase in revenue and a statistically significant result. It looks like my hypothesis holds true. My audience doesn’t understand the difference but just assumes it’s not possible to buy more than 1 item at a time when clicking the “Buy now” button. Therefore, I removed the dynamic checkout button. In the future, I might play around with the wording. But for now, it’s gone.

Let’s sum up. If I just looked at the conversion rate, the how often, I would have mistakenly thought that removing the dynamic checkout button didn’t change anything. But by measuring a metric that considers both the how often and the how much, I was able to measure a significant improvement and thus make a change that will increase my revenue numbers in the future.

I hope this gave you some food for thought and makes you consider more than just your conversion rate in the future.

If you enjoyed it, you can sign up to get notifications when I post more like this in the future. I have drafts in the making about how to optimize your Facebook and Google product feeds, what mathematical optimization theory can teach you about A/B testing, and why you should consider rethinking your navigation.

Subscribe now

Godspeed,Mathias