Imagine yourself playing a video game. However, this isn't a normal video game with defined rules and goals. In this game, there isn't a point system. No scoring mechanisms define who wins or tell you whether you're doing good or not. How do you crown a champion? And more importantly, how do you get better at the game without any feedback to guide your learnings?

This is the exact problem many businesses have without even knowing it. They evaluate everything based on revenue metrics without realizing that it's close to being blind. Revenue doesn't help you understand your own performance in a way that informs future decisions. Yes, your revenue went up. But if you don't understand exactly why your revenue went up, you didn't learn anything. You didn't improve, and you can't replicate it over and over.

It's not accidental that Eric Ries's The Lean Startup became a best-selling book. In it, he describes the Minimum Viable Product (MVP) Framework that many successful companies have used to guide their growth.

In the MVP framework, you start by building your minimum viable product - the minimum required to measure the response of the early adopters of your product. Then you use the insights from those measurements to make an informed decision (hypothesis) on how to make your product better. Finally, you measure whether the changes actually made your product better. And then you repeat, iteration by iteration until your business is the next Dropbox, Airbnb, or Shopify.

From this, let's define rule number one of data-driven eCommerce:

The point of measuring things is to be able to make informed decisions based on those measurements.

If you're tracking something that doesn't help you make informed decisions in the future, you're wasting your time tracking them. Luckily, now’s your chance to start tracking the right metrics for your business. If you do, you generate feedback loops that give you feedback on how to make your products better, attract more customers, and eventually make more money.

What not to track

As with many problems, it's easier to understand what metrics to track by understanding what not to track. Often called vanity metrics (VM), metrics not worth tracking are metrics that don't help you understand your own performance. They might make you feel good, but they aren't helping you inform your future decisions.

You can find examples of vanity metrics everywhere. How often have you seen something like "7 figures in sales" or "more than 10.000 customers"? While 7-figures in sales and 10.000 customers sounds amazing and can be a sign of traction or a successful business, it's not really saying anything. How many of those customers are actually happy, returning customers? What if most of the 7-figures in sales are being returned?

The problem with vanity metrics is that they can give you a false sense of success. You can have an eCommerce business with more than 100.000 customers. But if your source of new customers is declining (it will at some point) and your existing customers aren't coming back, it's just a matter of time. Your business isn't healthy and will die.

How do I know if I'm tracking a Vanity Metric?

It's important to understand that any metric can be a vanity metric in the right context. What defines a good measure varies from business to business.

Therefore, to understand whether a metric is a vanity metric for your business, ask yourself the following two questions. If it's a no to any of them, or if you're doubting whether it's a yes or no, it's a vanity metric.

• "Can this metric lead to a course of action or inform a decision?"

• "Can I intentionally reproduce the result this metric is measuring?"

In other words. If a metric doesn't help you in your decisions in the future, it's not usable. And if the scenario you're measuring isn't happening again in the future, there's no point becoming better at decision-making in that scenario.

The perfect counter-example to a vanity metric in eCommerce is split-testing a landing page. If your changes increase your conversion rate, you know to make those changes permanent. And you can apply those permanent changes again and again for all future visitors of that landing page.

Finally, make sure that your metric is accessible and trust-worthy. It's not a vanity metric if it isn't accessible, but if your metric is buried deep down in a data warehouse behind a wall of friction, no one will use it. And if you can't trust it, maybe it's generated by some crazy code, you are most likely not feeling good using it to direct your business.

How Airbnb’s hockey-stick growth came from tracking the right metric

At SXSW 2012, Joe Zadeh, Product Lead at Airbnb, shared an amazing data-story. It turns out that the founders of Airbnb were inspired by the Lean Startup / MVP mindset.

One day, the founders hypothesized that listings with professional photography would attract more business, and that hosts would be willing to sign up for professional photography as a service. As MVP-followers, they didn’t built a whole photography service just to see it fail, but built a MVP convierge service where hosts could book a professional photographer to come and take photos for their Airbnb listing.

Initial tests showed that professionally photographed listings got two to three times more business than the average listing. So by late 2011, Airbnb employed 20 photographers, tested adding watermarked photos to add authenticity, let customer service offer photography as a service when potential renters called, and much more. With every change, every iteration of their product, Airbnb tracked how it affected one key metric: photoshoots per month. As listings with professional photographies were booked 2 to 3 times more often, this was their main focus. Their North Star. By February 2021, Airbnb was doing close to 5,000 photoshoots per month, and the famous hockey-stick growth started.

With this story, I challenge you to challenge yourself to think about which metrics you are spending your time on. Are they vanity metrics making you feel good? Or are they actually guiding your decisions in the future as they were guiding Airbnb.

Godspeed,Mathias