layout: blog title: “Installing a Nest thermostat in Australia” hero-header: “Installing a Nest thermostat” date: 2019-09-05 17:16:49 +0100 categories: [blog] permalink: “a-b-testing-hypothesis” excerpt_separator: –
Vague hypotheses makes analyse a nightmare.
You only need to try and analyse one experiment with a vague hypothesis to understand why you never want to do this again.
Your hypothesis must include the metrics
Too often people build AB tests around goals which include vaguely defined ambitions such as “user engagement”.
Sometimes that can mean “seeing the thing” (this is ok, if you’re measuring that), “clicking on the the thing” or something else entirely.
The formula for great hypothesis
The format which I use is as follows:
By doing X we expect to see Y as measured by Z.
This makes it clear from the outset:
- What you’re doing
- Why you’re doing it
- What you think the result will be
- How you’ll measure it
A strong hypothesis is the foundation on which an effective experiment is made.
I stole this approach from Craig Sullivan (if you don’t follow him on Twitter, you should be: he’s one of the most switched on optimisation minds out there) a few years ago.
When it comes time to analyse your results, there will be no ambiguity about whether or not you achieved your desired result and specifying the metrics from the outsr
Link your metric back to profitable actions
Let’s take the above example where someone has defined a vague hypothesis which speaks about ‘user emgavement’ without specifying a metric to measure that.
You’re unlikely to be in the user engagement business. You might be in the advertising business or you might be in the eCommerce business but in almost no case would you solely care about ‘user engagement’ as a measure in and of itself.
But sometimes when you’re formulating an exciting new test idea, it’s easy to get waylaid. You’re excited about trying a rotating new red widget on your homepage, not necessarily in thinking about how to measure the impact.
By following the hypothesises formula and specicallt the ‘As measured by’ section, you’re forced to measure how your test can be assessed by quantitative analysis.
Rather than thinking obliquely about ‘user engagenent’, this method forces you to take the next step to establish how you can measure this.
It might be Pages Viewed Per Session (or even better, advertisements loaded per user, if you’re in the media business), for instance.
And in some cases, you might find that your test concept isn’t suitable for A/B testing at all.
Fortunately for you, you will have discovered this before your test has finished and it’s time to write it up and your stakeholders are anxiously waiting for results.
Got another way of formulating hypotheses? Let me know in the comments as I’d love to hear it.