Traversing Product

Random things that I find & learn about Product Management

Experiments

Aug 2021

As Product people, we’re often wondering – what do we build next?

We want to be confident we’re working on the right things (that’ll have the most impact) at the right time (over anything else we could do).

We’ll likely have ideas for upcoming initiatives/projects/bets/epics and they may be loaded with assumptions. If that’s the case, before we invest too much of our teams time, we need to prove or disprove our assumptions and answer our open questions – with experiments.

Just a reminder, an experiment can be defined as

A scientific procedure undertaken to make a discovery, test a hypothesis, or demonstrate a known fact.

And a hypothesis can be defined as

An idea or explanation that you then test through study and experimentation.

So before we can run an experiment, we need a hypothesis to test. I learnt this hypothesis template a while back from an awesome agile coach and I really, really like it is:

we believe that this thing
will result in this outcome
we will know we have succeeded when this visible signal happens

An example of that in use

we believe that allowing users to sign up with Instagram
will result in more users signing up to use our product
we will know we have succeeded when signups increase by X%

So now we have a hypothesis, we need figure out and run the actual experiment while we ensure we are accurately measuring the results.

Once the experiment is complete, we should have either

  1. Enough evidence to proceed (with the next experiment or the full solution)
  2. Enough evidence that it’s not a good idea and move on for now

We may also see some surprising results or gain some insights that lead to other ideas and even more experiments!

Aside – In my example above, depending on your product, adding an Instagram integration might take lots of effort & have ongoing implications.

So before you can even run that experiment, you might want to come up with a minimal viable experiment – ideally one that involves minimal or no code, that you could do to validate your assumptions, as Trent mentions in his post about MVEs

We need to deeply understand users via usage data, surveys, in-person visits and anything else that can help us fill in the blanks. We should be prototyping our ideas, and showing users mockups long before a line of code is written.

This post from mind the product is a few years old, but it goes into a lot more detail if you’d like to read more about experiments!

- Aaron