For data and product professionals, one of the significant challenges we often face is conducting meaningful experiments.
This is especially true when product traffic is relatively low, a known scenario in B2B products. In these conditions, deriving valuable conclusions can become a time-consuming process.
There are many ways to combat this issue:
- Execute more aggressive tests. Keep in mind that the smaller the impact you're trying to detect, the larger the traffic volume needed to achieve statistical significance, or what we refer to as MDE (minimal detectable effect).
- Modify your significance level. Consider lowering it to 90%, or even 85% for B2B companies. Although there's some risk associated with this approach, it's a worthwhile trade-off to bypass the stagnation that can arise from aiming for unattainable results.
- Leverage previous data. Models such as Bayesian analysis and CUPED utilize prior data and can be valuable in this context as well. Use this historical data to guide your current and future strategies.
- Pinpoint a larger leading indicator, e.g., previous steps in the funnel or the user journey. If you're able to prove that it is a good predictor of the desired KPI, and more users go through this stage, you can shorten the experiment significantly.
- Opt for one-sided tests over two-sided tests. This approach is particularly effective if your aim is to confirm that the treatment isn't inferior to the control.
- Leverage causal inference as an alternative for A/B tests. unique algorithms simulate A/B tests by isolating noise factors such as seasonality, marketing changes, trends, etc. They enable you to understand the impact of your efforts even without A/B tests.
Carrying out experiments with limited traffic doesn't have to be a daunting task.
With boldness, ingenuity, and automation, it's possible to manage effectively and even thrive in this scenario.
