Overview:
Using Loops, Atera’s analysts detected that a particular modification was the primary cause of a KPI decline. Armed with this newfound insight, Atera’s team was able to adjust their views about the success of a new feature release, ultimately implementing a third round of updates in the same product area. These subsequent changes resulted in a substantial boost in Atera’s funnel conversion rates. Thanks to the insights provided by Loops, Atera’s team rapidly learned from their previous iterations and could react accordingly until achieving success.
Loops and the Case of Unexpected KPI Drops
Like other companies, Atera undertakes periodic modifications to their funnel in order to bolster product growth. However, during one particular instance, and following a series of updates, Atera observed a rather unexpected decrease in two of their funnel KPIs. In response, the company used Loops, the product analytics platform, to promptly pinpoint the underlying reasons behind the change to one of the product modifications rather than seasonality or other external influencers. Atera’s team adopted the methodology of using a sophisticated pre-post analysis, instead of relying on AB testing which requires more data and time.
Method and Results:
Atera utilizes Loops’ Release Impact analysis, a distinctive, product analytics model which simulates an AB test environment in a pre/post change analysis, to measure changes to their business metrics in each iteration.
First Iteration – a negative impact revealed
Loops surfaced that the release had a significant, negative impact – even on a KPI for which the trend seemed positive (Business Metric #1).
This was a significant finding – while one KPI appeared positive because of many variables that can impact it, the Loops model concluded that the actual effect of the change is negative (you can read HERE how this can happen).
With the second KPI, Business Metric #2, while the trend was negative from the get-go, Loops concluded that the trend was worse than first witnessed.
Only with a solution like Loops, that uses unique causal inference models, is it possible to attribute individual impact to each variable and detect a negative impact in an increasing KPI.
Second Iteration – the impact of a release
Loops showed that even though different changes were made simultaneously – the decrease in KPIs, especially Business Metric #2, was related to the release.
Third Iteration – a 33.88% Improvement!
Loops helped the team prove that this iteration of improvements caused the KPI to increase significantly and that the impact on Business Metric #1 was 33.88% higher than they initially witnessed (causal change vs. naïve change).
