How to measure
Please go to MARKETING>PUSH NOTIFICATION
Click the campaign name you wants to measure on this screen
You can see the result of the delivery on the next screen.
The graph shows the number of push notifications delivered, the open rate, and the conversion rate for each pattern.
Number of total target users of the campaign on a daily basis.
Number of total users who have opened the campaign on a daily basis. This includes both direct and influenced opens.
Number of users who have activated the app within 1 hour from the delivery of the notification.However, if load balancing option is used, there will be a gap in the number of opens from the actual one of opens. IOS SDK 1.7.34 and above, Android SDK 1.2.9 and above is required to measure influenced opens.
Number of users who have activated directly from the push notification.
The number shows the daily total of users who have achieved the goal designated in the creation page within one hour of “opening” the push notification.
Number of conversions devided by deliveries.
Results of the deliveries, opens and occurrence of the conversion event on a daily basis: Note that the results represent number of users.
How to measure A/B test
During the A/B test, the results of the test will be displayed.
Here, you will be able to select different patterns including the control group as a baseline or change targeted markers to CVR or open rate.
Each pattern used to deliver the A/B test, including the control group, will be displayed here.
Here, you will be able to select the baseline pattern that will be used as the standard when comparing each pattern. By changing the baseline, you will be able to compare between a variety of different patterns.
An indicator is used when comparing patterns to baselines in A/B tests. You can select a ratio value (i.e. CVR, open rate) as the indicator. Also, the indicators listed can be changed through the editing page.
90％ confidence interval
The 90% confidence interval will be displayed when each pattern is compared to the baseline. For the details on the 90% confidence interval, please see the development guide.
When displayed as such, it means that the difference between the pattern and the baseline is significant. We recommend moving onto the winner pattern if enough time has passed since delivery.
When displayed as such, it means that the difference between the pattern and the baseline is not significant. Either wait until the results appear, or consider moving on to the next campaign if enough time has passed since the delivery.
When displayed as such, it means that the difference between the pattern and the baseline is significant. If this is the only window displayed, we recommend determining the baseline pattern as the winner pattern.
Below is an example of when the winner pattern is selected from the A/B test significant difference results. The comparison results of the rest of each pattern will not be displayed.
Status List Screen
You will be able to check the status of the A/B test for each campaign here, when the operating campaign is either live or not. Icon display conditions are as follows.
In the instance of a campaign that has a significant difference with the baseline having both exceeding and falling below results, it will be displayed as the results of the pattern with at least one exceeding the baseline.
Select the winning pattern
Once you select the winner, all after deliveries of the same push notification will be using the selected pattern to all target users.