The purpose of performing a significance test can vary but the most common purpose is to identify differences between subgroups or subgroup’s results compared to the totals in the survey. Significance testing can be performed for comparisons of proportions or for comparisons of mean values.
Significance (Z test) tests can be used in StoryTeller charts; The test can be performed in both mean values and in proportions. It can also for instance be performed between brands, target groups and time periods. To enable significance test in a chart, under Analysis tab, go to Statistical analysis menu and enable significance tests, as shown in the image below.
Below some examples are shown. Significant differences are indicated with a suffix followed by a + or a -. The prefix shows to which data point the test has been performed, the prefix is added automatically be the system when significance tests are active. The prefix is only added when the test is performed between all data points, if the test is performed against the previous or next data point no prefix will be added.
The image below shows some examples of significance testing in charts.
As shown in the image below some series have been marked with a “-“. This means that the base or the respondent count for those particular answers is too low and no test has been performed.
When performing a statistical test to the first or last data point in a StoryTeller chart/table you can select to remove the letters (a, b, c, d etc.) indicating to which data point the test was made as this information, in many cases, is added to reports via text boxes.
Here we see an example, the left chart shows the old logic where letters always were used to indicate to which data point the test was made, in the right example you see that the letters are removed and an info text added which explains how the test was made.
The option to remove the letter indicators is present in the dropdown list for selecting starting letter as show in the image below, here you can now also select “No letter indicators” which also is the default option when creating a new chart/table.
Here we see the setting appearing when testing to first or last data point. The option is not available when testing all data points to each other as in that case the letters must be shown as each data point is tested to multiple other data points.
In case you would like to see significance testing in action and some tips and tricks before really getting started, please click here.