To set up a significance test in the StoryTeller, you can go to the Statistical Analysis tab in Analysis. Here you will be able to enable Significance tests and select the Type of test you would like to do.
When using significance testing the following selections can be set up:
- Set the base size limit – no test will be performed if the base size is too low
- Set the respondent count limit – no test will be performed if the respondent count of the particular answers does not fulfil the limit
- Define if you want the base size to be calculated on weighted, unweighted numbers or on the effective base
- Define if the tests should be performed between all data points, or against next data point or against previous data point
- Define if the test should be performed “within the group” or between “groups”
The image below shows the significance test options for the Z test in the Settings Tab in the chart setup in StoryTeller.
A group (see point 4 above) is a text on the series axis; some example images are shown below. The highlighted area in each chart example below is one group. If the test should be performed within those variables and not against other variables the selection should be “Test within groups”.
The example charts below highlight a “group” within each chart.
The example images below show some different combinations of significance testing.
Example 1 shows a test within groups and All vs. All. This means that all age groups are tested against each other within a brand.
Example 2 shows a test between groups and All vs. All. This means that all brands are tested against each other within an age group.
Example 3 shows a test between groups and against a previous data point. This means that the result of each brand is compared to the result the month before.
Example 4 shows a test within groups and All vs. ALL. This means that the result of each brand is compared to the result of the other brands the same month.
Significance testing can be done in all chart types except Scatter and Bubble Charts.
Tip, when running a test to one (1) other data point you can select to display the T value, Standard deviations, Degrees of freedom and Critical value in the chart tooltip. The purpose of this function is to give the administrators the ability to check which values used in the calculations.
Here we see the setting for showing the used values in the T-test calculation in the chart tool tip. This option is not present when “All vs All” is selected in the Comparison type list.