In the Apply analysis panel you can select if the calculations shall be based on weighted or unweighted data, apply significance testing calculations, apply benchmark calculations and add Mean series to the chart.


In the Significance testing panel you can test the results to each other to see if there are any significant differences between them.  In StoryCreator, you can choose between a Z test for a categorical calculation (percent share or mean) or a T test for categorical or numeric calculations (percent share or mean). 


The significance testing can be made across the items in the legend or across the items in the axis. You can also select how the comparison should be made; a data point can be tested to the first data point in the chart, the previous data point in the chart, the last data point in the chart or all the other data points.  See a few different examples of different significance testing below.


Example 1, the significance testing is made across the intervals (the axis items) and each data point is compared to the previous data point. In this example the significance testing indicates if it is a significant difference from the previous week or not. 

 

Example 2, the significance testing is made across the intervals (the axis items) and each data point is compared to the first data point. In this example the significance testing indicates if there is a significant difference from the first week or not. 

 

Example 3, the significance testing is made across the brands (the legend items) and both brands are tested against each other. In this example the significance testing indicates if there is a significant difference between the brands for each of the weeks. 


Example 4, the significance testing is made across the brands (the axis items) and each data point is compared to all other data points. In this example the significance testing indicates if there is a significant difference between any of the brands. 



Example 5, the same chart as above but in this case only the positive significance differences are displayed.

 


The setup panel of the Significance testing is divided into three different sections (Calculation settings, Test and Visualization) as shown in the image below and each of these are described in detail further down. In StoryCreator charts and tables the T-test option appears in the Statistical analysis panel when Numeric means is the selected calculation type. The settings are the same as when applying a Z test on categorical means.


Here we see the significance testing panel, where you can select type of test and related settings.


Calculation settings
In the Calculation section you define the following:

  • Select the significance testing level to be used (1%, 5% or 10%) in the “Significance level” list
  • Select if any base size limit must be fulfilled to perform the test in the “base limit” input field. If the data is weighted you can select if the base size limit shall be based on the unweighted count, the weighted count or the effective count of respondents
  • Select if any Respondent count limit shall be applied in the “Respondent count limit” input field
  • Select which base size option to apply to the calculation. This option is only appearing if the result is weighted. If the result is unweighted, then an unweighted base is always used in the calculations.

Test
In the Calculation section you define the following:

  • Select if the test will be made across Axis groups or Legend items in the “Test between” list.  Note, this list appears only if the chart contains items that make it possible to perform significance testing in both the legend and the axis. If the chart, for example, contain items that makes it possible to perform significance testing in the Axis only the “Test between” list is not appearing.
  • Select if all data points will be tested against each other or if the test will be made against the first, last, previous or next data point in the “Comparison type list”
  • Select if the comparison data point should be hidden or not in the “Comparison behavior” list Note: this option is not appearing if “All vs All” is selected in the “Comparison type list”.


Visualization
In the Visualization section you define the following:

  • If “All vs All” is selected in the “Comparison behavior” list you can select which significance test result to be displayed (positive only, negative only or both) in the “Result to show list”.
  •  Select which symbols to be used to indicate the significance differences in the “Symbols” list


Limiting Types of Test


The Administrator can limit which statistical options to be available the report, that is done in the Setup panel of the StoryCreator as shown in the image below.

Here we see how you can specify if both Z-test and T-test should be available or just one of these (or none).