Significance testing for overlapping and independent sample (AskiaAnalyse)
The diagram below explains the correct tests to use when you want to test for significance in independent or overlapping sample (click or tap the diagram to enlarge it).
In essence, if you are sig-testing columns, and a respondent belongs to more than one column (e.g. in a multi-coded question or loop summary table), then the most statistically accurate tests to use are paired tests. Otherwise, the most accurate are independent tests.
Here are some examples of what is both 100% statistically accurate for testing, and available in AskiaAnalyse:
- if you are comparing means in your rows and have a single-coded question (with at least three categories) in your columns, then you need your col-sig advanced options to use ANOVA one way;
- if you are comparing means in your rows and a have multi-coded question (with at least three categories) in your columns, then you need your col-sig advanced options to use ANOVA with random subject;
- if you are comparing means in your rows and a have multi-coded question (with no more than two categories) in your columns, then you need your col-sig advanced options to use Wilcoxson's rank sum test.
Set the appropriate test in the advanced options for column significativity, as shown below (click/tap image to zoom in):
Here are some examples of what is 100% statistically accurate for testing but not available in AskiaAnalyse:
- If you are comparing proportions (on a closed question) in your rows and have a multi-coded question (with two categories) in your columns, then you need your col-sig to use the McNemar Test
- If you are comparing proportions (on a closed question) in your rows and a have multi-coded question (with at least three categories) in your columns, then you need your col-sig to use the Cochran Q Test
Although the above analyses are not available, askiaanalyse offers the following alternative way to achieve a good approximation:
- Set the count threshold to 5 - default.
- Set the base threshold to 30; the Statistical Normal Law (used for approximation) is applicable only if we have at least 30 in the base.
- The efficiency coefficient is to be used if analysing weighted data.
Set the appropriate test in the advanced options for column significativity, as shown below (click/tap image to zoom in):
The following tests are not yet implemented, but are on our development roadmap (they will be supported in a future version of askiaanalyse):