Sample size Calculations (Power Calculations)

Testing with a to small sample size introduces a risk to falsely detect a difference or not detecting a difference. To reduce this risk it is important to do a sample size calculation. The bigger the samples size is the better the detection is. When using a samples size calculation it is known what the risk is and minimizes the amount of samples for a good power.

Power (β) Type II error (false negative) is the sensitivity of the statistical test the higher this value the smaller the possibility that is falsely rejects the null hypothesis. Develve is using β value of 0.8 that is commonly used.
Statistical significance (α) Type I error(false positive) the lower this value, the more difference is result or the bigger the sample size, before the result is significantly different.

1 How determine the minimum sample size

A good start for the sample size is between 20 and 40. With this amount a distribution test can be carried out.

2 Perform the statistical test

When performing the test Develve will calculate the minimum sample size when the calculation is available see below in table with sample size calculations in Develve.

3 Interpreted the result of the statistical test

• Is the result significant
• Is the shape of the data(sets) OK for the test
  • Normal distributed
  • Not normally distributed and all sets have the same shape
• More information see specific test
  

4 Interpreted the sample size result

If the amount of samples is bigger then the calculated samples size, you can rely on the outcome of the test, because the statistical power is high enough.

If the test is significant. And if the amount of samples is smaller than the calculated minimum sample size, increase the amount of samples to the minimum samples size.

If the test is not significant. When a test is not significant increasing the sample size will not help to make the difference significant. Increasing the sample size will reduce the risk on false detection, but increase the sample size with care because it is possible that is waste of testing.

Develve is capable to calculate the minimum sample size of the following statistical tests:

• One sample t-test
• 2 sample t-test
• Variation F-test
• 1 sample Wilcoxon median test
• 2 sample Mann-Whitney median test
• One way Anova
• Kruskal-Wallis Test
• 1 sample Proportion calculation
• 2 sample Proportion calculation
• Linear Correlation

Example 2 sample t-test (not equal)

To use the 2 sample t-test test first unselect "non normal distributed" when the box is selected the 2 sample Mann-Whitney median test is calculated. Then select Diff mean.

• The difference between data set A and B is not significant and the sample size is to small (Row t-test p >0.05).
• The difference between data set A and C is significant and the sample size is big enough (Row t-test p <0.05).
• The difference between data set A and D is not significant and the sample size is big enough (Row t-test p >0.05).

Data file

Links

• http://en.wikipedia.org/wiki/Statistical_power
• http://www.itl.nist.gov/div898/handbook/ppc/section1/ppc134.htm
• http://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm
• http://www.itl.nist.gov/div898/handbook/ppc/section3/ppc333.htm