Sample Size
Strength in Numbers
Lesson 1:
When bigger is better
Summary:
This lesson introduces sample size as a rigor problem, using an example study to go through the various issues that can arise when a sample size is too small. It also presents the concept of effect size and how to use it as a tool for determining an appropriate sample size. In the example, data from multiple sample sizes will be collected, analyzed, and interpreted to demonstrate how n can impact the results of a study. The rest of the unit will provide additional strategies that can be used to calculate sample size with rigor.
Goal:
- Understand the challenges that arise when a sample size is too small for the needs of a study.
- Recognize the importance of effect size in selecting an appropriate sample size.
- Interpret the effects of sample size on the analysis of a dataset.
1.1 Getting started with an example study: Protein X and longevity
Many problems in scientific rigor trace back to one deceptively simple issue: sample size. In science, the number of observations we collect can determine whether a result reflects a real phenomenon, random chance, or even a misleading conclusion. Yet determining the “right” sample size is rarely straightforward. To understand this problem, let’s explore an example study from start to finish.
Figure 1. Representative distribution of Protein X expression levels in the population. The highest levels of expression (orange circle) are associated with a lifespan five years longer than the average population.
We begin with a hypothesis: Protein X is a biomarker for longevity. Expression levels of Protein X have been shown to follow a normal distribution in both human and animal populations. Current research suggests that higher levels of Protein X predict a longer lifespan, with one paper suggesting individuals with high expression may live up to five years longer than average.
An extra five years of life could be a major finding! Our research team collaborates with a biobank that allows us to cross-reference individual samples with medical records and death registries. To begin exploring this hypothesis, we propose a pilot study of n = 20. We can do this quickly, without too much time or cost.
Figure 2. Relationship between Protein X expression and lifespan in a pilot study of n = 20. Relative expression levels of Protein X are standardized to the population average (x=0). The line of best fit shows a positive relationship that supports the hypothesis that Protein X is a biomarker for longevity.
After obtaining and assaying samples, we perform a linear regression to test the relationship between Protein X expression and lifespan. This looks promising! The pilot study suggests support for our hypothesis. With exciting preliminary data in hand, we decide to move on to a full study. This must be conducted with different samples than those used for the pilot study, and we should choose a sample size that can capture this relationship accurately.
Deep DiveWhat is Linear Regression?1.2 The importance of effect size
One of the first questions asked when designing an experiment is “How many?”. It might be tempting to just pick a number that feels right, but there are important considerations to establish in order to know what “right” really is.
We should think carefully about the question we want to answer and what kind of answer we will be able to obtain. What are the possible outcomes of this research? What will it mean if there is an effect? If there’s no effect? Does it matter how big the effect is? This is one of the most essential questions when deciding sample size, and it is often overlooked.
Determining an estimated effect size is an important part of rigorous statistical planning and analysis. The estimated effect size is a quantitative measurement, often derived from a pilot or previous study, that serves as an indicator for how strong a relationship between experimental variables might be. Described mathematically, the effect size is the magnitude of the difference or association between one or more groups.
TipUnits of effect sizeStandardized calculations of effect size, of which there are many, are useful when working with outcomes that are abstract (e.g. Likert scales) or need to be compared across studies that apply different units of measurement. Because these are field specific and can be difficult to conceptualize, in this unit we will focus on effect size as it relates to biological significance.
This means rather than thinking about effects in conventional categories of small, medium, or large, etc., we will discuss them with units that have intrinsic meaning. For example, in the Protein X study, when we express effect size in years of life, we can better grasp the value of a 1 year gain compared to 5 or 10 years. Even “large” standard effect sizes can produce biologically trivial results if they do not have functional consequences. This lesson emphasizes the importance of interpreting effect sizes intuitively, alongside their biological context.
Using "How big?" to decide "How many?"
Effect size estimations are used when determining sample size as a way to ensure the study has enough subjects to detect an effect of interest. This has major implications for the n of a study. In simple terms, the bigger the effect size is, the easier it is to find, and thus the fewer subjects needed to find it. Conversely, the smaller the effect, the larger the sample size needed to reliably detect it. In this unit we will discuss many reasons why small sample sizes are not ideal, a major factor being that small samples have a high risk of being insufficient to detect a small effect. Similarly, an overly large sample size can yield a statistically significant result for an effect size too small to be practically relevant.
Effect size estimations can be determined from several sources:
Published records It is useful to begin by exploring the literature. Has an effect size been previously published? Are the methods involved similar enough to be comparable to our work?
Prior measurements Preliminary data and pilot studies are a common source of effect size estimates and are especially representative when derived from the same source as the current project.
Future goals If an effect size has not been previously reported, we can select a target effect size and design the study to be able to detect it.
Whether based on prior research or the current project, the effect size is invaluable for determination of sample size, and gives us an idea of where to start. For a more detailed look on effect size estimations and calculations, see our upcoming unit on Effect Size!
Other considerations for sample size
In addition to having an idea of what effect size we are looking for, we also need to consider and plan for unexpected circumstances.
- Is the sample size achievable within any outside limitations (e.g., funding availability, ethical restrictions, etc.). If not, what effect size can we detect with adequate power?
- Do we expect attrition or other exclusions? If attrition is expected in the course of the study, sample size must be proportionally increased. Outliers are also a common source of exclusion and should be planned for in advance.
Activity: Analyze the data
We have an idea of what our effect size will be from published literature and our pilot study - about 5 years of life. Using this estimated effect size, we do a web search and find an online tool that tells us we need n = 100. In this activity, we will conduct the study with the recommended sample size and interpret the results.
Post-activity questions:
- What stood out to you while interpreting these data?
- What questions do you still have about the conclusions?
- Do you have ideas for what could be done differently?
Want to try this yourself in Python? Check out the code below!
# Set a sample size to draw from the example dataset
N = 100
# Load modules
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import pearsonr
import statsmodels.api as sm
# Load custom functions
import requests
url = "https://raw.githubusercontent.com/c4r-io/BU-METER-Units/main/sample_size_functions.py"
response = requests.get(url)
exec(response.text)
x,lifespan = load_data(N) # ... and load the data with this many N.
#plot the data
plt.figure()
plt.scatter(x,lifespan)
plt.xlabel('Genetic biomarker x')
plt.ylabel('Lifespan (years)')
plt.show()
# Estimate a line from the data.
from statsmodels.formula.api import ols
dat = {"x": x, "lifespan": lifespan}
regression_results = ols("lifespan ~ 1 + x", data=dat).fit()
# Estimate the slope and p value
slope_estimate = regression_results.params.iloc[1]
print('Slope estimate = {:.3f}'.format(slope_estimate))
print('Standard error of slope estimate = {:.3f}'.format(regression_results.bse['x']))
print('p-value = {:.3f}'.format(regression_results.pvalues.iloc[1]))
# Prepare to visualize the predicted line
pred = regression_results.get_prediction().summary_frame()
mn = pred['mean']
ci_low = pred['mean_ci_lower']
ci_upp = pred['mean_ci_upper']
# And plot it.
indices_sorted = np.argsort(x,0)
plt.figure()
plt.scatter(x,lifespan)
plt.plot(x[indices_sorted],mn[indices_sorted], 'r')
plt.plot(x[indices_sorted],ci_low[indices_sorted], ':r')
plt.plot(x[indices_sorted],ci_upp[indices_sorted], ':r')
plt.xlabel('Genetic biomarker x')
plt.ylabel('Lifespan (years)')
plt.show()1.3 Interpreting ambiguous results
Despite the promising results we found in the pilot study, it appears there is no effect when repeated with n = 100. However, while exploring the analysis in the activity, questions emerged that challenged our certainty in the rigor of this result. Let’s review what these variables really tell us.
Figure 3. Relationship between Protein X expression and lifespan in a sample of n = 100. Relative expression levels of Protein X are standardized to the population average (x=0). Shaded regions indicate the 95% confidence intervals. Analysis does not support the hypothesis that Protein X is a biomarker for longevity.
The analysis tells us:
- Slope = 0.9
The slope indicates that for every unit increase in the expression of Protein X, there is an increase in lifespan by 0.9 years. Taken over the range of Protein X expression (middle 95% interval of -2 to 2) gives us a measured effect size of
4 x 0.9 = 3.6 years
This is smaller than the 5 years estimated from the literature, but could still be impactful.
- Standard error of the slope = 1.0
The standard error of the slope is a measure of precision that provides us with our 95% confidence intervals. This represents sample to sample variability and indicates an uncertainty in the magnitude of the slope by ± 1. This leaves a slope of 0 (a null result) within the margin of error.
- P-value = 0.37
The p-value is used to test the null hypothesis that the slope of the regression line is zero (no relationship between Protein X and lifespan). A statistical test provides the final nail in the coffin - this result is not significant, meaning any possible relationship may only be due to random chance rather than a true effect.
With this n = 100 sample, we found a smaller than expected effect size that statistical analysis suggests may be the result of chance, meaning there is not enough evidence to reject the null hypothesis. However, the confidence intervals determined by the standard error tell a more complicated story. A null result (no relationship) is possible - but so is a true positive or negative relationship.
Figure 4. Range of possible relationships between Protein X and lifespan based on the shaded 95% confidence intervals. A) A slope of 2.9 predicts a positive relationship. B) A slope of 0 predicts no relationship. C) A slope of -1.1 predicts a negative relationship.
This means that we cannot say with certainty whether there is or is not an association between Protein X expression and longevity. These data are inconclusive!
What do we do now? Could a problem in the experimental design have contributed to the uncertainty of these results? After consulting with a statistician, we learn that our sample size estimate was way off. They walk us through their calculations and show that for this project, we would need at least n = 1000 to be able to reliably detect any effect. This means our study of n = 100 was underpowered. In this case, the online calculator used did not lead us to the correct sample size because it was not designed for the type of analysis we needed to perform, and we didn’t know!
Sample size estimations aren’t always perfect, but for any calculation it is important to understand why and how an estimate might not be right for the study. Collaborating with a statistician is one of several ways to gain this understanding. In this unit, we will uncover some of the hidden assumptions that go into sample size estimation and provide tools to help make this choice carefully and rigorously.
1.4 Choosing the right effect size
After performing a follow-up with a full sample size of n = 1000, we find a conclusive effect.
Figure 5. Relationship between Protein X expression and lifespan in a sample of n = 1000. Relative expression levels of Protein X are standardized to the population average (x=0). Shaded regions indicate the 95% confidence intervals. Analysis supports the hypothesis that Protein X is a biomarker for longevity.
This analysis tells us:
- Slope = 0.8
The slope indicates that for every unit increase in the expression of Protein X, there is an increase in lifespan by 0.8 years. Taken over the range of Protein X expression (middle 95% interval of -2 to 2) that gives us a measured effect size of
4 x 0.8 = 3.2 years
This is even smaller than the effect size predicted by our n = 100 sample. However, our sample size is large enough now to have confidence in our detection. The other variables will tell us if this is a true effect.
- Standard error of the slope = 0.2
Sample to sample variability indicates uncertainty in the magnitude of the slope by ± 0.2. Using the correct sample size shrunk the standard error by 80%. This excludes a possible null result from our margin of error.
- P-value = 0.001
A statistical test reveals these results are significant, meaning the relationship is unlikely to be due to random chance.
With this n = 1000 sample, we have a conclusive positive result. Increased expression of Protein X is associated with a longer lifespan, with a measured effect size of about 3 years. Had we stuck with the n = 100 dataset, we would have missed this.
The issues demonstrated in this lesson reveal just a few of the rigor problems that can be caused by small sample size. Sometimes, we may not check carefully when we know our options are limited. This is a mistake. If we had known that our effect size of interest would likely go undetected, we could have explored the smallest effect we could detect with adequate certainty, given our limited n. Then, we would have had a basis to evaluate whether to proceed.
1.5 The consequences of using a small sample size
In the rest of this unit, we will learn the ins and outs of sample size determination, how to calculate it correctly, and how to proceed with rigor when reading or reporting an underpowered study.
Though constraints on our sample size are real, before choosing a less than ideal sample size, we must understand the dangers of:
- Inconclusive results: When confidence intervals include all possible hypotheses and alternatives.
- The winner's curse: When results that pass a significance test have an inflated effect size.
- Low power: When significance tests fail to detect a true relationship or difference.
- False discoveries: When significance tests falsely detect a relationship or difference that is not real.
This list is not comprehensive. For additional risks that can be associated with improper sample size, see our units on Controls and P-values.
1.6 Methods for sample size estimation
This unit presents a workflow with several options for planning and performing sample size estimation. Importantly, this should occur before the study begins and be developed alongside the statistical analysis plan (SAP). The SAP will not be covered in this unit, but see our forthcoming curriculum on Statistical Analysis Planning for more!
Scientists spend enormous effort developing new tools and technologies, but even the most sophisticated experiment can produce misleading conclusions if the sample size is poorly designed. A little care in planning sample size can go a long way!
Takeaways:
- Using an inappropriate sample size can produce datasets that give false or inconclusive results.
- Effect size estimations can be used to more accurately select a sample size and improve rigor.
- Confidence intervals show the margin of error and can be used to determine whether results are conclusive.
Reflection:
- What are the tradeoffs between using a small and large sample size?
- What kinds of effect sizes do you work with in your own research?
- How would you convince a colleague that a large, fully powered sample size is worth the potential added time and cost?
Lesson 2:
Sample Size and Significance Tests
Summary:
In this lesson, sample size will be discussed as it relates to rigorous significance testing. How does a chosen sample size influence the results of a statistical test? What happens if the sample size is not appropriate for the effect being measured? Statistical power will be introduced as a framework that can be used along with estimates of effect size to properly calculate the sample size for a study. Finally, these concepts will be presented together in an interactive example that demonstrates how sample size may contribute to a correct or incorrect significant or non-significant result.
Goal:
- Recognize the benefits that a larger sample size provides for significance testing.
- Understand how sample size can limit the reliability of significance tests.
- Explore the ways statistical power, effect size, variance, and sample size influence the interpretability of outcomes.
2.1 What is sample size?
Sample size refers to the number of experimental units used in a study, which may be divided into one or more experimental groups. Depending on the context, the term sample size may be used in different ways. This requires some unpacking:
Experimental unit
An experimental unit is the smallest entity that can be independently assigned to an experimental group or condition (a.k.a. biological replicate). This may include humans, animal models, cell culture wells, neurons, observations, etc.
An experimental unit is:
- A human or animal assigned to a treatment group
- A cell culture well given a particular treatment
- Anything that can be given a treatment independent of others of the same unit
An experimental unit is not:
- A measurement performed multiple times on one subject
- A cell or group of cells within a cell culture well
- Anything that cannot be given a treatment independent of others of the same unit
Correct identification of the experimental unit is essential for proper sample size determination. Pseudoreplication occurs when n is inflated by counting measurements instead of experimental units. For example, if a treatment is administered to 3 cell culture wells, and 5 neurons are recorded in each well, the experimental unit is the total number of independent wells treated, not the total number of neurons recorded. Within a well, each neuron recorded is dependent on a single administered treatment; therefore they cannot be considered independent units.
One or more groups
Sample size may be further abbreviated depending on whether working with a single or multiple groups. When presented as N, the size refers to the total in the entire study, whereas n refers to the size of a single experimental group. If a study only has one experimental group, such as with our Protein X example, N and n may refer to the same thing.
While these definitions are standardized, their application is not uniform across fields. Therefore, it is always best practice to clearly define the type of experimental unit as well as the total (N) and group (n) sizes. Ultimately, sample size is not just a number; it is a cornerstone of experimental design that ensures data will be robust enough to support or refute a hypothesis.
Deep DiveWhat are we counting?Consider the following example: We have 2 ravens and are recording 100 neurons from each. Is the sample size 2 ravens? 100 neurons? 200 neurons? The answer depends on the experimental design. Review each case below:
Case 1:
n = 2 ravens
Prior to recordings, each raven receives an injection that induces an inflammatory response. Following the injection, activity from 100 neurons in the sensory cortex is recorded. Because the injection is administered once per raven, it is the raven, not the neuron, that is independently manipulated. The 100 recordings are repeated measurements within that single manipulation. The findings of this study are able to precisely describe each raven’s response to the manipulation, but they are not able to generalize to the effect of the manipulation on ravens broadly.
Case 2:
n = 100 neurons per raven
Each raven receives an implanted electrode that simultaneously stimulates and records from discrete neurons. In this case, 100 stimulations are independently performed on 100 different neurons in each raven. The findings of this study can precisely describe the response of single neurons to the manipulation. Repetition of the experiment in the second raven gives a sense of how consistent the observed response could be, but cannot generalize to the response of neurons in all ravens.
Why not n = 200?
In this scenario, n = 200 would be considered pseudoreplication regardless of the case. Combining both ravens' recordings into a single pool for analysis requires assuming that no meaningful variance exists between animals, which cannot be demonstrated with only 2 ravens. Even if all neurons are independently stimulated, every neuron in the sample shares one of two physiological contexts. Two hundred remains an accurate description of how many neurons were recorded, but does not represent the true sample size.
2.2 What’s wrong with n = 1?
In Lesson 1, the sample size influenced whether or not we were able to detect a true effect in our data. The smaller the sample size, the more likely this is to occur. This is because any single observation could be misleading or atypical. These are called outliers and are problematic when one or two extreme recordings have an outsized impact on estimates. Adding data points allows us to be more confident about our expectations for future observations under the same conditions. Consider the two graphs below:
Figure 1. In studies with small sample sizes, outliers have an extreme impact on the interpretation of data. (Top) With n = 3, there appears to be a perfect linear relationship between the measured variables. (Bottom) With n = 20, it is clear that one point from n = 3 was an outlier (orange diamond) and the true relationship is quite different.
In the first graph, there are three points that appear to form a perfect linear relationship. If these were three data points, it would suggest there is a strong relationship between the two variables graphed.
However, in the second graph it becomes apparent that one of these points is an outlier, and most of the measurements are distributed between the first two points. This shows how having more data gives a more representative snapshot of the population. Both the relationship between variables and the variability of that sample population are more clearly depicted.
With more data, we can have more confidence that the results of the experiment are real.
2.3 Significance testing
In research, the most common statistical analysis involves significance testing. This is a statistical framework that contrasts the null hypothesis with an alternative hypothesis, usually via a p-value threshold. If a statistical test shows a result is significant, it indicates there is sufficient evidence to reject a null claim at the given threshold.
Careful - there are many assumptions we should be aware of when discussing whether a result is significant or not:
- A non-significant result does not mean the null hypothesis is true. Rather, it only indicates that we have insufficient evidence to prefer the alternative. This is commonly summarized as “absence of evidence is not evidence of absence”. Therefore, significance testing is not designed to be interpreted as whether or not we can "accept" the null but whether we can “fail to reject” it.
- A significant result also does not mean we accept the alternative hypothesis. Rather, it indicates the chance of getting these results would be unlikely if the null were true. Therefore, significance testing only allows us to reject the null hypothesis.
Significance testing is not foolproof. As we saw in the previous lesson, a true effect can produce a non-significant result when the sample size is too small, meaning it was unable to be detected under the conditions used. There are two types of errors that cause us to incorrectly reject or fail to reject the null hypothesis.
- Type I error (False positive): This occurs when we wrongly determine that a difference or relationship is present. In other words, random or unrelated variation is interpreted to be relevant or causal.
- Type II error (False negative): This occurs when we wrongly determine that a difference or relationship is absent. In other words, relevant or causal variation is interpreted to be random or unrelated.
Significance testing is primarily designed to limit the probability of Type I errors. Additional measures are necessary to reduce Type II errors, and this is where the role of sample size in rigorous significance testing is most relevant.
Deep DiveNull Hypothesis Significance Testing (NHST)2.4 Specificity and sensitivity
Type I and II errors convey how reliably a test detects true signals and rejects false ones. This measure of distinguishing signal from noise can also be presented in terms of specificity and sensitivity.
When we keep type I errors to a minimum, we can say that our test is specific. When we set p<0.05, if the null hypothesis is true and we test repeatedly, we should correctly “fail to reject” the null at least 95% of the time. Only <5% of true null results should be incorrectly rejected. This is the foundation of most tests of significance. A threshold of p<0.05 or lower is accepted as standard practice to minimize false positives. Specificity should be set at the start of the study and is generally independent of sample size.
When we keep type II errors to a minimum, we can say that our test is sensitive. When a difference or correlation is true and we test repeatedly, the percentage of the time we should correctly reject the null hypothesis is described as the statistical power of the test. This is not a feature of significance tests themselves but should be considered during experimental design. By maximizing statistical power, and thus sensitivity, we are able to minimize false negatives.
Ideally, our experiments and subsequent analyses should be both sensitive and specific. However, in real-world experiments there are tradeoffs that make this relationship complicated.
2.5 Designing experiments with statistical power
Statistical power is the probability that a test will correctly detect an effect. Higher power reduces the risk of a Type II error, where a real effect is missed. When designing experiments, it is our job to calculate a sample size that is powered to be able to yield statistically significant results, enabling us to test our hypothesis with precision.
Larger samples provide more information about the population, leading to more precise estimates of the population parameters. This precision reduces the standard error and widens the gap between the null hypothesis and the alternative hypothesis, making it easier to detect significant differences. Therefore, increasing the sample size typically increases the power of a statistical test.
Choosing a statistical power of 80% is a common convention in many fields of research, particularly in the social and biomedical sciences (Cohen, 1977). This means there’s a 20% chance of failing to detect a true effect (i.e., of a Type II error). With this guidance, our goal then is to determine the smallest sample size possible that can provide 80% power. This is considered a good trade-off between increasing precision and controlling operational constraints associated with larger sample sizes.
Let’s examine statistical power in the framework of the Protein X study in Lesson 1. With n = 100, we failed to detect a significant relationship between Protein X and lifespan. This failure could have occurred because of either a:
- Null result. There was no relationship between Protein X and lifespan.
- Type II error. There was a relationship between Protein X and lifespan, but we did not collect enough data to detect it.
Many true effects will go undetected due to type II errors when tests of significance are underpowered, which is exactly what happened here. The statistician showed us that the actual power of the n = 100 study was 0.15, meaning we only had a 15% chance to detect the relationship between Protein X and lifespan. Or in other words, if we had repeated the study with n = 100 another fifty, one hundred, or a thousand times, 85% of those experiments would have failed to detect an effect. This shows the more sinister downside of operating with low power: unless incredibly lucky, underpowered studies are wasteful and require more resources to detect true effects in the long run. To learn more about statistical power and performing traditional power calculations, see our upcoming unit on Effect Size.
Statistical power in the wild
Aiming for 80% power is a widely adopted convention that is considered best practice for rigorous experimental design. In the real world, however, we find that research falls short of this threshold the majority of the time. A report by Button et al. (2013) suggests that the average power of most neuroscience studies is 30% or less. This means quite a lot of true effects may be getting missed due to type II errors! And, as we shall see shortly, low power can have consequences for positive results as well. This is a widespread problem that has a major impact on scientific knowledge.
Under-powered studies can occur for a variety of reasons. In some cases, there are real limitations that prevent us from being able to use the sample size we need! We’ll talk more about trade-offs and communication under limited conditions in Lesson 8. But also, this is an area where it is quite common for scientists to simply make mistakes, choosing a sample size based on convention or intuition or applying a calculation method that isn’t appropriate to the data and analysis at hand. Once we fully understand the implications of sample size, we will walk through methods for estimating n that can be used for any statistical analysis (Lessons 3-6) as well as tips and tricks for avoiding mistakes when applying other resources.
2.6 Looking at distributions
To see how sample size, type II errors, and statistical power come together, consider an example involving measuring the mean firing rate of two groups of neurons, A and B. There are two possible scenarios:
TipEffect size as the difference between two meansNotice there is overlap in these two distributions. Significance tests typically avoid falsely rejecting H0 (type I error) by setting a threshold (ɑ) so that when the null distribution is true, only 5% of samples will be mistakenly rejected. However, if we follow that threshold into our Ha distribution, we can see a problem. Only the right half is detected by correctly rejecting H0 when Ha is true. Everything to the left will be a type II error (β). If half of the distribution is at risk of being a false negative, this test will lack the power to detect half of the likely outcomes when Ha is true.
Low power and positive results
Missing a true effect is not the only danger in this example. There are hazards for positive results as well!
What if an observation is made that falls within the above probability of detection? Since this observation is above the cutoff, it will almost certainly be higher than the true effect size. When we only pay attention to “significant” effects - those from the right side of the Ha distribution - we will always overestimate the true average effect size.
This phenomenon is known as the winner’s curse. When Ha is true and an observation exceeds the cut-off for an under-powered test, it results in the perceived effect size being inflated compared to the true effect.
In the example above, alpha is set at 5%, which means that if H0 is true, a difference of that magnitude would only be observed about 5% of the time. As we’ve established, though, when that cutoff leaves us with 50% power, the same difference would still only be observed about 50% of the time if Ha is true.
Now, let’s extend that reasoning to studies with even lower power. What if the alpha cutoff means we only reject the null hypothesis 10% of the time when Ha is true? What if our power is as low as 5%? This may be extreme but it illustrates this point nicely: at that point, a positive result is equally likely for both distributions. We reject H0… but interpreting this as support of the Ha distribution would be a major reasoning error.
When power is low and results are positive, it is likely that those positive results may be false!
Activity: Statistical Power Visualization
In this activity, we will use the above example of mean firing rate in groups of neurons to observe the role of sample size in generating studies that are powered to minimize type II errors. Pay attention to how the power changes with differences in variability, effect size, and sample size.
Post-activity questions:
- What surprised you about how each component altered the ability to detect an effect?
- What is the influence of variability in determining statistical power? Why is this important from a biological perspective?
- How can we account for all these components when we haven’t done the study yet?
Deep dive
False positives and long-shot tests
As we've seen, when power is low AND results are significant, the alpha value we set can give us a false sense of security. Even though it's true that there is a very low probability (e.g. alpha = 0.05) that our significant result would occur when the null distribution is true, when we compare this with the probability when the hypothetical distribution is true (beta), the chances may be about the same! When power is low, significant results will be rare - this might make us think that they are more trustworthy when they occur, but in fact the opposite is often true! Significant results from low-powered tests suffer from both inflated effect sizes (when true) and a high likelihood of being wrong.
This problem has a particularly tangible impact in areas of research where many tests are performed, each of which has a low chance of actually having any effect, like screening for drug candidates for genes. First off, we should recognize that these fields tend to have very specific statistical practices that are intended to avoid precisely the kinds of problems we'll discuss here. However, we think this example is useful because it also helps to think about the impact of false positives more broadly.
In this context, it can be useful to have a metric that tells us the probability that a significant result is actually a true positive. This metric has a name: the positive predictive value (PPV).
Let's imagine an example in early stage pharmaceutical testing. For the purposes of making our visualization clear, we'll go with a generous estimate that 20% of our candidate drugs might have a real effect. In the figure below, you can see a pie chart showing that 20% in white, with the expected ineffective candidates (80%) in grey. Now, let's look at what proportion of those outcomes are captured by "significant" results. Here, we start with a conventional value for alpha and a small sample size. First, try shrinking alpha to a more stringent value. PPV should look better! But… what is the power of the test? What proportion of true effects (white) are being detected (purple)? Now, play with the sample size slider.
One standard practice in statistical scenarios like this is to use a more stringent alpha. However, if that's the only change we make, our PPV gets much higher, but the vast majority of candidates with a real effect are left undetected, with power getting lower as alpha shrinks. Adding more samples improves our positive predictive value AND our power - both increasing the proportion of significant results that are true positives, and increasing the proportion of true positives that are detected with a significant result!
2.7 Bringing it all together
In this lesson, we have explored the many, many factors that play a role in the reliability of tests for significance and how sample size connects to each of these. We have demonstrated that sample size can be a determining factor in whether positive results will be incorrectly missed or negative results will be incorrectly detected. And we have seen how sample size can be properly applied to produce rigorous, conclusive results. Now it is time to go from concepts and practice to meaningful applications.
The examples we have worked with so far have been relatively simple. However, increasing sample size improves sensitivity without sacrificing specificity at a given effect size in more complex analyses, too. To calculate n rigorously, we need a goal for detecting effect size and some expectation of the variance we will be working with. In the next lessons, we will learn methods with and without pilot data that will allow us to choose a sample size with confidence!
Takeaways:
- Data from large sample sizes are less likely to be skewed by outliers, providing a more accurate representation of the distribution and variance of a population.
- Statistical tests with large sample sizes have greater power, making them less likely to produce false negatives.
- Effects are less likely to be detected by significance tests with small sample sizes, and when they are detected they yield inflated effect size estimates.
- Sample size, effect size, variance, and statistical power all contribute to the reliability of statistical tests and the likelihood of type I and type II errors.
Reflection:
- What sample sizes are standard for your field? How well-powered are these studies?
- Have you ever been skeptical about the results of a significance test due to a small n?
- Name one way you think differently about your data after exploring how sample size and significance tests are related.
Built with Shorthand