1.1 The role of statistical analysis plans (SAPs) in research

When should you stop data collection? What qualities determine an outlier? How effectively can another researcher (or your future self!) apply your methods? Answering these questions prior to the start of a study helps safeguard against bias and the flexibility in research choices that might degrade the integrity of your findings. 

Advanced planning of analyses supports transparency, reproducibility, and rigor across all stages of research by mitigating the opportunity for swapping analytical approaches based on preliminary results. Specifically, a statistical analysis plan (SAP) documents the statistical methods and techniques that will be used to analyze data. 

These plans explicitly outline primary and secondary outcomes, covariate adjustments, and specific statistical tests in advance. This transparency ensures that all planned analyses are reported, regardless of the direction or significance of the results.

While these plans are most often required in larger clinical trials, the utility of pre-specified analysis plans applies to all research from early exploratory and pre-clinical work to clinical and applied studies. 

In laboratory and translational science, where sample sizes are often small and variables are tightly controlled, SAPs prevent the over-interpretation of chance findings. In applied and observational fields, they help researchers navigate complex datasets without introducing post-hoc rationale. 

Ultimately, adopting an SAP across all tiers of research fosters global scientific trust, ensuring that conclusions are driven by robust, pre-planned methodology rather than chance.

1.2 Risks of unplanned analyses

But what happens when a study is run without advanced planning? In the absence of a pre-specified analysis plan, the boundary between planned confirmatory research and exploratory data analysis quickly blurs.

Without specific decisions established prior to data collection, research is highly vulnerable to cognitive biases and accidental methodological flaws (see our unit on Confirmation Bias for more on biases and researcher degrees of freedom). When statistical analyses are decided after data look-offs, the validity of classical hypothesis testing breaks down. This transforms what should be a rigorous test of a specific hypothesis into an exercise in pattern-seeking.

To illustrate these risks, consider a hypothetical mouse memory study.

However, the experimental reality diverged from the initial design:

• The pre-specified memory outcome did not show evidence of an intervention effect.
• The primary analysis yielded a non-significant p-value, failing to reject the null hypothesis.

Faced with a null result for their primary hypothesis, the research team shifted their approach. After reviewing the initial results, the team examined 10 additional behavioral outcome measures captured during the experiment, like open-field activity, maze navigation metrics, and grooming behaviors.

Using this new subset of data, multiple statistical tests and model specifications were explored until a statistically significant result was identified.

The researchers altered covariates, switched between parametric and non-parametric tests, and tested various subgroups. Once a sub-analysis finally yielded a p < 0.05, iterative testing stopped.

This significant finding was then emphasized as evidence of an intervention effect without distinguishing it as exploratory or post-hoc. In the final write-up, the narrative was reframed around this single positive result, presenting it with the same confidence as if it had been the primary hypothesis all along.

Why these problems matter
The accumulating issues demonstrated in this scenario undermine reproducibility because each additional outcome and analysis creates another opportunity to obtain a statistically significant finding by chance. 

These choices also severely compromise the statistical power of the experiment. Behavioral outcomes may have expected effect sizes and variability, so a study powered for the originally planned outcome may not be adequately powered for every additional outcome. 

Power calculations depend inherently on the specific variance, effect size, and sample size of the primary metric. When researchers pivot to alternative behavioral outcomes, they frequently test metrics for which the study is severely underpowered. This leaves the study unable to detect true effects or, conversely, exaggerates the effect sizes of any random fluctuations that happen to cross the significance threshold.

When planning is deferred until after data collection, the objectivity of the scientific method is severely compromised and decisions are often made: 

• In response to observed results rather than an a priori hypothesis.
• Inconsistently across analyses such that methods may differ between experiments or cohorts .
• Without documentation or justification, obscuring the highly exploratory nature of the work.

This lack of early planning creates risk of these Questionable Research Practices (QRPs):

P-hacking: Running multiple analyses (often changing criteria for how inclusion, outliers, or exclusion is handled post-hoc) or manipulating data until significant p-values (p < 0.05) are found.

HARKing: Constructing a post-hoc hypothesis after analyzing the data and presenting unexpected or accidental findings as if it were the primary planned prediction all along (Kerr, 1998).

Selective reporting: Omitting non-significant results or selectively publishing only outcomes and data that validate a preferred theory. This creates a highly skewed body of scientific literature, making it hard for future researchers to see the real boundaries and variation of what they are studying.

Inappropriate test selection: Selecting inappropriate statistical methods, ignoring core data assumptions, or picking a misaligned test solely because it yields a more favorable output.

Each QRP reflects decisions that were not explicitly made or documented during planning.

1.3 Benefits of pre-planning

So, what can we do to help mitigate these questionable research practices?! Pre-plan! Explicitly documenting your analytical intentions prior to observing your data improves research transparency, rigor, and reproducibility.

Let’s revisit the mouse memory study now run with advanced planning!

The team specifies:

After data collection, the pre-specified analyses are conducted as planned and any alternative strategies used are noted for transparent reporting. The primary result is also interpreted and reported regardless of statistical significance. Similarly, secondary findings are interpreted in light of their effect sizes, precision, multiplicity, and statistical power. Additional analyses can still be performed, but they are clearly identified as exploratory or post-hoc.

Why pre-planning analyses matters:

Enhances efficiency: Avoids unnecessary data collection, repeated experiments, or “false starts” (studies that must be repeated because outcomes or methods weren’t appropriately defined).

Protects data through methodological rigor: Prevents bias and misleading conclusions caused by post-hoc, data-driven analytic decisions. Distinguishes confirmatory proof from exploratory evidence.

Improves transparency, credibility, and reproducibility: Supports relevance by ensuring that final analysis strictly aligns with original study goals.

Provides appropriate context for accurately interpreting statistical significance.

Let’s apply some of these principles!

Researchers may skip advanced planning for a number of reasons. Sometimes these reasons amount to limitations like a lack of training or mentorship or time pressure under tight deadlines. 

In more concerning circumstances, however, researchers might believe their small or exploratory studies may feel too “minor” to require planning or, worse, they may underestimate consequences of post-hoc decisions.

Before we explore ways to incorporate planning into your own research, let's review the actions of an unplanned analysis contrasted against a planned approach.

Planning in advance can mitigate the most problematic post-hoc decisions that would cause unreliable results or false-positives.

1.4 How should you incorporate planning in your research?

Planning applies across all types of research, spanning human, animal, and cell-based studies, and it applies across all research stages (from pre-clinical, pilot, simple experiments, clinical trials, etc.).

As you plan your own research, think carefully about all the necessary components and potential decisions you may have to make, and identify what you will do in advance.

Overall, you will want to:

• Identify your study goals and hypotheses clearly.
• Define primary and secondary outcomes before data collection. 
• Choose appropriate statistical tests for each outcome.
• Decide how to handle missing data, exclusions, outliers, and multiple comparisons.
• Document exploratory analyses separately.
• Communicate your plan with collaborators early.

While you would ideally plan exploratory searches in advance, not all unplanned analyses are inappropriate, but they must be described transparently.

The issue is not exploration itself, but presenting exploratory or post-hoc findings as if they were planned in advance misrepresents how a search was conducted and creates room for both error and bias to impact your choices simply because they won’t be reported.

2.1 Theory meets reality

Behavioral changes are often easier to discuss in the abstract than they are to translate into daily praxis. This is because integrating new practices is rarely a purely moral calculus – rather, it is often a process shaped by the real constraints governing scientific work.

Constraints such as:

Many of these constraints are systemic and may be largely outside of our individual control.

While we may not be able to remove these constraints ourselves, we can account for them when integrating rigor practices into our workflows. This mitigates the incongruence between theory and praxis and minimizes the perceived cost of rigorous science by grounding it in the daily realities of scientific work.

However, doing so effectively requires that we first acknowledge the complex reality of modern research timelines

2.2 Planning for iteration

It's not uncommon to conceptualize research workflows in highly idealized, linear terms:

Yet in practice, such models rarely hold true. This is because as a project proceeds, things like unexpected data, methodological hurdles, and guiding analyses can often demand mid-process adjustments to our workflows, such as modifying measurements, troubleshooting methods, and exploring additional outcomes.

In other words – research is often iterative, and the orderly, linear timeline above actually looks more like this when put to practice:

In order to minimize the implementation costs associated with the type of planning for which this unit advocates, our planning processes need to fit into that reality by anticipating iterations while still keeping core decisions pre-specified.

2.3 Foundations and flexibilities

Facilitating iteration requires that our planning processes support some degree of flexibility, which we can ensure by scoping the various elements comprising our plans and classifying the elasticity of the analytical decisions involved according to the relationship they have with our broader research aims.

Note that here, scientific aims refers to both the essential scientific questions the project poses and procedural questions that arise as a consequence of your methodology and define its epistemological validity.

Scientific aims can include questions like:

Each of your scientific aims should be feasibly mapped to specific planning decisions you will need to make at some point during your project:

Once we have a robust sense of how different planning decisions relate to each scientific aim in our project, we can divide them into three broad categories:

• The core components, which would represent foundational pivots in the project’s central aims if changed.
• The flexible components, which may be dependent on the outcomes that precede them.
• The exploratory components, which are supplemental, of secondary or tertiary interest, and are mostly independent from the rest of the project.

Let’s first focus on the former – the core elements of analysis planning. These are the aspects of your plan that tend to define your primary outcomes and analyses, such as:

• Your research question(s) and associated hypotheses.
• Your study design: E.g. randomized, longitudinal, cross-sectional, exploratory, etc.
• Your main outcome(s): In pre-clinical or exploratory studies, this may be identifying the key measurement or variable that best aligns with the research question. In clinical studies, these are often referred to as “primary outcomes”.
• Inflexible or static components of your methodology: Such as, sample size/power, replication strategy, and/or statistical tests/methods for each outcome.

Meanwhile, the flexible elements of your plans are generally context-dependent:

The distinction between those elements that are foundational to your study and those that must remain flexible can be trickier to make than these abstract lists might suggest. So let's illustrate this idea in a real-world context by taking a look at how they apply in an example study:

Distinguishing between exploratory analyses and pre-specified analyses helps ensure that results are not misinterpreted and that appropriate caution is taken when interpreting findings. However, the time may come when even inflexible elements of our plans push up against the unruly constraints of daily work. Thus, a flexible planning process should ideally encode some degree of prioritization – that is, which aspects of your work are most important to plan out, and notably, to keep tethered to the original plan.

2.4 The greater good

Once you’ve developed a robust sense of your plan’s epistemological scaffolding, it can be prudent to revisit how your project fits into the knowledge generating pipeline. Where the component scaffolding describes the relationships between the various decisions involved and the validity of your project’s intellectual contributions, your work’s position in the broader wheel of scientific enterprise determines which components will be most important to the broader research community.

For example:

• Exploratory studies should focus on feasibility, pilot outcomes, and generating specific hypotheses to test.
• Confirmatory studies should focus on defining primary and secondary endpoints, as well as key covariates.
• Clinical trials should focus on pre-specified plans and regulatory requirements.

2.5 Bespoke priorities and crisis management

Once you know which step in the pipeline your project is targeting, you have the second of the two pivotal variables involved in determining how your planning process should be laid out, how your plan’s various components should be prioritized, and how changes to these components affect both the epistemological validity and the scientific value of your work. In other words – we now know how any given part of our plan affects another, as well as how important each part is to our downstream audience.

These are precisely the elements that enable the flexibility we sought at the beginning of this lesson. We now know which parts need to be planned first and most thoroughly. When pressed for time, we now know which parts can safely be cut or curtailed. Most importantly, if something changes – if a machine fails, a model underperforms, or a reagent is in short supply – our maneuvering of any downstream consequences is informed by a robust understanding of the epistemological connections between affected components.

The value of planning ahead can often appear abstract or nebulous, but in practice it is no less fundamental an exercise as the ideation of a research question or the perusal of relevant literature – in many ways, planning is just the act of putting connected ideas to paper to concretize, chart, and vet their epistemological relationships. It strengthens our understanding of our projects, structures our scientific investigations, and empowers us to act proactively if and when faced with the unexpected – which, in the sciences, is an inevitability