Planning & Conducting Experiments

Phase 1 - Define your research question and hypothesis

Foundation

• State your research question in one specific, answerable sentence
• Identify your independent variable (what you will manipulate) and dependent variable (what you will measure)
• Write a testable hypothesis using an if-then-because format or equivalent
• Confirm the question is not already answered in the literature — conduct a preliminary literature search

Phase 2 - Identify and operationalise your variables

Variable planning

• Define exactly how you will manipulate the independent variable (levels, doses, concentrations, conditions)
• Define exactly how you will measure the dependent variable — specify units, instruments, and precision
• List all potential confounding variables (e.g. temperature, pH, batch effects, time of day)
• Plan how each confounding variable will be controlled (held constant, randomised, or measured and adjusted for)

Phase 3 - Design your control conditions

Experimental controls

• Include a negative control (no treatment) to establish a baseline
• Include a positive control (known outcome) to verify your assay or method is working
• Ensure experimental and control conditions differ only in the independent variable
• Plan for blinding where possible — if you assess outcomes, can you be blinded to which condition each sample belongs to?

Phase 4 - Determine sample size and replication

Statistical power

• Decide on the number of independent biological or experimental replicates (not just technical replicates)
• Distinguish between technical replicates (same sample measured multiple times) and biological replicates (independent experimental units) — biological replicates are required for statistical inference
• Consider conducting a power calculation to determine the minimum sample size needed to detect a meaningful effect
• Plan for attrition — add 10-20% to your target sample size to account for failed experiments or excluded data points

Phase 5 - Write your experimental protocol

Reproducibility

• Write a step-by-step protocol with sufficient detail that another researcher could replicate your experiment exactly
• Specify materials, reagents, equipment, and their sources (manufacturer, catalogue number, lot number where relevant)
• Define your inclusion and exclusion criteria for experimental units
• Pre-register your hypothesis and analysis plan if possible — this strengthens the credibility of your findings

Phase 6 - Plan your data collection and analysis

Data integrity

• Create a data recording sheet before you begin — decide in advance where and how data will be recorded
• Decide which statistical test you will use to analyse results before collecting data (see the Statistical Analysis Catalogue)
• Define your criteria for outlier exclusion in advance — do not remove data points post-hoc without justification
• Plan how you will visualise your results (see Presenting Your Data)

Phase 7 - Conduct a pilot experiment

Before the full study

• Run a small-scale pilot before committing to the full experimental protocol
• Use the pilot to check that your assay or measurement method works and produces interpretable data
• Use the pilot to refine your protocol, troubleshoot equipment issues, and estimate variability
• Do not include pilot data in your final analysis unless pre-specified

Phase 1 - Define your research question, hypothesis, and design

Foundation

• State your research question in one specific, answerable sentence
• Identify your independent variable (what you will manipulate) and dependent variable (what you will measure)
• Write a testable hypothesis
• Decide on your experimental design: between-subjects (different participants in each condition) or within-subjects (same participants in all conditions)

Phase 2 - Obtain ethical approval

Required before any data collection

• Submit your protocol to your institution's IRB or Ethics Committee before recruiting or testing any participants
• Prepare an informed consent form explaining the study purpose, procedures, risks, benefits, and right to withdraw
• Identify any vulnerable populations in your sample (minors, people with cognitive impairments) and follow additional protections
• Plan your data anonymisation or pseudonymisation strategy to protect participant confidentiality
• Do not begin data collection until written ethical approval is received

Phase 3 - Define your participant population and sampling plan

Who will you study?

• Write clear inclusion and exclusion criteria for participants
• Determine your recruitment method (e.g. convenience sample, stratified sample, snowball sampling) and justify it
• Conduct a power calculation to determine the minimum sample size required to detect your expected effect
• Plan for dropout — recruit more participants than your minimum target
• Consider the representativeness of your sample and note any limitations on generalisability

Phase 4 - Identify and operationalise your variables

Variable planning

• Define exactly how the independent variable will be manipulated (what participants in each condition will experience)
• Define exactly how the dependent variable will be measured (scale, instrument, timing)
• List potential confounders (e.g. participant age, prior experience, time of day, order effects)
• Plan controls: randomise condition order for within-subjects designs; use random assignment for between-subjects designs

Phase 5 - Design your procedure and materials

Standardisation

• Write a fully standardised procedure — all participants should experience conditions in an identical way except for the independent variable
• Script or standardise any verbal instructions given to participants
• Pilot your materials and procedure with 2-3 participants before the full study
• Plan for blinding: if possible, keep outcome assessors unaware of which condition a participant was in
• Decide how you will counterbalance condition order in within-subjects designs to control for order effects

Phase 6 - Plan data collection, storage, and analysis

Data integrity

• Create your data recording forms or digital data entry system before beginning
• Store participant data securely, separately from identifying information, per your institution's data management policy
• Pre-specify your statistical analysis plan and primary outcome measure
• Define outlier exclusion criteria in advance
• Plan your visualisation approach (see Presenting Your Data)

Phase 1 - Define your research question and approach

Foundation

• State your research question — is this purely observational, or will you introduce an intervention in the field?
• Clarify your epistemological stance: are you measuring objective phenomena (positivist) or interpreting naturalistic behaviour (interpretivist)?
• Identify your unit of observation (individual, dyad, group, ecosystem, site)
• Confirm ethical requirements — field research involving people requires the same IRB approval as lab research

Phase 2 - Select your field site and access

Site planning

• Identify and justify your field site selection — how representative is it of the phenomenon you are studying?
• Obtain necessary permissions to access the site (institutional, governmental, community)
• Conduct a preliminary site visit to assess feasibility, safety, and logistical constraints
• Identify potential seasonal, temporal, or environmental variables that may affect your observations

Phase 3 - Define your variables and measurement approach

Operationalisation

• Define your independent variable (if experimental) or your primary observational variables (if purely observational)
• Define your dependent variable and how it will be measured in the field — standardise measurement tools
• Identify confounding environmental variables (weather, time of day, human disturbance) and plan how to document them
• Decide whether you will use structured observation (predefined coding scheme) or unstructured (open field notes)

Phase 4 - Design your sampling strategy

Who, what, and when to observe

• Choose a sampling method: random, systematic, stratified, or opportunistic — justify your choice
• Determine the sampling unit, observation interval, and total observation time per unit
• Plan for inter-rater reliability if multiple observers will be used — train observers and calculate agreement (e.g. Cohen's kappa)
• Conduct a power calculation or justify your sample size based on prior literature

Phase 5 - Prepare your data collection materials

Standardisation in the field

• Design your field data recording sheet or digital form before beginning observations
• Pilot your observation protocol at the field site with a small number of cases
• Establish protocols for unexpected events (equipment failure, safety incidents, participant withdrawal)
• Plan how field notes will be organised, stored, and backed up daily

Phase 6 - Plan your analysis

Before you begin

• Pre-specify your primary outcome variables and the statistical or qualitative methods you will use to analyse them
• For quantitative field data, consult the Statistical Analysis Catalogue to select an appropriate test
• For qualitative field notes, consider thematic analysis or content analysis (see Quantifying Qualitative Data)
• Document any deviations from your planned protocol in a methods log

Phase 1 - Define your research question and design

Foundation

• State your research question — what attitudes, behaviours, or characteristics do you want to measure?
• Decide on your survey design: cross-sectional (one time point) or longitudinal (multiple time points)
• Identify your target population and define inclusion/exclusion criteria
• Obtain ethical approval from your IRB or Ethics Committee before recruiting participants

Phase 2 - Select or develop your survey instrument

Measurement

• Search the literature for validated, previously published scales measuring your constructs of interest
• If adapting or creating new items, pilot them with a small sample (n=5-10) and revise based on feedback
• Ensure response scales are appropriate: Likert scales (e.g. 1-5 or 1-7) for attitudes; yes/no for binary items; open text for qualitative data
• Keep the survey as short as possible — each additional page increases dropout rates
• Include attention check items if using online platforms to identify inattentive responders

Phase 3 - Plan your sampling and recruitment

Who will respond?

• Choose a sampling method: random (probability) sampling is preferred but requires a sampling frame; convenience sampling must be acknowledged as a limitation
• Conduct a power calculation or use published benchmarks to determine the required sample size
• Plan your recruitment channels (email lists, social media, institutional networks, research participation pools)
• Set a target response rate and plan follow-up reminders — typical online survey response rates are 20-40%
• Consider incentives, but ensure they do not coerce participation

Phase 4 - Design your consent and data collection process

Ethics and standardisation

• Include a clear informed consent statement at the start of the survey
• Ensure participants can withdraw at any point without penalty
• Standardise administration conditions: all participants should receive the same instructions and question order (or pre-specified randomised order)
• Store survey data securely and anonymise responses at the point of collection where possible

Phase 5 - Plan your data analysis

Before you distribute

• Pre-specify your primary variables and hypotheses before distributing the survey
• Plan how missing data will be handled (listwise deletion, imputation) — document your approach
• For scale data, plan to assess internal reliability (e.g. Cronbach's alpha) before running primary analyses
• Select appropriate statistical tests (see the Statistical Analysis Catalogue) based on your variable types and research questions
• For any open-ended questions, plan your qualitative analysis approach (see Quantifying Qualitative Data)

Phase 1 - Define your research question and data requirements

Foundation

• State your research question precisely — what will you test, model, or analyse?
• Identify whether existing public datasets can answer your question, or whether you need to compile data from multiple sources
• Check whether ethical approval is required for your dataset (human data, even de-identified, may require IRB review)
• Review the terms of use and licensing for any public or commercial datasets you plan to use

Phase 2 - Identify and access your data source

Data provenance

• Identify the specific database(s) or dataset(s) you will use (e.g. NCBI GEO, UK Biobank, PubChem, Census data)
• Document the exact version, access date, and citation for each dataset
• Download and store a local copy of the data with checksums to verify data integrity
• Review the data documentation (data dictionary, codebook) to understand variable definitions and collection methods

Phase 3 - Pre-process and clean your data

Data quality

• Inspect the raw data for missing values, duplicates, inconsistencies, and implausible values before analysis
• Document all preprocessing and cleaning steps in a reproducible script (e.g. R or Python)
• Define and apply inclusion/exclusion criteria to your dataset and document how many records are excluded at each step (CONSORT-style flow diagram)
• Assess and report the completeness and representativeness of your final analytic dataset

Phase 4 - Plan your analysis pipeline

Reproducibility

• Pre-specify your primary analysis plan before exploring the data — post-hoc hypotheses must be clearly labelled as exploratory
• Write your analysis code in a scripted, version-controlled environment (e.g. R, Python with Git)
• Plan how you will validate your pipeline: test on a held-out subset or compare against published results where possible
• Consider making your code and processed data publicly available to support reproducibility (e.g. GitHub, OSF, Zenodo)

Phase 5 - Address confounding and bias in secondary data

Validity

• Identify potential sources of bias in your secondary dataset: selection bias, measurement error, missing data patterns
• Plan sensitivity analyses to test how robust your conclusions are to different analytical assumptions
• Acknowledge the limitations of your data source explicitly — findings from secondary data cannot demonstrate causation without strong design assumptions
• Select appropriate statistical methods for your data structure (see the Statistical Analysis Catalogue)