Design of Experiments

Peter Humburg
22/07/2019

Outline

  • Why Design Experiments?
  • From Research Question to Results
  • General Principles
  • Case Studies

Why Design Experiments?

Asking the Right Questions

R. A. Fisher

Nature will best respond to a logical and carefully thoughtout questionaire; indeed, if we ask her a single question, she will often refuse to answer until some other topic has been discussed.

Control Variation and avoid Confounding

  • Real world processes are complex.
  • Outcome will depend on more than one factor.
  • Ignoring important factors will increase variance of observed outcomes.
  • Factors ignored by the experimental design may confound results.

Gru's plan

  • The right design ensures research question can be answered.

From Research Questions to Results

The life cycle of an experiment

Experiment life cycle

Experiment life cycle

  • Formulate research question.
  • Determine relevant factors.

What are we testing?

Experiment life cycle

Experiment life cycle

  • Design experiment to measure response while
    • varying variables of interest
    • controlling for possible confounders
    • avoiding unintended side effects

Experiment life cycle

Experiment life cycle

  • Determine required sample size.
  • Carry out experiment.
  • Analyse data and interpret results.

General Principles

Definitions

Experiment: An investigation that establishes a set of particular circumstances under a specific protocol to observe and evaluate implications of resulting observations.

Treatments / Conditions: The set of circumstances created for an experiment.

Experimental unit: Subject exposed to the treatment independently of other units.

Definitions

Experimental error: Variation among identically and independently treated experimental units. May be due to

  • natural variation among experimental units
  • variability in measurement of the response
  • variability in treatment conditions
  • interaction between treatments and experimental units
  • external factors not controlled by the experiment

Errors

Observational Studies

Experiment

  • Good control over treatment conditions.
  • Can control or eliminate many extraneous variables.
  • Can be replicated.

  • Artificial environment.
  • May not generalize to real life situations.
  • Not always practical.

Observational study

  • Study change of target variable in real life.
  • Feasible when experiments are not possible.

  • Poor control of treatment conditions.
  • May not be able to control extraneous variables.
  • Can't replicate.

General Considerations

Minimize Error Variance

  • Account for major sources of variation.
  • Limit heterogeneity within groups.
  • Limit variability between subjects.

Avoid Confounding

  • Don't change several variables in parallel.
  • Limit heterogeneity between subgroups.

Avoid unintended side effects

  • Consider carry-over effects.
  • Consider treatment order.

Planning like a boss

Creating homogeneous groups

Blocking

  • Create groups based on the value of a factor.
  • Increases homogeneity within groups.
  • Allows testing of differences between factor levels.
  • Not always practical.

Avoiding Confounding

Randomization

  • Assign experimental units randomly to groups.
  • Helps to avoid bias.
  • Helps to avoid confounding.
  • Can reduce extraneous differences between groups.
  • Beware of unsuitable randomization patterns in small samples.

Creating homogeneous groups

Exclusion

  • Eliminate nuisance variable by holding it constant.
  • Choose one factor level, exclude all others from the study.
  • Increases homogeneity of sample.
  • May substantially reduce number of available participants.

Avoiding Confounding

Trait Matching

  • Choose composition of one group to match composition of another.
  • Popular in Case/Control studies.
  • Patients often have to be recruited opportunistically.
  • Balance resulting heterogeneity in patient group with matching pattern in control group.

Avoiding side effects

  • May want to collect data from same individual for several conditions.
  • Earlier conditions may affect the outcome of later ones.
  • The order of conditions may be a confounding factor.
  • Assign participants to groups with different treatment orders.

Case studies

Identifying words in a second language

Research Question

Is it more difficult to identify words in a second language, compared to the native language?

  • How much more difficult?

Things to consider

  • Which languages are we studying?
  • Is each participant tested for both languages?
    • In what order?
  • How to choose words for testing?
    • Familiarity?
    • Complexity?
  • What about language proficiency?
  • Other factors that affect reaction time?

Ask the right questions

Identifying words in a second language

Research Question

Is it more difficult to identify words in a second language, compared to the native language?

  • How much more difficult?
  • Does word complexity affect difficulty?

Things to consider

  • How do we measure complexity?
  • Need to choose set of simple and complex words.
  • In what order should words be presented?
  • What about switching between languages?

Ask the right questions

Your Turn!

Wrap-up

  • Remember the basic principles.
  • Consider additional factors that might affect the outcome.
  • Incorporate relevant factors into design.
  • But only vary one at a time.
  • Find the slides here: