Experimental Design Revisited

 Teaching Tips

• Just do it!!! Students will not learn about the design of experiments by reading the text. They need to get dirty with the data: planning, collecting, organizing, summarizing, analyzing, communicating, and revising the design. Or, as Dan Teague once put it (in a slightly different context): "Students won't be good at it until they're bad at it first." So give them plenty of opportunities to make mistakes and improve.

• Early in the year, students can work on a first project using data from a published source or the data sets in the Fathom files. Working as a group will add to the students’ experience and lessen the time for grading class presentations. Assigning data projects early in the term helps us to know our students, their interests and strengths. Think of all those posters for Back-to-School Night! And besides, projects are fun for students.

• The Gettysburg address activity (see the Activity page)  introduces the concept of BIAS, so that students can distinguish BIAS and VARIATION in sampling. Students tend to overuse the word "bias" as a critique of experiments. They'll say a study is "biased" by some factor when, in fact, that factor merely introduces extra variability. However, "bias" occurs only when one treatment is favored over another by the experimental design.

• The bulls-eye diagram on BIAS and VARIATION in Yates Moore & Starnes is helpful and so is the histogram question in the Practice Problems.

• Some students will find it helpful to think of data as a combination of a "signal" with "noise." Any factor that adds to the "noise", thereby making it harder to detect the signal, is increasing the variability. Anything that distorts the signal -- makes it "tune" to the wrong station -- is adding bias.

• One of the most common misunderstandings students have is that they confuse treatment groups with blocks. Subjects are assigned to blocks BEFORE a treatment is performed, and are assigned to blocks based on their similarities. Subjects are then assigned to treatment groups WITHIN a block, so that all treatments are represented within each block.

• Students often confuse randomizing treatments in an experimental design with choosing a random sample for a survey. Selecting a simple random sample from a population is done so that we can make inferences to the larger population. However, often in experimental design we're more interested in establishing whether one factor CAUSES another. And the purpose of randomizing subjects to treatment groups is to help us establish a causal relationship EVEN IF IT IS JUST BETWEEN THESE SUBJECTS. In other words, often in an experiment it is not possible to make a statistical inference to a larger population, and sometimes this is not desirable.

• Students are easily confused by confounding variables. A confounding variable is not one that "stumps you". Confounding variables are variables that have a strong association with at least one of the treatment variables and the response variable. We might want to know, for example, if there is an association between race (treatment variable) and chances of admission to UCLA (response). But race is confounded with socio-economic status, which in turn influences which schools children attend which influences whether or not they apply to UCLA and also affects their chances for admission. Randomization across treatment groups minimizes the effects of confounding variables. (Of course, it is not possible to randomly assign someone a race.)

• Later in the course  a group project provides the opportunity for students to demonstrate what they have learned. It is best to establish a time-frame so that students do not wait until the last minute and establish clear guidelines for what is expected of them.
(See the Groth and Powell article in the NCTM Addendum Series 9-12 Data Analysis (1992) pp. 61-70 for suggestions and guidelines.)

• When doing projects, give students a chance to critique each other's designs. They can gain a lot from this activity.

• If you have AP Stat students in journalism, they can improve the quality of their newspaper surveys by having you and your students edit the questions, distribute the surveys (usually in clusters by randomly selected classrooms at a certain time) and then you can use these data for discussion, review, or exam questions.

•Students can use what they learn in AP Stats to improve their projects/reports in other courses, such as Biology, Physics, Econ, or Government, by adding a statistical component to their work. Better yet, AP Stat students can teach tools like Chi-squared to other biology students and regression to fellow Econ students.

• When assessing students' work, make sure they are not just mindlessly parroting the key terms (such as "control", "blocking", "bias"). Make sure the students understand why these concepts are important within the context of the design of the experiment.

• Historical Footnotes on randomization: for those of us who believe a little history makes things more interesting.

• The NCSSM Leadership Institute of 2001 had some useful notes on experimental design (pdf format).

• You might just find the entire NCSSM leadership institute website to be useful.