Jane Allyn Piliavin- Sociology at UW Madison, bascom graphic
Sociology 357 Piliavin

FIELD EXPERIMENT EXERCISE

DUE DATES: See summary page.

For this exercise you design and carry out a small field experiment using a randomized post-test-only between-subjects (control group) design. You devise a manipulable independent variable and a measurable dependent variable. You may use any hypothesis and methodology that you are comfortable with.

About Teams.

You are encouraged but not required to do this exercise with another class member. Teams have two options:

  1. write a joint report, or
  2. each person write the whole report individually.

Option 1 is appropriate when team members are truly working and learning together. It is unfair and unethical for one student to do most of the studying and writing while another "free rides" under the guise of option 1. Those doing the work have the right to refuse to "give" partners papers they did not help write. If you choose option 2, you work together until the data are collected and, if you wish, put into a statistical table, but you must not collaborate in writing your separate reports. If you find yourself in an ambiguous position about these options because of unforeseen problems, speak to me and I will help you to determine the fairest thing to do.

Planning Your Experiment

Experiments are planned in detail in advance. You should not do your "official" experiment until your planning is complete. In planning your experiment, you need to figure out each of the following:

  1. A "setup," the basic thing that will be happening in your experiment (e.g. a survey, a petition, a request for help).
  2. A manipulable independent variable, with a careful operationalization of the differences among the categories or treatments. There should be only one variable with two or three categories.
  3. An operationalized dependent variable which can be objectively and consistently measured for all subjects, regardless of the treatment they receive.
  4. A plan for randomly assigning subjects to treatments that is properly random and that is practical in the situation you have devised.
  5. A plan for holding constant all aspects of the situation and your behavior except the independent variable or things which can be randomized.
  6. A sampling plan. (You do not need a random sample, although you may use one if it is possible with your setup.)

It is often very useful to run small scale pretests of parts of your idea, to see if it will work. You might try out your independent variable or your dependent variable before wasting too much time on something that might not work.

Your experiment should have only one independent variable and one dependent variable. You may do additional independent or dependent variables only if you are sure you know what you are doing. People who are not experienced in doing experiments often throw in extra variables "just for the fun of it" or by mistake, so I must have the rule that extra variables must be handled correctly to get credit. If you think you know how to do this in your experiment and want to do so, please talk with me first and I will explain factorial designs.

Doing your Experiment

Follow your plan and obtain a minimum of 20 cases (i.e. units of analysis: subjects or trials or groups) for each category of the independent variable. If the independent variable has 2 categories, this will be 40 cases. More cases per category of the independent variable is better, especially with a categorical dependent variable. This is a very easy goal to obtain in the vast majority of field experiments. It is common for people to have so much fun they do more. Do not go below 20 cases per category unless you have written permission from me. Usually when it seems unreasonable to do so many cases it is because your idea is impractical for a class exercise and you need advice.

Record the data according to your plan, and make a note of any unexpected things that happen. Make sure you turn in your data with your report. You may analyze your data in SPSS again, or you can do it by hand. I have provided you with the formulas for t and chi-square so you can do the appropriate test.

WRITTEN REPORT


PLEASE FOLLOW THIS FORMAT EXACTLY.

About Truthfulness. Science depends on researchers telling the truth about what really happened in their research, not what they wish had happened. At the same time, students worry that they will be graded down if they tell the truth. So, for each question, I insist that you tell the truth about what really happened in the research, but then follow it with an opportunity to explain what you now think you should have done. If there was a mistake and your self-criticism gives a correct statement about what you should have done, you will receive full credit as if you had done things right in the first place. The only exception is if you have failed to manipulate your independent variable; then you will have to do it over.

OUTLINE

  1. Title page. Title of report, author(s), date. Put partner's name in parentheses at the bottom of the page if you worked with someone but wrote reports separately.
  2. Abstract. Write one paragraph which summarizes your hypothesis, experimental research methods, and findings. You may include this on the title page if you wish.
  3. Body of paper.
    1. Introduction. Write a paragraph stating your bivariate hypothesis and why it is worth researching. Sometimes teams agree on the variables but disagree on what they predict will happen. This is OK for the purposes of this assignment; just say who makes what prediction. (Note: Citations to readings are not needed, but go here if something you read went into your thinking on this project.)
    2. Methods of research. (Note: To aid grading, number each section of this discussion as it is numbered here.)
      1. Set-up.
        1. The "set up" of an experiment is the context within which the independent and dependent variables operate. Describe the "setup" of your experiment in enough detail so that a reader can picture the situation. DO NOT PUT OTHER DETAILS HERE.
        2. Why you chose it.
        3. Evaluation: why you think these procedures were good, or what you now believe should have been done differently. How it worked out in practice.
      2. Independent variable. (Make sure this is manipulable)
        1. What was the conceptual variable you were trying to capture by your experimental manipulation? Define it.
        2. Give complete details on your operational definition of that conceptual variable, as you planned it, and why you chose this particular operationalization.
        3. How the operationalization actually worked out. Evaluate the operationalization: why do you think these procedures were good, or what do you now believe should have been done differently.
      3. Dependent variable.
        1. What was the conceptual variable you were trying to measure? Define it.
        2. Give complete details on your operational definition of that conceptual variable, as you planned it, and why you chose this operationalization.
        3. Why subjectivity (the views of the observer affecting the measurement) is not a problem with this way of measuring the concept, or what you did to reduce the possibility of bias due to subjectivity.
        4. Evaluate this operationalization: why do you think these procedures were good, or what do you now believe should have been done differently. How the operationalization actually worked out.
      4. Experimental Control.
        1. Details on the things about your procedures or behavior you consciously brought under experimental control through holding constant.
        2. How this actually worked out.
        3. Evaluation: why you think these procedures were good, or what you now believe should have been done differently.
      5. Sampling.
        1. Specify your unit of analysis (individual, group, time period, trial). Ask me, if there is any doubt.
        2. Describe your sampling procedures, including location and time, any restrictions placed on eligible subjects, or other procedures for deciding whom to study.
        3. Evaluation: why you think these procedures were good, or what you now believe should have been done differently.
      6. Randomization. In real professional experiments, randomization is usually taken for granted and not discussed. Because this is a course assignment, I want you to spell these procedures out.
        1. Describe how you randomly assigned units of analysis to categories of the independent variable. Say what your random device was (slips of paper, coin flips, shuffling, etc.) but, more importantly, explain how you worked the randomization into the flow of the whole experiment. [If you did a within-subjects experiment, explain how you randomized order of presentation.]
        2. Describe any additional steps you took to prevent selection biases from entering after the randomization (e.g., subjects being able to selectively avoid one or the other condition), or explain why they were not necessary.
    3. Results. (Attach the original messy data collection sheet to the back of your paper as an appendix.)
      1. Present a bivariate statistical table to show the relationship between your independent variable and dependent variable. Do either a contingency table or a difference of means table.
        1. If your dependent variable is categorical, present a contingency table. (Treat a dependent variable with only 2 or 3 ordinal (ranked) attributes as categorical.) This should include the percentage for each category of the dependent variable separately for each category of the independent variable. For the statistical test of the hypothesis, do a chi-square.
        2. If your dependent variable is continuous, compute the mean for the dependent variable separately for the cases in each category of the independent variable. (We will define a variable as continuous if it has at least 4 attributes with ordinal, interval, or ratio properties.) Do a t-test or a means test to determine the statistical significance of results.

          3. (If it turns out that you have done a within-subjects experiment, in which each subject has served in both conditions of your independent variable -- which you weren't supposed to do but could be the best way to tackle your problem -- you will need to see me for a different way of analyzing your results.)

      2. Write a paragraph discussing your statistical results, saying what they show and whether your hypothesis is confirmed or disconfirmed. Use p<.05 as your significance level.
      3. Discuss anything else worth mentioning that you learned in your research, including unexpected events or surprising findings.
    4. Conclusions and interpretations. This is where you talk about the larger issues your research raises. In this context, please discuss the ethical issues that may have been involved in "manipulating" people in your setting. This is also the place to talk informally about what you liked or didn't like about the assignment or the way you did your research.
  4. Questions testing your understanding of the logic of experiments. This would not normally be part of a research report, but is used because this is a class. Everything in this section refers to extraneous variables that might threaten the internal validity of your results, that is, to things that might lead your results to be spurious. This is a different question from what is asked under "setup", which dealt with actions you took to eliminate these threats. Here, in contrast, I want to know what extraneous factors were controlled by doing it.
    1. What extraneous variables were eliminated as alternative explanations through randomization? (Discuss the several different kinds of extraneous variables that were controlled, using a variety of examples of things that might actually have been relevant to your experiment.) If you did a within-Ss experiment, discuss (1) what is controlled by use of that design and (2) what is controlled by the way you handled order of presentation of conditions. Label these discussions as (1) and (2).
    2. Under "Experimental Control" (III,B,4) you discussed factors you held constant. What alternative explanations were eliminated through holding those things constant? (Discuss the several different kinds of extraneous variables that were controlled, using examples of things that might actually have been relevant to your experiment.)
    3. What threats to internal validity remain uncontrolled in your experiment? (NOTE: I do not mean "random error," the possibility that randomization failed to equate the groups. I mean factors that were not randomized and not held constant, that might have been correlated with the independent and dependent variables and could therefore have led to the results observed.)
  5. Group process report. Pick the category that applies to you and answer the relevant questions.
    1. No partner.
      1. How did you feel about working alone? Would you do it again, or would you prefer a group?
      2. How much effort did you have to put into this project?
      3. How well prepared did you feel in terms of course materials and understanding what to do.
      4. Tell me if there is anything I should know about you or your life that you want me to know, especially if it might affect your grade or my ability to be fair in grading your work.
    2. Had partner, wrote separate papers.
      1. Compare you and your partner in the effort you put into the project and in the extent to which you studied course materials and knew what to do for the assignment.
      2. Compare you and your partner in the extent to which you studied course materials and knew what to do for the assignment.
      3. Who did the design, data collection, statistical analysis?
      4. Did you start trying to work together before deciding to write separate papers? How far did you get?
      5. Were there some things you found necessary to discuss in preparation for writing your papers? What?
      6. How did the group process work out? Was it a positive or negative experience? Would you do things differently in the future?
      7. Tell me anything else I should know that might affect your grade or your partner's, or that I should know to be fair in grading your work, or that you would like me to know even if it is not relevant to your grade.
    3. Wrote joint paper.
      1. Do you stand by the paper as written, or is there something you feel should have been said differently? Any corrections you offer at this point will be factored into your grade. This answer should be as long as you feel is appropriate.
      2. Compare you and your partner in the effort you put into the project.
      3. Compare you and your partner in the extent to which you studied course materials and knew what to do for the assignment.
      4. Who did the design, data collection, statistical analysis?
      5. How did you go about getting the writing done?
      6. How did the group process work out? Was it a positive or negative experience? Would you do things differently in the future?
      7. Tell me anything else I should know that might affect your grade or your partner's, or that I should know to be fair in grading your work, or that you would like me to know even if it is not relevant to your grade.

 

 

Top

Questions? Comments? Please contact jpiliavi@ssc.wisc.edu

Social World textbook cover

Home

Vita

Sociology 236

Sociology 357

Sociology 647

Sociology 965

Sociology Homepage