UNDERSTANDING SCIENTIFIC EVIDENCE
Module 1: The Scientific Method
Module Overview & Learning Objectives
What is the scientific method and why is it important? The scientific method is a series of defined steps that include observation and experimentation. It is a systematic way to test if a predicted outcome is supported by evidence (i.e. observations/data). While there is a great deal of variation in the specific techniques scientists use to test hypotheses, the following steps characterize most scientific investigations:
Step 1: Make observations, research, and ask questions about that observation.
Step 2: Propose a hypothesis (i.e. a prediction) statement to explain observations.
Step 3: Test the hypothesis with experimentation and further observation.
Step 4: Collect data, evaluate observations for trends, and state results.
Step 5: State conclusions about the hypothesis based on evidence provided from data.
In the following sections, you will learn about each of these steps in greater detail, after which you will be tasked to develop your own scientific experiment via an online discussion.
After completing this lesson, you should be able to:
-
describe the steps that generally characterize the scientific method;
-
explain the importance and utility of the scientific method;
-
list the qualities that characterize a valid scientific hypothesis;
-
formulate a testable hypothesis to explain a given set of observations;
-
apply the scientific method to a hypothetical situation and formulate your own conclusion;
-
critically evaluate the research design of scientific experiments.
Introduction Video
Video length (~4 minutes). Reference: "Scientific Method Box" by Sciencefix is used under the terms of the CC BY license. This video is to be updated with a new production using similar concepts.
Steps of the Scientific Method
1. Background Information: Describe Observations, Current Research, and Question of Interest
The scientific process typically starts with observations. Whether a problem to be solved and or an explanation for a phenomenon, observations set the stage for further inquiry and experimentation. Often, simple observations will trigger a question prompting further research. This research is often initially in the form of looking up information in the literature, but it can also lead to unanswered questions that require experimentation to answer. Let's think about a simple problem that starts with an observation and apply the scientific method.
Observation: A researcher sees that monarch caterpillars feed on a particular species of milkweed plants, but rarely sees them feeding on other types of plants in his garden.
Question: Do monarch caterpillars prefer milkweed over other food choices?
"Monarch Caterpillar on Swamp Milkweed" by Friends of Seney National Wildlife Refuge is licensed under CC BY-SA 2.0.
2. Hypothesis
The researcher develops a hypothesis (singular) or hypotheses (plural) to explain these observations. A hypothesis is a tentative explanation of a phenomenon or observation(s) that can be supported or falsified by further observations or experimentation. Here is an example of a hypothesis that the researcher could test:
Hypothesis: Monarch caterpillars prefer to feed on milkweed compared to other common plants.
Notice that the hypothesis is a statement, and not a question as seen in step 1.
Qualities of a Good Hypothesis
-
A hypothesis must be testable or provide predictions that are testable. It can potentially be shown to be false by further observations or experimentation.
-
A hypothesis should be specific. If it is too general it cannot be tested, or tests will have so many variables that the results will be complicated and difficult to interpret. A well-written hypothesis is so specific it actually determines how the experiment should be set up.
-
A hypothesis should not include any untested assumptions if they can be avoided. The hypothesis itself may be an assumption that is being tested, but it should be phrased in a way that does not include assumptions that are not tested in the experiment.
-
It is okay (and sometimes a good idea) to develop more than one hypothesis to explain a set of observations. Competing hypotheses can often be tested side-by-side in the same experiment.
3. Test the Hypothesis (Experimental Design)
Once a hypothesis is made, an experiment can be designed to test the hypothesis. An experiment is a controlled situation created by a researcher to test the validity of a hypothesis (i.e. whether the hypothesis is supported or not by data). While specific procedures can vary considerably between experiments depending on the hypothesis being tested, most experiments have the following elements in common.
-
The group of subjects being experimented on should ideally be alike in all ways except from the factor or factors that are being tested. From this, all experimental conclusions should be confined to the criteria outlined for the group. Using the example above, the only caterpillars being tested are Monarch caterpillars. Thus, any conclusions drawn from the experiment could not be applied to other species of caterpillars, as only Monarch caterpillars were specified and tested.
-
All variables outside of those being compared in the experiment should be identical. Variables are the factors that can vary within an experiment, which can include the independent variable (what is intentionally being altered), the dependent variable (what is being measured) and the control variables (the factors that can vary but are being controlled throughout the experiment to avoid external influences on results), all of which are discussed in greater detail in the next section.
-
There should be sufficient replication within the experiment to verify that the results are accurate and representative of what is being tested. Additional details about the justification for and importance of replication are discussed in the next section below.
Variables and group, replication within an experiment, and the write-up of your methods are described in better detail below.
Variables and Groups
The independent variable is the factor that you are interested in testing the effect of (i.e. the factor being manipulated or changed in the experiment). Another way to look at it is that the independent variable is the one factor that differs between the experimental and control groups. The experimental group is the treatment from the independent variable that you are changing and comparing to your control group to see if there is an effect. The control group is the treatment of the independent variable that you keep constant to be able to compare to your experimental group.
“Groups and variables” image by kautzcarmen can be reused under the CC-BY 4.0 license.
The dependent variable is what is being measured in response to a change in the independent variable. This is the data being collected in the experiment.
Notably, both the independent variable and dependent variable should be included in the hypothesis, which is often written in the if/then format. For example, "If given the choice between milkweed or tomato plants (independent variable), then Monarch caterpillars will more often choose milkweed plants to feed on (dependent variable)." Alternatively, the hypothesis can be phrased without the if/then statement, provided it includes what is being compared (the independent variable) and how it is being measured (the dependent variable). So for example, the above hypothesis could be phrased "When given the choice, Monarch caterpillars will more frequently eat (dependent variable) milkweed plants than tomato or plants (independent variable)."
Control variables (sometimes called controlled variables or constant variables) are all of the other external variables that remain constant in the experiment. Control variables are the opposite of the independent variable, in that control variables do not change so that the independent variable is the only factor being altered (i.e. tested) in the experiment. The control variables are all the factors that remain the same between the control and the experimental groups.
Replication within the experiment
Replication is an essential component to any experiment because of sampling error and measurement error.
(This section will be updated with either imagery or a short video and text will be significantly reduced)
Sampling Error:
There is a great deal of variation in nature. In a group of experimental subjects, much of this variation may have little to do with the variables being studied but could still affect the outcome of the experiment in unpredicted ways. For example, assuming we know nothing of Monarch caterpillar food preferences, it would be unreasonable to assume that all Monarch caterpillars have the same extent of food preferences, as it is possible that some may have stronger food preferences to certain plants than others. Additionally, being a toxic plant, organisms that feed on milkweed must produce specific enzymes to be able to metabolize the toxins. There may be significant variation in enzyme abilities between individual caterpillars. Some of this variation might be due to differences in genetic make-up, to varying levels of previous milkweed plant exposure, or any number of factors unknown to the researcher. Thus, to ensure that the experiment is able to be representative for Monarch caterpillars as a whole species, many subjects must be used in order to determine what an average food preference is. Sampling error is “the use of a sample (or subset) of a population, an event, or some other aspect of nature for an experimental group that is not large enough to be representative of the whole" (Starr, Cecie, Biology: Concepts and Applications, 4th ed. [Pacific Cove: Brooks/Cole, 2000], glossary). If too few samples or subjects are used in an experiment, the researcher may draw incorrect conclusions about the population those samples or subjects represent.
Because the researcher wants to discover the food preferences for the majority of Monarch caterpillars, but cannot test all Monarch caterpillars in the world, the experiment must be run on a number of different subjects. Suppose the experiment was performed on only two individuals, but one was sick. Do you think the average response calculated would be the same as the average response of all Monarch caterpillars? What if 100 individuals were tested, or 1,000? Do you think the average would be the same in each case? Chances are it would not be. So which average would you predict would be most representative of all Monarch caterpillars?
A basic rule of statistics is, the more observations you make, the closer the average of those observations will be to the average for the whole population you are interested in. This is because factors that vary among a population tend to occur most commonly in the middle range, and least commonly at the two extremes. Thus, one reason why repetition is so important in experiments is that it helps to assure that the conclusions made will be valid not only for the individuals tested, but also for the greater population those individuals represent.
Measurement Error:
The second reason why replication is necessary in research studies has to do with measurement error. Measurement error may be the fault of the researcher, a slight difference in measuring techniques among one or more technicians, or the result of limitations or glitches in measuring equipment. Even the most careful researcher or the best state-of-the-art equipment will make some mistakes in measuring or recording data. Another way of looking at this is to say that, in any study, some measurements will be more accurate than others will. If the researcher is conscientious and the equipment is good, the majority of measurements will be highly accurate, some will be somewhat inaccurate, and a few may be considerably inaccurate. In this case, the same reasoning used above also applies here: the more measurements taken, the less effect a few inaccurate measurements will have on the overall average.”
Writing up the methods used
The specific description of the methods used to conduct an experiment is an essential component to the experimental design. Having a well-described methods description allows others to be able to systematically review your design for potential biases and to be able to replicate your experiment again to verify results and conclusions. Overall, the methods section could include specific details about how and where you set up the experiment, how long the experiment lasted, what are the qualifying details of the test subjects, and if there were any extra steps taken to ensure external variables did not influence the outcome of the experiment. It should also include what the variables and groups of an experiment are, and how replication was introduced within the experiment. Let's look at an example methods write-up for our experiment.
Methods: To establish the plants for the experiment, twenty trays of soil were planted with one milkweed, one tomato, and one snap pea seed. Each tray had the same three species of plants grown. The plants were grown in these trays for the duration of two months prior to experimentation to establish growth, where they obtained natural sunlight in a greenhouse setting (temperatures ranging from 70°F - 80°F daily), were hand watered daily with 500mL water per tray, and were maintained in the same type of soil between trays. After 45 days of growth, all eighteen trays that successfully grew all three plants were used in the experiment. To test caterpillar food preferences, caterpillars were placed in the center between the three plants, alternating which plant they faced between trays. Once placed in the trays, the frequency of plant food preference was recorded for each tray by tallying which plant the caterpillar was eating every ten minutes for the span of one hour.
The groups and variables for the experiment are as follows:
-
Independent variable: the food source preferred by Monarch caterpillars.
-
Experimental group: the caterpillar accessibility to snap pea and tomato plants.
-
Control group: the caterpillar accessibility to milkweed.
-
-
Dependent variable: the frequency (i.e. the number of times within a given time period) of eating one type of plant versus another.
-
Control variables: sunlight, temperature, amount of watering, soil used to grow plants, the age of the plants, and the direction the caterpillars faced when being presented the plants.
4. Results
The researcher summarizes the information, or data, generated from the experiment. Ideally, the results section will include a table or table of the data collected and a brief write-up summarizing the results. The results may also include a graphs or figures, when appropriate, to illustrate the results in a more easy to read format. Using our example experiment above, we could have the following results:
Results: When eating, milkweed was chosen by Monarch caterpillars 100% of the time, relative to tomato and snap pea plants.
5. Conclusions
The researcher interprets the results of experiments or observations and forms conclusions about the meaning of these results. These conclusions are generally expressed as probability statements about their hypothesis. Furthermore, it should be stated whether the hypothesis is supported by the data. Using the Monarch example, we could describe the following conclusions
Conclusions: When given a choice, 100% percent of monarch caterpillars prefer to feed on milkweed over other common plants. Given this, the data supports the hypothesis.
Often, the results and conclusions of one scientific study will raise questions that may be addressed in subsequent research. For example, the above study might lead the researcher to wonder why monarchs seem to prefer to feed on milkweed, and they may plan additional experiments to explore this question. For example, perhaps the milkweed has higher nutritional value than other available plants.
Additional Considerations
In the Monarch caterpillar example presented above, preliminary research was not conducted to investigate what the current scientific literature had already discovered about Monarch caterpillar food preferences. If the researcher had done some initial background investigation, they would have found that Monarch caterpillars exclusively eat milkweed plants. However, the replication of this study further supports those findings, and the background literature further supports the findings from this study. So, this research builds on previous findings, further supporting the knowledge base on this topic.
Summary: Qualities of a Good Experiment
The following are qualities of a good experiment:
-
The hypothesis is clearly stated, describing both the independent and dependent variables.
-
The subjects being tested are narrowly defined (i.e. alike in all ways outside of the factor being tested)
-
Biasing variables are controlled throughout the experiment (i.e. control variables).
-
A control group is used to compare changes in the experimental group.
-
The procedures used to conduct the experiment are well-described so that others could easily replicate the experiment.
-
The measurements related to the factors being studied are carefully recorded.
-
There are enough samples or subjects are used so that conclusions are valid for the population of interest.
-
The conclusions made relate back to the hypothesis, are limited to the population being studied, and are stated in terms of probabilities (e.g. the data supports the hypothesis).
Now that you've learned about the steps of the scientific method, let's look at an example from the scientific literature related to the current topic. In a recent article, Monarch Butterflies Show Differential Utilization of Nine Midwestern Milkweed Species, researchers investigated if Monarch butterflies had preferences on specific milkweed species to deposit their eggs. Now, let's go through each step of the scientific method, summarizing the research conducted in this paper.
Background: Monarch butterfly numbers have declined over the past few decades, which is believed to be caused by loss of habitat. While butterfly sanctuary restoration habitats are being established, scientists asked the question if Monarch butterflies prefer to lay their eggs on certain species of Milkweed over others.
Hypothesis: If Monarch butterflies prefer a specific species of milkweed (A. incarnata species), than they will lay more eggs on that species over other milkweed species during their mating period.
Methods: Nine species of milkweed plants (A. exaltata, A. hirtella, A. incarnata, A. speciosa, A. sullivantii, A. syriaca, A. tuberosa, A. verticillata, and C. laeve) were equally planted on several plots across ten farms in the state of Iowa. Over the three month mating period (June - August), the number of eggs laid on each type of plant was counted for three years (in 2015, 2016, and 2017). Additional factors, such as plant height, number of blooms, and presence of seed pods was also recorded for further analysis later. At the end of the two month period, the total number of eggs was summed and compared.
Independent Variable: type of milkweed species
Control Group: A. incarnata species (one random species was chosen here to keep this simple)
Experimental Group: the other 8 species of milkweed plants (A. exaltata, A. hirtella, A. speciosa, A. sullivantii, A. syriaca, A. tuberosa, A. verticillata, and C. laeve).
Dependent Variable: total number of eggs laid by Monarch butterflies over the two month period each year
Control Variables: number of each species of plant present, amount of rainfall as dictated by weather, the amount of given to lay eggs (2 months)
*Replication - This experiment had multiple layers of replication introduced, where the number of plants for each species was replicated, the number of plots across ten farms was replicated across the state of Iowa, and (importantly) the number of years that egg quantities were measured on each plant.
Results: Following the summation of total number of eggs laid on each plant for each year, the total egg counts were averaged and compared using statistical tests. From this data, it was found that Monarch butterflies laid the most eggs on A. incarnata species over the course of three years.
Conclusions: The hypothesis is supported by the data. Monarch butterflies have a preferred milkweed species (A. incarnata) to lay eggs on. However, other species (i.e., A. syriaca, A. sullivantii, and A. speciosa species) could be suitable alternatives in planning for habitat restoration projects.
References:
Pocius, V. M., Pleasants, J. M., Debinski, D. M., Bidne, K. G., Hellmich, R. L., Bradbury, S. P., & Blodgett, S. L. (2018). Monarch butterflies show differential utilization of nine Midwestern milkweed species. Frontiers in Ecology and Evolution, 6, 169.
Module 1 Assignment:
In the discussion forum, describe an example of how you use the scientific method in your daily life. Include your observations, state a detailed and good hypothesis, describe the methods of how you are testing the hypothesis, include (hypothetical) results, and a conclusion. What are the control and experimental treatments or groups in your example? What are the independent, dependent, and controlled variables in your experiment?