What constitutes high-quality education research?
High-quality research that yields strong evidence about the efficacy of specific practices is rare in education. For example, although there is general agreement that randomized trials represent the "gold standard" for assessing the effects of an educational intervention, curriculum, or program, few randomized trials have been conducted in any area of education. The position of the U.S. Department of Education's Institute of Education Sciences is clear in its description of early childhood education research priorities:
The evidence that would allow informed choices of classroom curricula for early childhood programs is weak. Rigorous preschool program evaluations that exist are for programs designed and delivered decades ago. The results from historical evaluations of preschool curricula and current research on the learning and development of young children provide some insights into general features of successful preschool programs. However, they give little guidance for selecting from among the ever-expanding list of available preschool curricula. (U.S. Department of Education, 2002a)
The current emphasis on evidence-based education has its foundation in the No Child Left Behind Act of 2001, which describes the basis for evidence-based education as empirical research (that is, research relying on or derived from observation or experiment) that involves "the application of rigorous, systematic, and objective procedures" to research questions (U.S. Department of Education, n.d.).
Dr. Grover (Russ) Whitehurst, an experienced researcher in early childhood education and now head of the Institute of Education Sciences (IES) within the U.S. Department of Education, has defined evidence-based education as "the integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction" (Whitehurst, n.d.).
The No Child Left Behind Act specifies that the most respected research:
- uses experimental or quasi-experimental research methods;
- randomly assigns subjects to a control group and one or more experimental groups, "conditions," or "treatments"; and
- employs objective measures or observational methods that provide reliable and valid data across evaluators and observers, across multiple measurements and observations, and across studies by the same (or different) investigators.
In addition to emphasizing the importance of empirical evidence, Whitehurst and others have identified "levels of evidence" for evaluating research evidence upon which to base practice. These levels, from most rigorous to least rigorous, are as follows:
- Randomized
trials, often referred to as the "gold standard"
of research, randomly assign children to two or more conditions
in a study, with before and after (pre- and post-) testing. There
is general agreement that randomized trials are the best research
method for assessing the effects of an educational intervention
(also called a "treatment" or "condition")
on outcomes for children. In randomized trials, children are randomly
assigned to two or more conditions that differ in levels of exposure
to the educational intervention. Children are then tested for
differences in outcomes. Randomization is used to assure that
children in studies have the same characteristics across conditions
and to assure that differences between conditions do not result
from preexisting differences in the children.
Example: A randomized trial of the "Tools of the Mind" curriculum, being conducted by the National Institute for Early Education Research (2003), will study the effects on development and learning of the curriculum, described as "an innovative developmentally appropriate approach with an emphasis on improving early literacy 'instruction.'" (For a description of the "Tools of the Mind" curriculum, see http://www.mcrel.org/programs/literacy/ela/ToolsMind.asp.) [Editor's note (07-23-07): this url is no longer active.] - Quasi-experimental studies, including before and after testing, use nonrandom assignment of children to various conditions for ethical or practical reasons and include pre- and post-testing of all children in the study. Although quasi-experimental designs are used commonly, they are subject to numerous interpretation problems. For more information on this kind of design, see Gribbons and Herman (1997).
- Correlational studies with statistical controls examine the degree of association between two or more variables using one randomly selected sample group [I don't think most correlational studies use randomly selected samples]; although some correlational studies are useful in prediction (i.e., the higher the correlation between two variables, the more accurate the prediction), they cannot be used to infer or determine causal relationships. Statistical techniques are used to measure the association. Correlational studies without statistical controls examine the degree of association between two variables but do not make use of statistical techniques to measure the association between variables. Correlational studies look for relationships between variables. For example, do children from families in poverty families perform worse academically than the general student population? In each case, we are asking whether a relationship exists between variable X and variable Y. Correlational studies reveal a relationship between the two variables, but they do not tell us which variable "caused" the other.
- Case studies obtain detailed information about an individual to contribute to an understanding of general principles of behavior, but the findings cannot be generalized to whole populations from single case studies. Although most researchers are interested in general trends in behavior, those engaging in case studies approach individual subjects looking for depth and context. A large amount of information is generally gathered in this kind of study, and conclusions are based on a large set of information about the subjects—often more than in other types of research. Despite their strengths, however, case studies have significant drawbacks. First, like all non-experimental approaches, they describe what is occurring but not why it is occurring. Second, there is considerable room in case studies for researcher bias to creep in. No approach—not even the experimental approach—is immune from researcher bias when used by a novice or poorly trained researcher, but case studies are at great risk for bias even when conducted by well-qualified people.
In order to help educators make sense of complicated research reports, the U.S. Department of Education recently issued a new publication, titled Identifying and Implementing Educational Practices Supported By Rigorous Evidence: A User Friendly Guide, produced by the Washington, DC-based Coalition for Evidence-Based Policy (2003). This guide is intended to assist practitioners in distinguishing between practices supported by rigorous evidence from those practices that are not well supported.
The American Educational Research Association (AERA) Council unanimously passed a resolution on January 26, 2003, promoting the essential elements of sound scientific research. It should be noted that although the Council affirmed the importance of randomized trials as a valuable research method, the AERA Council also expressed concern that the U.S. Department of Education is limiting its commitment to the kinds of problems and issues that might better be addressed through other scientifically appropriate methodologies (AERA, n.d.-a). More information is available from AERA, the largest educational research association, on the technical aspects of conducting research and on the ethics of research in education using human subjects (AERA, n.d.-b).
For more information on descriptive research, including correlational studies and case studies, see Grimes & Schulz (2002); Descriptive and Correlational Designs (2003); and Schmidt (n.d.).
Have there been many research studies that have used randomized trials in early childhood education?
One way to determine how many randomized trials have been conducted is to search the ERIC database, education's database of record. Very few occurrences of the phrase "randomized trials" are found in ERIC database descriptions of research studies about early education programs or interventions, and many of those studies are old. In a search of the ERIC database from 1966 through June 2003, using the broadest possible strategy (i.e., looking for the word random within two words either way of the words control, trial, or design), 463 items were retrieved (out of more than one million items in the ERIC database), of which 258 were research or evaluative reports and 19 were information analysis reports. When combined with an array of early childhood education terms, these numbers were reduced to 52 items for the same time period, of which just 34 were research or evaluative reports.
Other databases, however, also contain studies related to education. For example, the National Research Panel reported that they found many more randomized trials about teaching children to read in PsycInfo (the psychology database) than they found in ERIC.
Although these numbers tend to support the claim that few randomized trials have been conducted in early childhood education research to date, that situation is changing. In July 2002, the U.S. Department of Education awarded Preschool Curriculum Evaluation Research Grants to carry out randomized clinical trials of seven preschool curricula. The researchers receiving these grants are implementing rigorous evaluations of these curricula in order to provide information that can support informed choices of classroom curricula for early childhood programs. Over time, the U.S. Department of Education expects that the results from these studies will strengthen the research base for early childhood practice.
Conclusion
Research of all kinds continues to be conducted in early childhood settings. As work continues to improve early childhood programs, it is probably more important than ever to remember, as the American Educational Research Association points out, that all high-quality research consists of well-specified theory, sound problem formulation, the application of designs and methods appropriate to the research questions being investigated, and integrity in the conduct of research and the community of research findings. If early childhood educators want to know which curriculum, program, or teaching approach works, they should look for the most rigorous research available, as discussed in this Guide.
References
American Educational
Research Association (AERA). (n.d.-b). Ethical standards of AERA
[Online].
Available: http://www.aera.net/aboutaera/?id=222 [2003, December 4].
Coalition for
Evidence-Based Policy. (2003). Identifying and implementing educational
practices supported by rigorous evidence: A user friendly guide
[Online]. Washington, DC: Coalition for Evidence-Based Policy.
Available:
http://www.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid.pdf
[2003, December 11].
Descriptive
and correlational designs [Online]. (2003).
Available: http://clcpages.clcillinois.edu/home/soc455/psycweb/
research/descriptive.htm
[2003, December 11].
Gribbons, B.,
& Herman, J. (1997). True and quasi-experimental designs. Practical
Assessment, Research & Evaluation [Online], 5(14).
Available: http://PAREonline.net/getvn.asp?v=5&n=14
[2003, December 11].
Grimes, D. A., & Schulz, K. F. (2002). Descriptive studies: What they can and cannot do. Lancet, 12:359(9301), 145-149.
National Institute
for Early Education Research (NIEER). (2003). Ongoing NIEER research
[Online].
Available: http://nieer.org/docs/index.php?DocID=35
[2003, December 10].
Schmidt, S.
R. (n.d.). Research methods for the digitally inclined [Online].
Available: http://www.mtsu.edu/~sschmidt/methods/correlational.html
[2003, December 10].
U.S. Department
of Education. [n.d.]. The No Child Left Behind Web Site [Online].
Available: http://www.nochildleftbehind.gov/next/overview/index.html
[2003, December 10].Editor's note: This url has changed:http://www.ed.gov/nclb/landing.jhtml
U.S. Department of Education. (2002a, September 30). Preschool curriculum evaluation research. Notice [Online]. Federal Register, 67(189). Available: http://www.ed.gov/legislation/FedRegister/proprule/2002-3/093002e.html [2003, December 8].
U.S. Department of Education. (2002b, July). Preschool curriculum evaluation research grants [Online]. Available: http://www.ed.gov/offices/OERI/pcer_materials/ [2003, December 8].Editor's note: This url has changed:http://www.ed.gov/offices/OERI/pcer_materials/index.html
Whitehurst,
G. J. (n.d.). Evidence-based education (EBE) [Online].
Available:
http://www.ed.gov/nclb/methods/whatworks/eb/edlite-slide001.html [2003, December 8].
Web Resources on Research Terminology and Methods
Many universities and research centers provide Internet-accessible glossaries of research terms. The following Internet documents define and discuss many of the research terms used in this Web Guide:
Project Gold
Research Methods Glossary
http://www.bath.ac.uk/e-learning/gold/glossary.html
Project Gold's research methods glossary was intended for use by nurses in a distance education course.
Research Methods: An Introduction to Systems of Human Inquiry: Comparison
of Some Qualitative and Quantitative Research Methods
http://www.edb.utexas.edu/faculty/scheurich/proj1/inquiry.html
An early childhood friendly comparison of some quantitative and qualitative research methods.
Writing@CSU: Writing Guide
http://writing.colostate.edu/guides
A listing of "key terms" related to research is part of the Colorado State University Web Site.
Resources for Understanding Research
A CSR [Comprehensive School Reform] Practitioner's Guide to Scientifically Based Research
http://www.csrclearinghouse.org/index.cgi?l=guides_tools_guide_to_sbr
Editor's note: This url has changed: http://www.csrclearinghouse.org/pubs/pg/.
Online Education
Research Resources. ASCD Research Brief.
http://www.ascd.org/portal/site/ascd/menuitem.08d1afa0ba92
c1cf989ad324d3108a0c
NCREL's Learning
Point: A Call for Evidence
Responding to the New Emphasis on Scientifically Based Research
http://www.ncrel.org/info/nlp [Editor's note (07-23-07): this url is no longer active.]
No Child Left
Behind: Scientifically Based Research
http://www.ecs.org/html/issue.asp?issueid=195&subIssueID=114
Research Methods
Knowledge Base
http://www.socialresearchmethods.net/kb/
Scientific
Research in Education (2002)
This full text online text, from the Committee on Scientific Principles
for Education Research, identifies the guiding principles and features
of education research. Editor's Note: this url is no longer active.http://search.nap.edu/books/0309082919/html/ Editor's note: See also:Advancing Scientific Research in Education (2004)
http://books.nap.edu/catalog.php?record_id=11112
Scientifically
Based Research
http://eric.uoregon.edu/publications/digests/digest167.html
Understanding
Research: Top Ten Tips for Advocates and Policymakers
http://www.voicesforamericaschildren.org/Content/ContentGroups/Publications1/
Voices_for_Americas_Children/Advocacy/20024/understandingresearch.pdfEditor's note: This url has changed:http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED459965
Using Research
and Reason in Education: How Teachers Can Use Scientifically Based
Research to Make Curricular & Instructional Decisions
http://www.nifl.gov/partnershipforreading/publications/k-3.html
Major Federal Research Grant Programs
U.S. Department of Education
Preschool Curriculum Evaluation Research Grants
http://www.ed.gov/offices/OERI/pcer_materials/index.html
National Institute of Health and Human Development (NICHD)
Funding Opportunities
http://www.nichd.nih.gov/funding/funding.htm
Ongoing Research in Early Childhood Education
Prekindergarten
Research in Progress Database
http://prekrip.org/ [CEEP Editor's note: this link is no longer active]
This new listing of research studies in progress that are focused on state-funded prekindergarten programs will be available in early 2004.
Researchers
in Early Childhood Education
View a list of researchers compiled by the American Educational
Research Association.
http://ec.wmich.edu/aera/search/TopicSearch.htf?
InterestList.%25LIKE%25=Early+Childhood
Major Research Organizations in Early Childhood Education
National Institute
for Early Education Research (NIEER)
http://nieer.org/
NIEER was funded by the Pew Charitable Trusts in 2002. NIEER:
…supports early childhood education initiatives by providing objective, nonpartisan information based on research. The goal of NIEER is to produce and communicate the knowledge base required to ensure that every American child can receive a good education at ages three and four. The Institute seeks to provide policy makers with timely information addressing the practical problems they face. The Institute offers independent research-based advice and technical assistance to four primary groups: policy makers, journalists, researchers, and educators.
On the NIEER Web site of particular interest to the early childhood community are the State Databank (http://nieer.org/yearbook/states) and its other resources (http://nieer.org/resources/), including publications, a database of experts, and descriptions of studies now underway at NIEER.
National Center
for Early Development and Learning (NCEDL)
http://www.fpg.unc.edu/~NCEDL/index.cfm
National Institute
of Child Health & Human Development (NICHD)
http://www.nichd.nih.gov/
The National Institute of Child Health and Human Development (NICHD) is part of the National Institutes of Health, the biomedical research arm of the U.S. Department of Health and Human Services. The mission of the NICHD is to ensure that every person is born healthy and wanted, that women suffer no harmful effects from the reproductive process, and that all children have the chance to fulfill their potential for a healthy and productive life, free of disease or disability.
To learn more about the research priorities of NICHD, browse through Dr. G. Reid Lyon's testimony before the House of Representatives: http://www.nichd.nih.gov/crmc/cdb/r_house.htm]Editor's Note: this url is no longer active.
In 1999, Dr. Lyon described NICHD's research related to young children in this way:
The NICHD has developed and supports a large research network consisting of 42 sites in North America, Europe, and Asia that are working in a concerted multidisciplinary fashion to identify: (1) the critical environmental, experiential, cognitive, genetic, neurobiological, and instructional conditions that foster strong reading and writing development; (2) the risk factors that predispose children to difficulties in learning to read and write; and (3) the instructional approaches and procedures that foster optimal reading development, as well as practices and procedures for preventing and remediating reading and writing difficulties.
Child Development and Behavior Branch
http://www.nichd.nih.gov/about/org/crmc/cdb
What Works
(WWC) Clearinghouse
http://w-w-c.org/Editor's note: This url has changed:http://ies.ed.gov/ncee/wwc
A major new resource for educators interested in finding out more about evidence-based education is the What Works Clearinghouse, although this resource does not yet deal with early childhood topics. Established in 2002 by the U.S. Department of Education's Institute of Education Sciences, the goal of WWC is to identify for educators, policy makers, and the public interventions or approaches in education that have "demonstrated beneficial causal relationships to important student outcomes." Administered by the U.S. Department of Education through a contract to a joint venture of the American Institutes for Research and the Campbell Collaboration, the WWC has established standards for high-quality research (see http://w-w-c.org/reviewprocess/standards.htmlEditor's note: This url has changed:http://ies.ed.gov/ncee/wwc/pdf/study_standards_final.pdf), identifies new topics annually, and prepares systematic reviews of the evidence on those topics. None of the first round of topics is related primarily to early childhood education, but the home page of the What Works Clearinghouse Web site encourages readers to suggest topics that might be considered in future years.
Center for
Early Childhood Research (University of Chicago)
http://eci.uchicago.edu/
Center for
Improving the Readiness of Children for Learning & Education
http://www.uth.tmc.edu/circle/
Child Care
Research Collaborative and Archive
[no url publicly available yet]
Child Outcomes
Research and Evaluation
http://www.acf.dhhs.gov/programs/core/index.html
Child Trends
http://www.childtrends.org/
Children, Youth,
and Families Education and Research Network
http://www.cyfernet.org/
Federal Interagency Forum on Child and Family Statistics
http://www.childstats.gov/
Clearinghouse
on International Developments in Child, Youth, and Family Policies
http://www.childpolicyintl.org/
Education Development
Center
http://main.edc.org/
Education Commission
of the States
http://ecs.org/
Families and Work Institute
http://www.familiesandwork.org/
Fight Crime:
Invest in Kids
http://www.fightcrime.org/
Institute of
Education Sciences
http://www.ed.gov/about/offices/list/ies/index.html
National Institute
for Early Education Research (NIEER)
http://nieer.org/
National Center
for Early Development and Learning
http://www.fpg.unc.edu/~NCEDL/index.cfm
Research Forum
on Children, Families, and the New Federalism
http://www.researchforum.org/
Research Reports
from the National Research and Development Centers
http://research.cse.ucla.edu
U.S. Census
Bureau
http://www.census.gov/
Program-Focused Research Centers
Carolina Abecedarian
Project
http://www.fpg.unc.edu/~ncedl/
High/Scope
Educational Research Foundation
http://www.highscope.org/
State Comparisons and State Data Collections
National Child
Care Information Center
State Child Care Profiles
http://nccic.org/
The state profiles include demographic information about the children, families, and child care in each state, as well as contact information for different state agencies involved in child care. The profiles also contain links to additional state and national resources.
National Institute for Early Education Research (NIEER)
http://nieer.org/
State "yearbook" will soon be available on early education and care.
National Center for Education Statistics
http://nces.ed.gov/surveys/SurveyGroups.asp?Group=3
The Early Childhood Longitudinal Study (ECLS) program has been designed to include two overlapping cohorts: a Birth Cohort (N = 13,500) and a Kindergarten Cohort (N = 22,000). The birth cohort will follow a sample of children from birth through the first grade. The kindergarten cohort will follow a sample of children from kindergarten through the fifth grade. For Project Summaries, visit:
http://nces.ed.gov/ecls/birth.aspand http://nces.ed.gov/ecls/kindergarten.asp.ECLS provides national data on children's status at birth and at various points thereafter; children's transitions to nonparental care, early education programs, and school; and children's experiences and growth through the fifth grade. ECLS also provides data to test hypotheses about the effects of a wide range of family, school, community, and individual variables on children's development, early learning, and early performance in school.
Education Commission of the States
Kindergarten Database
http://ecs.org/html/educationIssues/EarlyLearning/KDB_intro.asp
The site includes state profiles, state comparisons, and 50-state reports. The site also includes state policy information (statutory and finance) for all 50 states. For a smaller number of states, information on how state rules and regulations address kindergarten is included.Pre-Kindergarten Database
http://ecs.org/clearinghouse/27/24/2724.htm
State profiles, state comparisons, and 50-state reports are included on this site, along with the following:
- A description of each of the pre-kindergarten programs including a brief legislative history of the program.
- Information on the administrative auspices of the program
- Funding information including how much the state appropriates to fund the program and how the state distributes those funds to providers, as well as information on funders other than the state.
- Whether or not the state supplements Head Start, and if so, how much is appropriated for these programs.
- Eligibility requirements for participating agencies.
- The number of children served by each program and the eligibility requirements for participating children.
- Hours of program operation and availability of wraparound services.
- Program standards such as teacher qualification requirements, teacher-to-child ratios, and maximum class size restrictions.
- Evaluations performed by each of the state programs.
- Legislation concerning pre-kindergarten initiatives in the states.
- Web site and contact information for each of the programs.
Compilation of State Standards for K-12 Curriculum
http://www.mcrel.org/standards-benchmarks/
Assessing the New Federalism State Database
http://www.urban.org/content/Research/NewFederalism/Data/StateDatabase/
StateDatabase.htmFree registration required.http://anfdata.urban.org/drsurvey/login.cfm
This database includes information on the 50 states and the District of Columbia in areas including income security; health; child well-being; demographic, fiscal, and political conditions; and social services.
Online Libraries and Resource Collections
National Child
Care Information Center Online Library
http://ecap-iel.crc.uiuc.edu/cgi-bin/nccic/searchnccic.cgi [CEEP Editor's note: this link has changed: http://oll.nccic.acf.hhs.gov/nccic-OLL/searchnccic.cgi]
ERIC Database
http://eric.ed.gov/
Online Research Periodicals in ECE
Contemporary
Issues in Early Education
http://www.wwwords.co.uk/ciec
Early Childhood
Focus
http://earlychildhoodfocus.org/index.phpEditor's note: This url has changed:http://www.earlychildhoodfocus.org/artman2/publish/topics/index.shtml
Early Childhood
Research and Practice
http://ecrp.uiuc.edu/
