Research and Evaluation News

Page Contents

2002/07/29: Education research is under the microscope      U. S. Department of Education Awards Contract For 'What Works Clearinghouse'      New Directions for Program Evaluation at the U.S. Department of Education      Department of Education Announces New Approach to Program Evaluation and Data Collection      Roddy Stinson: Monitoring of COPS' Project Quest would earn statistical 'F'

2002/07/29: Education research is under the microscope

Education research is under the microscope

Quality of data crucial as schools base more decisions on studies

07/29/2002

The Dallas Morning News

Education research has been a punching bag for decades – largely for good reason.

Streams of studies serve up contradictory facts. Anecdotes and hearsay are often as prominent as hard data. And some scholars, motivated by ideology, seem to reach their conclusions before the research even starts.

"When you look back, there have been entire movements in education launched on the basis of a few anecdotes, a lot of rhetoric, and not much evidence," said Gerald Sroufe, director of government relations for the American Educational Research Association.

But now the field is facing unprecedented pressure that could change the nature of research into what works in schools.

A new federal law requires schools to base dozens of policy decisions on research or risk losing federal money, and a bill pending in Congress would, for the first time, attempt to set quality standards for government-funded education research.

"There's some very good work out there, but there's a lot of very bad work, too," said Stanford professor Richard Shavelson. "You just have to figure out how to separate the two."

The federal education bill passed in January uses some variant of the phrase "scientifically based research" 110 times.

It requires states and districts to use research to determine their approaches to everything from teacher training to the hiring of security guards. The Bush administration has spoken often about its desire to make education an "evidence-based field."

That, in turn, has fostered the desire to establish quality-control measures for research.

"I've seen some colossal missteps in education – things like New Math and 'schools without walls' – that were seemingly never tested before being tried," said Rep. Michael Castle, R-Del., chairman of the House education reform subcommittee and author of the pending bill.

Teachers looking for ways to improve their skills find that different researchers give contradictory evidence for what works and what doesn't.

Academics line up on both sides of every major issue – vouchers, testing, social promotion – and are often accused of cooking the data to get the results they want.

Peer reviews

Reformers such as Mr. Castle would like the field of education research to look more like the world of medical research.

Until the 1950s, doctors operated largely on anecdotal evidence about what worked and what didn't. As independent professionals, many felt that they – not some distant academic researcher – were best able to decide the effectiveness of treatment.

"The prevailing attitude was that each doctor was his own experimenter," said Robert Baruch, an education researcher at the University of Pennsylvania. That's not far from how some teachers feel today, he said.

Over time, physicians were convinced that controlled clinical trials could lead to better medicine. Now, medical research is strictly regulated by government – the Food and Drug Administration – and by an academic community that conducts rigorous peer reviews of scholarly work in prestigious publications such as The Journal of the American Medical Association.

Mr. Castle's bill, which passed the House on April 30, would require peer reviews of all federally funded studies and would model research regulations on those of other federal agencies, such as the National Institutes of Health and the National Science Foundation.

The bill is now sitting in the Senate committee chaired by Sen. Edward Kennedy, D-Mass. A Kennedy spokesman said that Senate work on the bill will begin in September, but that the committee will likely create its own bill rather than work from Mr. Castle's.

Some academics agreed that the overall quality of education research needs to be raised but said they are wary of government standards.

"The intent is appropriate, but how it gets translated into law is questionable," Dr. Shavelson said. "I don't think it's the role of the federal government to define science."

Raising bar

For example, the most successful style of medical experiment has been the randomized trial, in which a group is randomly divided into two parts: One gets a certain treatment and the other doesn't. But local control of schools makes random assignment difficult for education studies, he said.

"When the researchers come in and say, 'We want you to abandon the way you teach and try this new way for three years so we can see if it works,' are parents going to go along with that? Or will local control come into play and say 'We're in charge, we're not going to [do] that'?"

Maintaining strict research conditions in a school is notoriously difficult: students move, principals get fired, policies and demographics change. In one recent long-term experiment led by Boston University researcher Christine Rossell, half of her subjects had withdrawn from the study within three years.

"The so-called hard sciences – chemistry, physics – are misnamed," she said. "Those should be called the easy sciences. You can keep ions and molecules in a beaker for decades. They don't have any human rights.

They don't move from one school district to another in the middle of a research study and mess up your data. Researching real human beings who don't always follow directions – that's a hard science."

"People think you can just look at test scores and figure out what works," she said. "Doing a strict scientific study is extremely difficult and gets very, very expensive."

According to a National Research Council study released this year, federal funding for educational research has dropped from more than $400 million in 1973 to about $130 million today. Only about one-tenth of 1 percent of education funding in America goes to research.

"If we were a drug company or GM or Ford, spending that little money on research, we wouldn't stay in business," Dr. Baruch said.

Even without government intervention, some private sources are trying to improve research quality. Dr. Baruch leads the Campbell Collaboration, a 2-year-old organization whose goal is to evaluate existing education research and "screen the good stuff from the poor stuff." He said that sometimes fewer than 10 percent of the studies they evaluate meet the highest standards the collaborative looks for.

"One of our biggest problems is that it's all called 'research,' no matter how good it is," Dr. Sroufe said. "Once you see the headline 'Researchers show such-and-such,' it's difficult for the public to get much deeper than that."

E-mail jbenton@dallasnews.com

U. S. Department of Education Awards Contract For 'What Works Clearinghouse'

Use of research-proven strategies is one of key principles of No Child Left Behind

The U.S. Department of Education

The U.S. Department of Education has awarded a five-year, $18.5 million contract to a special joint venture to develop a national What Works Clearinghouse, which will summarize evidence on the effectiveness of different programs, products, and strategies intended to enhance academic achievement and other important educational outcomes.

The clearinghouse will help provide education decision-makers with the information they need to make choices guided by the best available scientific research. The use of research-proven strategies based on sound scientific evidence is one of the key principles of No Child Left Behind. "By providing educators with ready access to the best available scientific research evidence, the clearinghouse will be an important resource for enhancing the quality of local decision-making and improving program effectiveness," said U.S. Secretary of Education Rod Paige. "And it will help transform education into an evidence-based field."

The What Works Clearinghouse will provide the following easily accessible and searchable online databases:

An educational interventions registry that identifies potentially replicable programs, products, and practices that are claimed to enhance important student outcomes, and synthesizes the scientific evidence related to their effectiveness.

An evaluation studies registry, which is linked electronically to the educational interventions registry, and contains information about the studies constituting the evidence of the effectiveness of the program, products, and practices reported.

An approaches and policies registry that contains evidence-based research reviews of broader educational approaches and policies.

A test instruments registry that contains scientifically rigorous reviews of test instruments used for assessing educational effectiveness.

An evaluator registry that identifies evaluators and evaluation entities that have indicated their willingness and ability to conduct quality evaluations of education interventions.

The contract was awarded to Campbell Collaboration of Philadelphia and the American Institutes for Research of Washington, D.C., along with their subcontractors, Aspen Systems of Rockville Md., Caliber Associates of Fairfax, Va., and the Education Quality Institute of Washington, D.C. The Campbell Collaboration-American Institutes for Research Joint Venture was established specifically to develop and maintain the clearinghouse, and brings together nationally recognized leaders in the field of rigorous reviews of scientific evidence.

Robert Boruch, principal investigator for the clearinghouse, chairs the Steering Group of the Campbell Collaboration, an international consortium of social science researchers who conduct systematic reviews of randomized and some non-randomized trials on the effectiveness of interventions in education and other social sectors. He is also University Trustee Chair Professor of the Graduate School of Education, the Statistics Department at the Wharton School, and the Fels Center for Government at the University of Pennsylvania.

Boruch's work on the design of randomized field trials for planning and evaluating social and educational programs has received recognition from the American Educational Research Association (Research Review Award), the Policy Studies Organization (Donald Campbell Award), and the American Evaluation Association (Gunnar and Alva Myrdal Award).

Rebecca Herman, project director for the clearinghouse, is a principal research Analyst at the American Institutes for Research, and lead author of An Educator's Guide to Schoolwide Reform, the premier review of scientifically based evidence on the effectiveness of prominent school reform models. She has also served as Principal Investigator for the National Longitudinal Evaluation of Comprehensive School Reform, the largest federal government investment in studying whole-school reform efforts and their impact on student achievement.

New Directions for Program Evaluation at the U.S. Department of Education

Background

What's the Problem?

The Department spends upwards of $100 million per year on program evaluation and program data collection. Yet key decision makers in the Department, in the Office of Management and Budget, and on Capitol Hill continue to operate without the information they need. At the same time, our evaluation studies are not as helpful as they could be to practitioners at the local level, nor can they answer questions about causation. There must be a better way to conceptualize, organize, and execute program evaluation.

We propose a significant shift in program evaluation, away from a compliance model and towards a system of research and evaluation focused on results and the effectiveness of specific educational interventions.

From Compliance to Performance

The Department has recently developed a business case for OMB for a "Performance-Based Data Management Initiative." This initiative seeks to replace the hundreds of program compliance reports with an integrated electronic system of data collection focused on outcomes. One major goal of the initiative is to reduce data burden on our partners, but we also aim to collect higher quality data, data related to what matters most: results.

These data will tell us whether the education system and its components are performing well, and they might help us understand which of our programs are having the greatest impact, but they will not tell us why. How can we supplement this data management system with program evaluations that give decision makers the information they need to allocate funds, make policy changes, and consider new directions? How can we build on the knowledge base so that practitioners know "what works" and can spend their federal dollars wisely?

Four Types of Program Evaluation

Generally speaking, there are four types of program evaluation, each with their own key audiences, questions and methodologies. We need to develop a balanced evaluation portfolio that embraces each of these four types of studies:

Type of Evaluation Audience Key Questions Timeline Methodologies Continuous Improvement Program Staff How can we continuously improve our communication and guidance in order to achieve our objectives? ASAP Market research methods like fast response surveys, focus groups, etc.

Performance Data Appropriators/OMB Which federal programs are working? Are some programs more effective than others? Annual Analysis using Performance-Based Data Management System Implementation Studies Authorizers How well are programs being implemented? Are the policy changes we made leading to improved outcomes? 5-7 Years Passive, descriptive evaluation studies, using methods like self-reported surveys and case studies Field Trials Practitioners What works? What specific educational interventions lead to increased student achievement? Long-term Random assignment field trials with longitudinal data

Evaluation Type #1: Continuous Improvement

The No Child Left Behind Act gives the Department a mandate to make sweeping changes in federal policy. We are currently launching a national campaign to transform the culture of education. Especially in areas like research-based reading, we need to know immediately if our message is having an impact. We need good information, right away, to fine-tune our materials and approaches.

We also need to know right away if there are any snags at the state or local levels in the implementation of this complex law. Waiting for an annual performance report or a five-year evaluation study is simply too long.

Methods common in market research would be appropriate for this purpose. Fast-response surveys--even via telephone--might solicit quick feedback from target audiences (like reading teachers). Focus groups would allow program staff to probe reactions to communications pieces.

Because these data would not be shared externally, but would provide a feedback loop for continuous improvement, speed is more important than rigor.

Evaluation Type #2: Program Performance Data

Every year, Congressional appropriators and the Office of Management and Budget want to know if individual programs are working so that budget decisions can be based on data. The Government Performance and Results Act requires program-by-program performance information for this very reason.

Unfortunately, answering the question, "Is this program working?" is surprisingly hard to do. This is for four reasons:

1.Many Department programs, like Title I, Perkins, and Adult Education, are not programs, but funding streams. In other words, they are not specific educational interventions, but rather sources of flexible funds that can be spent in a myriad of ways.

2.New flexibility provisions encourage a blending of resources. For good reason, federal policy encourages States and localities to view federal funds holistically. New provisions encourage our partners to take funds targeted for one purpose and use them for another. This is good policy (and an acknowledgment of the appropriate federal role in education) but it makes evaluating the impact of individual federal funding streams quite tricky.

3.Linking interventions to impacts takes time. Even for the (relatively few) Department programs that mandate a specific educational intervention, it is very difficult to provide annual impact data. Ideally, studies to track these impacts would be longitudinal and would use field trials with random assignment of treatment and control groups. These methods will not tend to produce annual or nationally representative data.

4.Isolating the impact of Federal policy or funding (versus state or local efforts) is difficult or impossible. Rightly, federal policy increasingly seeks to coordinate federal programs and approaches with state and local efforts. A good example is Title I, which in effect mandates standards-based reform, a strategy already practiced by many states. It is impossible, then, to attribute outcomes to federal policy changes, versus state efforts.

So with these caveats in mind, how can we provide good information to Congress and OMB on an annual basis? How can we identify program-by-program GPRA indicators that are linked to student outcomes? It depends on the type of program.

Flexible Funding Streams

For large formula-based grant programs, like Title I, we will report:

National achievement trends, as appropriate for the program (ideally aligned with the Department's strategic plan indicators) Achievement trends for schools receiving program dollars

The data for these indicators will come from the Performance-Based Data Management System. Already, through the comprehensive database built for us by AIR, we can track achievement trends for every school in America. As the system becomes more sophisticated, we will know which schools receive which pots of federal funds.

Several funding streams will, in effect, be lumped together with specific achievement indicators. For example, Title I, Title II, Title III and Reading First of the ESEA all aim to increase student achievement (through various means). So national achievement trends will be used as indicators for all of them. But the ability to analyze achievement by school will also allow us to find out if certain pots of money appear to be more effective than others. For example, we'll know, over time, if schools receiving both Title I and Reading First funds do or do not outperform schools receiving only Title I funds.

There are many methodological concerns with this approach, but it will give appropriators a rough idea of which formula grant programs are having the largest impact.

Competitive Programs

For the Department's smaller, competitive programs, we will also look at progress in student achievement, but only for the schools or students served by the program. For example, for the American History program we would look at history achievement indicators for schools receiving money under the program.

In many cases, our smaller programs will struggle to find relevant achievement data. We might decide that no performance data is better than bad, compliance-based information.

This approach to annual performance data is not perfect, but is a step in the right direction. To more fully understand the impact of federal programs, or specific educational interventions funded by them, we must use other, longer-term methods.

Evaluation Type #3: Descriptive Studies of Program Implementation

Authorizing committees on Capitol Hill rightly want to know how well various federal education programs and policies are being implemented, and what impact they are having. This information would feed into subsequent program reauthorizations, typically on a five-year cycle.

The bulk of evaluations produced by the Planning and Evaluation Service (PES) are targeted toward this purpose. Their methodologies include a variety of "passive research" designs, usually including:

Nationally representative, self-reported surveys of districts, principals, and/or teachers Case studies of representative sites Co-relational analysis

PES is very good at producing these types of descriptive studies. They can provide a wealth of information to Congress about what's happened in the real world since the previous reauthorization, and what role federal program and policy changes have played.

For example, descriptive studies of the Title I program have helped us to understand the challenges faced by state and local agencies when implementing the accountability provisions of the law. They also give some insight into the relationship between the law's requirements and the actions taken by Title I schools.

However, these descriptive studies have failed to provide solid evidence proving a causal link between specific policies or interventions and changes in student achievement. It is impossible for these studies to make that causal link, since they do not include randomly assigned treatment and control groups. (Plus, as explained above, Title I is not in itself an intervention, making a study of its effectiveness difficult.)

Descriptive implementation studies play a crucial role in understanding the impact of policy changes, but they are no substitute for rigorous field trials of specific interventions.

Evaluation Type #4: Rigorous Field Trials of Specific Interventions

Even with high-quality fast-response surveys, annual performance data, and descriptive studies, we still cannot answer the question on the minds of practitioners: "What works?" To be able to make causal links between interventions and outcomes, we need rigorous field trials, complete with random assignment, value-added analysis of longitudinal achievement data, and distinct interventions to study.

This approach might be considered "research" rather than "evaluation." Whatever the name, the Department's evaluation agenda would be incomplete without it. It is a fair use of evaluation dollars because federal program funds are paying for the interventions to be studied.

In some cases, these types of field trials will be able to answer the question, "Is the federal program working?" There are a few federal programs that are, in and of themselves, specific interventions.

In other cases, these trials will instead answer the question, "Does a specific intervention (funded by federal dollars) produce results?" For example, field trials might examine the effectiveness of specific reading interventions, or whole-school designs, or professional development regimes. All of these are funded by large formula programs (like Title I) but are not the program itself.

Who's Responsible for What?

While coordination in program evaluation is essential, each type of evaluation will be assigned to a distinct part of the Department:

Continuous Improvement Studies will be the responsibility of the Program Assistant Secretaries. For example, the assistant secretary for elementary and secondary education will be responsible for developing and implementing market research studies related to Title I and Reading First. These studies are integral pieces of the day-to-day work of the program offices.

Program Performance Data will be the responsibility of the Performance-Based Data Management Team, housed within the Executive Management Team. The Program Assistant Secretaries will play a crucial role, though, in helping to define the performance measures for individual programs.

Descriptive Implementation Studies will continue to be the responsibility of the Planning and Evaluation Service, reconstituted as the "Policy and Program Studies Service." Evaluation questions should be developed in cooperation with the Program Offices, but also with the needs of the authorizing committees in mind. This new office will also serve as an incubator of new policy ideas, and will continue to commission timely policy studies.

Field Trials of Specific Interventions will be the responsibility of a new evaluation unit within the Office of Educational Research and Improvement. OERI will need to develop the capacity to oversee numerous high quality evaluation studies; a regular funding stream should allow it to do so.

This page last modified—April 4, 2002 (jer).

Department of Education Announces New Approach to Program Evaluation and Data Collection

(202) 401-1576

U.S. Secretary of Education Rod Paige announced today the major reorganization of the department's program evaluation and data collections services.

This move is a shift away from a compliance model toward a system of research and evaluation focused on results and specific effective education interventions.

"We spend millions of dollars every year to collect data on and evaluate our programs," Paige said. "This is a serious effort to provide more value for the taxpayer's dollars in these activities. We aim to establish a more efficient data collection and dissemination system, one that provides timely and more useful information to those who work every day to improve student achievement."

A major piece of this new approach to program evaluation is the Performance-Based Data Management Initiative.

This initiative, first proposed in the department's FY 2003 budget submission to Congress, would consolidate the department's myriad data collections—some of which still use pencil and paper—and replace them with a centralized, consolidated, electronic system. This initiative aims to dramatically reduce the reporting burden on elementary and secondary schools, while gathering much better data about program effectiveness.

In addition, the department's Planning and Evaluation Service will be reconstituted as the Policy and Program Studies Service (PPSS), but will continue to produce descriptive studies of program implementation, and will commission occasional policy papers on important topics.

"This new office (PPSS) will still have the responsibility of providing descriptive implementation studies," said Under Secretary Eugene Hickok. "But we also envision it as the department's 'think tank,' to serve as an incubator of new policy ideas for more effective teaching and learning."

Another aspect of the reorganization will focus on field trials of department-funded initiatives. For example, rather than just studying the Title I program itself, the department will also conduct rigorous field trials of education interventions, such as comprehensive school models and reading programs, funded through the Title I program. These new studies will seek to identify, through rigorous methodology, "what works" in education.

These field trials will be conducted by a new evaluation unit within the Office of Educational Research and Improvement (OERI).

"Through these evaluations, we hope to get a better handle on what specific education interventions lead to increased student achievement and better learning environments," said OERI Assistant Secretary Grover "Russ" Whitehurst.

Paige said the new changes are scheduled to be implemented as soon as possible.

Details of the departments new data collection and program evaluation plans may be viewed in the publication New Directions for Program Evaluation at the U.S. Department of Education at our Web site at http://www.ed.gov/PressReleases/04-2002/evaluation.html.

###

Roddy Stinson: Monitoring of COPS' Project Quest would earn statistical 'F'

Web Posted: 08/13/2002

Since 1992, San Antonio taxpayers have shelled out $13,216,226 to Project Quest, a workforce-training agency created, promoted and vigorously protected by the controversial activist group, Communities Organized for Public Service

This column's efforts to determine how well City Hall monitors the COPS-controlled agency's finances and job-training effectiveness continue to produce troubling information and worrisome documents. To wit:

At my request, the city's Department of Community Initiatives (DCI), which evaluates Project Quest, sent me a copy of its last "Program Performance Review" of the agency, dated Aug. 1, 2001.

Under a section titled "Assessment of Effectiveness," city staffers Melody Woosley and Ayten Cibildak reported:

"Project Quest retained a staff person to monitor effectiveness and job retention of former graduates.

Since 1992, Quest has served over 1,900 participants with 1,480 graduating from the various job-training programs.... Staff tracked the outcomes of 277 Project Quest alumni during Fiscal Year 2001. The average salary of these alumni has increased 18% from $10.15 to $11.94 per hour.

Eighty-two percent of the alumni receive additional compensation through benefits packages from their employer."

The numbers sounded swell.

One problem:

On inquiring about how the information was obtained, I was told by DCI director Dennis Campa:

"The respondents were not randomly selected. Project Quest attempts to contact all graduates. The 277 reported were those who responded."

On making an inquiry to the agency, I learned that since the city's 2001 "review," Project Quest has tracked down 83 additional "alumni"... for a grand total of 360 (out of 1,480) graduates.

To put it another way:

After a yearlong search, the agency hasn't been able to locate 75.7 percent of its graduates.

Yet the city's reviewers implied that the number of alumni found and interviewed by Project Quest represented the agency's total "effectiveness" — a conclusion that would earn an "F" in any statistical analysis class on the planet.

During an exchange of e-mails, Campa promised:

"For Fiscal Year 2003, we will strengthen performance reporting by making long-term follow-up a mandatory contract requirement."

Eleven years — and $13,216,226 — too late.

In the same "performance review," under "Client Surveys," the city's reviewers noted:

"Fifty-two surveys were administered to Quest clients....

"Overall, the respondents believe the quality of service Quest provides is good or excellent."

As a result, the reviewers concluded that Quest staff should be "commended."

Only after I inquired about the validity of the surveys did DCI reveal another instance when the agency and the city used an evaluative procedure that would earn any college freshman an "F."

Campa:

"The 52 surveyed clients were not randomly selected. Project Quest staff distributed and collected the surveys from participants who visited the Project Quest office."

Again, he promised:

"DCI will implement alternate methods of data collection to improve the methodology of survey implementation."

Which leads to more disturbing questions:

— Barring this inquiry, how many decades would have passed before the city initiated a serious analysis of Project Quest's effectiveness?

— Why has lax monitoring of the COPS-controlled agency been the standard operating procedure at City Hall?

For answers to those questions (and many others), keep reading this space.

To contact Roddy Stinson, call (210) 250-3155 or e-mail rstinson@express-news.net. His column appears Sundays, Tuesdays, Thursdays and Saturdays.

Contact Us

Enter feedback, comments, questions, or suggestions:

Email this page

amazon.com

Frederic M. Wolf: Meta-Analysis: Quantitative Methods for Research Synthesis

Search

(Indexed quarterly)

positivepractices.com

WWW

Translations

Caution: Machine generated language translations may contain significant errors. Use with care.